Todo

-- For other people
- Multithread vc1.
- Multithread an intra codec like mjpeg (trivial).
- Fix mpeg1 (see below).
- Try the first three items under Optimization.
- Fix h264 (see below).
- Try mpeg4 (see below).

-- Bug fixes

General critical:
- Error resilience has to run before ff_report_frame_progress()
is called. Otherwise there will be race conditions. (This might already
work.) In general testing error paths should be done more.
- 'make fate THREADS=2' doesn't pass. Most failures are due to
bugs in vsync in ffmpeg.c, which are currently obscuring real failures.

h264:
- Files that aren't parsed (e.g. mp4) and contain PAFF with two
field pictures in the same packet are not optimal. Modify the
nals_needed check so that the second field's first slice is
considered as needed, then uncomment the FIXME code in decode_postinit.
Ex: http://astrange.ithinksw.net/ffmpeg/mt-samples/PAFF-Chalet-Tire.mp4
- The code added to shorten frame gaps (to avoid allocating more than 16 new frames)
appears to be wrong by inspection. It does not handle prev_frame_num > frame_num,
and "h->frame_num - h->sps.ref_frame_count - 1" should be "h->frame_num - h->sps.ref_frame_count".

mpeg4:
- Packed B-frames need to be explicitly split up
when frame threading is on. It's not very fast
without this.
- The buffer age optimization is disabled due to
the way buffers are allocated across threads. The
branch 'fix_buffer_age' has an attempt to fix it
which breaks ffplay.
- Support interlaced.

mpeg1/2:
- Seeking always prints "first frame not a keyframe"
with threads on. Currently disabled for this reason.

-- Prove correct

- decode_update_progress() in h264.c
race_checking branch has some work on h264,
but not that function. It might be worth putting
the branch under #ifdef DEBUG in mainline, but
the code would have to be cleaner.
- MPV_lowest_referenced_row() and co in mpegvideo.c
- Same in vp3.

-- Optimization

- Merge h264 decode_update_progress() with loop_filter().
Add CODEC_CAP_DRAW_HORIZ_BAND as a side effect.
- EMU_EDGE is always set for h264 PAFF+MT
because draw_edges() writes into the other field's
thread's pixels. Needs an option to skip T/B fields.
- Check update_thread_context() functions and make
sure they only copy what they need to.
- Try some more optimization of the "ref < 48; ref++"
loop in h264.c await_references(), try turning the list0/list1 check
above into a loop without being slower.
- Support frame+slice threading at the same time
by assigning slice_count threads for frame threads
to use with execute(). This is simpler but unbalanced
if only one frame thread uses any.

-- Features

- Support streams with width/height changing. This
requires flushing all current frames (and buffering
the input in the meantime), closing the codec and
reopening it. Or don't support it.
- Support encoding. Might need more threading primitives
for good ratecontrol; would be nice for audio and libavfilter too.
- Async decoding part 1: instead of trying to
start every thread at the beginning, return a picture
if the earliest thread is already done, but don't wait
for it. Not sure what effect this would have.
- Part 2: have an API that doesn't wait for the decoding
thread, only returns EAGAIN if it's not ready. What will
it do with the next input packet if it returns that?
- Have an API that returns finished pictures but doesn't
require sending new ones. Maybe allow NULL avpkt when
not at the end of the stream.

-- Samples

http://astrange.ithinksw.net/ffmpeg/mt-samples/

See yuvcmp.c in this directory to compare decoded samples.

For debugging, try commenting out ff_thread_finish_setup calls so
that only one thread runs at once, and then binary search+
scatter printfs to look for differences in codec contexts.