According to the spec, the reference list for a slice should be
constructed by first generating an initial (what we now call "default")
reference list and then optionally applying modifications to it.
Our code has an optimization where the initial reference list is
constructed for the first inter slice and then rebuilt for other slices
if needed. This, however, adds complexity to the code, requires an extra
2.5kB array in the codec context and there is no reason to think that it
has any positive effect on performance. Therefore, simplify the code by
generating the reference list from scratch for each slice.
Fixes out of array access
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2049_f2192b6829ab6e0eefcb035329c03c60.264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
In case of bitstream errors the deblock filter and slices can access uninitialized
top_borders from previous slices which did not fill them as they stoped halfway due
to error or where entirely missing.
This also makes code using these tables deterministic in case of missing or damaged
slices
Found-by: Tyson Smith
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The AVCodecContext.get_format callback is not only used for pixel format
negotiation with the API user, but also for hwaccel init. For the
latter, it's required that some codec parameters, in particular the
codec profile, are set when the callback is invoked.
This patch removes a get_format invocation where this is not guaranteed.
The codec parameters, including the profile, are really set further
below. (The same code path that sets the profile also calls get_format
properly too.)
This just happened to work by coincidence in most cases. For example, if
the API user just copied or reused the AVStream's AVCodecContext when
decoding, the profile would be set properly. But in some cases it
fails., such as with the sample WolfensteinTwitch.mp4 on the samples
server.
Remove the redundant get_format call. Apparently it serves no purpose
anymore, although it is possible that this was different at the time it
was added in commit ffd77f94a2.
This fixes hwaccel usage for API users which do not set the profile
when setting up the AVCodecContext (which is allowed).
Deprecate older VA pixel formats (MOCO, IDCT) as it is now very unlikely
to ever be useful in the future. Only keep plain AV_PIX_FMT_VAAPI format
that is aliased to the older VLD variant.
This is an API change.
Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne@intel.com>
At least the new videotoolbox decoder does not actually set a frame if
end_frame fails. This causes the API to return success and signals that
a picture was decoded, even though AVFrame->data[0] is NULL.
Fix this by propagating end_frame errors.
Fixes out of array read
Fixes: asan_heap-oob_394322e_138_cov_4265020547_CVPCMNL1_SVA_C.264
Found-by: Samuel Groß, Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
* commit '3e3056f2a020dd77efdf379dbd4c06a65b4a499a':
h264: Allow stream and container cropping at the same time
Conflicts:
libavcodec/h264_slice.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
The container cropping is applied only when difference is within 16
pixels, and the smallest value between the two is chosen.
Bug-Id: 383
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
Fixes out of array access
Fixes: asan_heap-oob_4d5bb0_682_cov_3124593265_Fraunhofer__a_driving_force_in_innovation__small.mp4
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
* commit '5ec0bdf2c524224f30ba4786f47324970aed4aaa':
h264: do not update the context fields copied between threads after finish_setup()
Conflicts:
libavcodec/h264.h
libavcodec/h264_slice.c
See: f111831ed6 and others
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* commit 'e49e0f58e273237f83b4486f3536931ed1943d18':
h264: make sure the slices do not overlap during slice threading
Conflicts:
libavcodec/h264.h
libavcodec/h264_slice.c
See: 43b434210e
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Also use the frame pixel format instead of the one from the codec
context, which is more robust.
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Also use the frame pixel format instead of the one from the codec
context, which is more robust.
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Inconsistencies between the dimensions/pixel format of avctx and the
frame can confuse API users.
For example this can crash the demuxing_decoding example.
Back up the previous values and restore them, when decoding the next
frame. This is necessary, because these can be different between the
returned frame and the last decoded frame.
Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
The AVFrame values are closer to the AVFrame bitmap changed instead of
the AVCodecContext values, so this should be more robust
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
* commit '0181ae9af2de1526464d23209b82e6674d362f5d':
h264: Make sure reinit failures mark the context as not initialized
Conflicts:
libavcodec/h264_slice.c
See: e8714f6f93
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* commit '7a4f74eed51f914e9bbfebaffd4a92ac6791f819':
h264: embed the DPB in the context
Conflicts:
libavcodec/h264.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* commit '88c612e3a4d5b584e2d9f6e2541de78d67bdfb9f':
h264: merge the init and reinit paths in update_thread_context()
Conflicts:
libavcodec/h264_slice.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* commit '8a66fd40260b7aae6226d68c4dbad43b05a8e524':
h264: drop the reinit parameter from init_slice_header()
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* commit 'c9ccbc7333eddd025ebbde5cc4f27d68a950c623':
h264: update the current bit depth/chroma type in init_slice_header()
Merged-by: Michael Niedermayer <michaelni@gmx.at>
