Fixes: signed integer overflow: 50176 * 262144 cannot be represented in type 'int'
Fixes: 18629/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5182370286403584
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The intermediates are required to fit in 12bit (8.1.3.9 Coefficient Scaling)
See SMPTE 421M-2006 and Amendment 1-2007
Fixes: signed integer overflow: -20691 * 262144 cannot be represented in type 'int'
Fixes: 18479/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1_fuzzer-5128912371187712
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: signed integer overflow: 62220 * 262144 cannot be represented in type 'int'
Fixes: 17145/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5667394743173120
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: left shift of negative value -6
Fixes: 17810/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1_fuzzer-5638541240958976
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: left shift of negative value -7
Fixes: 16959/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3_fuzzer-5200360825683968
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: left shift of negative value -1
Fixes: 16424/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3_fuzzer-5656579055026176
Fixes: 16358/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5714436358144000
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array read
Fixes: 16331/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3IMAGE_fuzzer-5672735195267072
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: left shift of negative value -9
Fixes: 15299/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MSS2_fuzzer-5660922678345728
Fixes: 15557/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5673351911047168
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: left shift of negative value -5
Fixes: 15294/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1_fuzzer-5733921754447872
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: index -1 out of bounds for type 'const uint8_t [185][2]'
Fixes: 15720/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MSS2_fuzzer-5666071933091840
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: signed integer overflow: 32796 * 65536 cannot be represented in type 'int'
Fixes: 15430/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_VC1IMAGE_fuzzer-5735424087031808
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: index -1 out of bounds for type 'const uint8_t [185][2]'
Fixes: 15250/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMV3IMAGE_fuzzer-5648992869810176
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: ffmpeg_bof_4.avi
Fixes: ffmpeg_bof_5.avi
Fixes: ffmpeg_bof_6.avi
Found-by: Thuan Pham, Marcel Böhme, Andrew Santosa and Alexandru Razvan Caciulescu with AFLSmart
Reviewed-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The overlap filter needs to cover a full macroblock vertical edge when the
FIELDTX value for two neighbouring macroblocks is not equal. By changing
the internal ordering of the blocks from row major to column major, we do
not need to reinterlace a FIELDTX coded macroblock before running the overlap
filter.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
Simple and Main Profile also need unsigned put_pixels_clamped. Add an argument
to choose between signed and unsigned put_pixels and change function name to
vc1_put_blocks_clamped.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
This reverts commit 2065317db6dc3a219f8ed2de427fe7e19e02eb68.
I committed a wrong version of this patch.
Also reverts the follow-up commits 77a3dfb3 and 8331e591.
Simple and Main profile also need unsigned put_pixels_clamped. Add an argument
to choose between signed and unsigned put_pixels and change function name to
vc1_put_blocks_clamped.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
Direct prediction for interlace frame B pictures references the mv in the
second block in an MB in the backward reference frame for the twomv case.
When the backward reference frame is an I frame, this value may be unset.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
For interlace field pictures s->mb_height indicates the height of the full
picture in MBs, i.e. the two fields combined. A single field is half this
size. When calculating mquant for interlace field pictures, the bottom edge
is the last MB row of the field.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
DIRECTBIT was decoded before the intra/inter MB branching when decoding
interlace frame B pictures. Resulting in mistakenly also decoding it for intra
MBs where this syntax element is not present.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
In vc1_decode_i_blocks_adv mquant needs to be reset to its default value for
each macroblock, instead of once at the beginning of the slice.
DQPROFILE specifies which macroblocks can have an alternative quantizer step
size. When DQPROFILE specifies edges, the selection is applicable to the edges
of the picture. Slice edges are not selected by DQPROFILE.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
HALFQP should only be added to the inverse quantizer when the block is
coded with PQUANT. When PQUANT is equal to ALTPQUANT, the original test
for the addition of HALFQP fails. A negative value for mquant indicates
that the value was derived from VOPDQUANT.
Fixes#4372
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
HALFQP should only be added to the inverse quantizer when the block is
coded with PQUANT. See 8.1.3.8 in VC-1 spec.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
Add previously omitted overlap smooting and loop filtering for
frame/field-interlace pictures. For progressive pictures switch to the
re-implemented versions of overlap smooting and loop filtering.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
The existing implementation only used vc1_put_signed_blocks_clamped for I and
BI frames. This rewritten version is also applicable to P frame both
progressive and frame/field-interlace.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
The new overlap smooting filter smoothes image pixels stored in v->block.
Switch to v->block instead of s->block for storing decoded image pixels for P
frames. Additionally, we must take incrementing *_blk_idx out of the
vc1_put_signed_blocks_clamped function.
Signed-off-by: Jerome Borsboom <jerome.borsboom@carpalis.nl>
also fixes: runtime error: index 1456 out of bounds for type 'int16_t [16]'
Found-by: durandal_1707
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
* commit 'c2084ffcbfc11d1b6ed3a4a0df9cafd56fbb896f':
intrax8: Use the generic horizband function
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
* commit '8072345e9f86d88fbc4a15c17cb03f1e4701c9a5':
intrax8: Keep a reference to the GetBitContext reader
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
* commit 'eaeba6f241e0de0e797be10f8fda967ef8489e64':
intrax8: Pass the output frame to the decoding function
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
* commit '577393321c389ad2973bec6168a8045c94a9e099':
intrax8: Carry over the loopfilter value in ff_intrax8_decode_picture
Merged-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
The intrax8 decoding process does not imply any kind of error
resilience, and the only call present is more related to how mpegvideo
works rather than anything else.
Therefore have the parent decoders carry out er when actually needed.
* commit '1542ec96389f32e5081c6c607e4b6f5e257ccdf2':
cosmetics: Drop spurious spaces from if clauses
Conflicts:
libavcodec/vc1_block.c
Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
