Store XMM6 and XMM7 in the shadow space in functions that
clobbers them. This way we don't have to adjust the stack
pointer as often, reducing the number of instructions as
well as code size.
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
This fixes reads out of bounds.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
We read 2 bytes for 15 out of 16 pixels, therefore we need to
have at least 30 bytes, not 16.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
Decoded data is always written in pairs within this decoder.
This fixes writes out of bounds.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
Previously, we returned 0, meaning successful decoding but 0
bytes consumed, leading to an infinite loop.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
If the input buffer contains more data after the deflate stream,
the loop previously left running infinitely, with inflate returning
Z_STREAM_END.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
In lpc_prediction(), we write up to array element 'lpc_order' in
an array allocated to hold 'max_samples_per_frame' elements.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
This fixes breakage in a few fate tests on certain setups
(that for some reason didn't break on OS X) after the previous
commit (8812a8057). Currently, some video streams are initialized
in ff_MPV_common_init with width/height set at 0 and only changed
to a proper video size with ff_MPV_common_frame_size_change later.
The breakage was diagnosed by Anton Khirnov.
Signed-off-by: Martin Storsjö <martin@martin.st>
The actual usefulness of the hack is not known, and it does cause
infinite loops with some broken input files.
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
Previously, s->context_initialized was left set to 1
if ff_vc1_decode_init_alloc_tables failed, skipping the
initialization completely on the next decode call.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
If the height is zero, the decompression will probably end up
failing due to not fitting into the allocated buffer later
anyway, so this doesn't need any more elaborate check.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
This is similar to the fix in 35cbc98b.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
The init function reads one byte of extradata.
Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
