zmbvenc allocates a buffer for a picture with padding on all four sides:
The stride is choosen so large that it already contains padding on the
right; the height also includes padding rows. The padding on the right
of each row is also reused as padding for the left of the next row. So
one still needs to add padding on the left for the first row. This is done
by offsetting the actual pointer used to access the picture from the
pointer returned by av_mallocz and the formula for this offset was
wrong, because it ignored that a pixel can take more than one byte when
calculating the offset resulting from the left padding of the first row.
This fixes accesses outside of the allocated buffer that were reported
in tickets #7980 and #7994. No writes were ever attempted outside of
the buffer.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Tomas Härdin <tjoppen@acc.umu.se>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes the following ubsan errors with the sample from ticket #7980:
libavcodec/zmbvenc.c:243:29: runtime error: left shift of negative value -4
libavcodec/zmbvenc.c:244:28: runtime error: left shift of negative value -2
This consists mostly of the following changes:
- add newly supported pixel formats (RGB555LE, RGB565LE, BGR0)
- select the ZMBV format (c->fmt) and bytes per pixel (c->bypp) based on
avctx->pix_fmt
- multiply widths/x-values by c->bypp, in places where bytes, not pixels, are
expected
- disable palette-writing code for non-palette pix_fmts
- make a note about histogram[]'s datatype (it could need increasing if
ZMBV_BLOCK is increased)
- adjust the c->score_tab length to take up to (and including) 4 times the
number of pixels in a block
- initialise c->score_tab up to c->bypp * the number of pixels
Note: the ZmbvFormat enum allows for additional bit depths:
- 1,2,4-bit (palette)
- 24-bit (RGB)
At time of writing the specifics of these (e.g. channel order, bit alignment)
are not currently defined, and DOSBox only implements support for 8/15/16/32
bpp.
One might expect the 24-bit format - if implemented - to be BGR24, to have the
same channel order as BGR0.
However, the decoder in zmbv.c has been guessed to use RGB24, so I have chosen
to not contradict this, and omitted specific support for this format.
- Clamp ME range to -64..63 (prevents corruption when me_range is too high)
- Allow MV's up to *and including* the positive range limit
- Allow out-of-edge ME by padding the prev buffer with a border of 0's
- Try previous MV before checking the rest (improves speed in some cases)
- More robust logic in code - ensure *mx,*my,*xored are updated together
- Improve block choices by counting 0-bytes in the entropy score
- Make histogram use uint16_t type, to allow byte counts from 16*16
(current block size) up to 255*255 (maximum allowed 8bpp block size)
- Make sure score table is big enough for a full block's worth of bytes
- Calculate *xored without using code in inner loop
This is likely more precise and conveys the intent better.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
This parameter can be used to inform the allocation code about how much
downsizing might occur, and can be used to optimize how to allocate the
packet
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The rationale is that coded_frame was only used to communicate key_frame,
pict_type and quality to the caller, as well as a few other random fields,
in a non predictable, let alone consistent way.
There was agreement that there was no use case for coded_frame, as it is
a full-sized AVFrame container used for just 2-3 int-sized properties,
which shouldn't even belong into the AVCodecContext in the first place.
The appropriate AVPacket flag can be used instead of key_frame, while
quality is exported with the new AVPacketSideData quality factor.
There is no replacement for the other fields as they were unreliable,
mishandled or just not used at all.
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
Allocating coded_frame is what most encoders do anyway, so it makes
sense to always allocate and free it in a single place. Moreover a lot
of encoders freed the frame with av_freep() instead of the correct API
av_frame_free().
This bring uniformity to encoder behaviour and prevents applications
from erroneusly accessing this field when not allocated. Additionally
this helps isolating encoders that export information with coded_frame,
and heavily simplifies its deprecation.
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
* commit 'a837c4f2df96a30bf9aa4115b426d608487c7101':
zmbvenc: use the AVFrame API properly.
flicvideo: use the AVFrame API properly.
smacker: use the AVFrame API properly.
mmvideo: use the AVFrame API properly.
Conflicts:
libavcodec/flicvideo.c
libavcodec/mmvideo.c
libavcodec/smacker.c
libavcodec/zmbvenc.c
See: 76e27b1d0594199b4b1ff8520312069f42373944
See: 099e57bc38d7e53cf6823dfec349ff9fdaee99ba
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master:
Fix even more missing includes after the common.h removal
build: Factor out rangecoder dependencies to CONFIG_RANGECODER
build: Factor out error resilience dependencies to CONFIG_ERROR_RESILIENCE
x86: avcodec: Consistently name all init files
Add more missing includes after removing the implicit common.h
Add some more missing includes after removing the implicit common.h
Don't include common.h from avutil.h
rtmp: Automatically compute the hash for SWFVerification
Conflicts:
configure
doc/APIchanges
doc/examples/decoding_encoding.c
libavcodec/Makefile
libavcodec/assdec.c
libavcodec/audio_frame_queue.c
libavcodec/avpacket.c
libavcodec/dv_profile.c
libavcodec/dwt.c
libavcodec/libtheoraenc.c
libavcodec/rawdec.c
libavcodec/rv40dsp.c
libavcodec/tiff.c
libavcodec/tiffenc.c
libavcodec/v210dec.h
libavcodec/vc1dsp.c
libavcodec/x86/Makefile
libavfilter/asrc_anullsrc.c
libavfilter/avfilter.c
libavfilter/buffer.c
libavfilter/formats.c
libavfilter/vf_ass.c
libavfilter/vf_drawtext.c
libavfilter/vf_fade.c
libavfilter/vf_select.c
libavfilter/video.c
libavfilter/vsrc_testsrc.c
libavformat/version.h
libavutil/audioconvert.c
libavutil/error.h
libavutil/version.h
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master:
rtpdec_asf: Set the no_resync_search option for the chained asf demuxer
asfdec: Add an option for not searching for the packet markers
cosmetics: Clean up the tiffenc pix_fmts declaration to match the style of others
cosmetics: Align codec declarations
cosmetics: Convert mimic.c to utf-8
avconv: remove an unused function parameter.
avconv: remove now pointless variables.
avconv: drop support for building without libavfilter.
nellymoserenc: fix crash due to memsetting the wrong area.
libavformat: Only require first packet to be known for audio/video streams
avplay: Don't try to scale timestamps if the tb isn't set
Conflicts:
Changelog
configure
ffmpeg.c
libavcodec/aacenc.c
libavcodec/bmpenc.c
libavcodec/dnxhddec.c
libavcodec/dnxhdenc.c
libavcodec/ffv1.c
libavcodec/flacenc.c
libavcodec/fraps.c
libavcodec/huffyuv.c
libavcodec/libopenjpegdec.c
libavcodec/mpeg12enc.c
libavcodec/mpeg4videodec.c
libavcodec/pamenc.c
libavcodec/pgssubdec.c
libavcodec/pngenc.c
libavcodec/qtrleenc.c
libavcodec/rawdec.c
libavcodec/sgienc.c
libavcodec/tiffenc.c
libavcodec/v210dec.c
libavcodec/wmv2dec.c
libavformat/utils.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
Also break some long lines, remove codec function placeholder comments
and add spaces in sample/pixel format lists.
Signed-off-by: Martin Storsjö <martin@martin.st>
* qatar/master: (29 commits)
amrwb: remove duplicate arguments from extrapolate_isf().
amrwb: error out early if mode is invalid.
h264: change underread for 10bit QPEL to overread.
matroska: check buffer size for RM-style byte reordering.
vp8: disable mmx functions with sse/sse2 counterparts on x86-64.
vp8: change int stride to ptrdiff_t stride.
wma: fix invalid buffer size assumptions causing random overreads.
Windows Media Audio Lossless decoder
rv10/20: Fix slice overflow with checked bitstream reader.
h263dec: Disallow width/height changing with frame threads.
rv10/20: Fix a buffer overread caused by losing track of the remaining buffer size.
rmdec: Honor .RMF tag size rather than assuming 18.
g722: Fix the QMF scaling
r3d: don't set codec timebase.
electronicarts: set timebase for tgv video.
electronicarts: parse the framerate for cmv video.
ogg: don't set codec timebase
electronicarts: don't set codec timebase
avs: don't set codec timebase
wavpack: Fix an integer overflow
...
Conflicts:
libavcodec/arm/vp8dsp_init_arm.c
libavcodec/fraps.c
libavcodec/h264.c
libavcodec/mpeg4videodec.c
libavcodec/mpegvideo.c
libavcodec/msmpeg4.c
libavcodec/pnmdec.c
libavcodec/qpeg.c
libavcodec/rawenc.c
libavcodec/ulti.c
libavcodec/vcr1.c
libavcodec/version.h
libavcodec/wmalosslessdec.c
libavformat/electronicarts.c
libswscale/ppc/yuv2rgb_altivec.c
tests/ref/acodec/g722
tests/ref/fate/ea-cmv
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master: (36 commits)
adpcmenc: Use correct frame_size for Yamaha ADPCM.
avcodec: add ff_samples_to_time_base() convenience function to internal.h
adx parser: set duration
mlp parser: set duration instead of frame_size
gsm parser: set duration
mpegaudio parser: set duration instead of frame_size
(e)ac3 parser: set duration instead of frame_size
flac parser: set duration instead of frame_size
avcodec: add duration field to AVCodecParserContext
avutil: add av_rescale_q_rnd() to allow different rounding
pnmdec: remove useless .pix_fmts
libmp3lame: support float and s32 sample formats
libmp3lame: renaming, rearrangement, alignment, and comments
libmp3lame: use the LAME default bit rate
libmp3lame: use avpriv_mpegaudio_decode_header() for output frame parsing
libmp3lame: cosmetics: remove some pointless comments
libmp3lame: convert some debugging code to av_dlog()
libmp3lame: remove outdated comment.
libmp3lame: do not set coded_frame->key_frame.
libmp3lame: improve error handling in MP3lame_encode_init()
...
Conflicts:
doc/APIchanges
libavcodec/libmp3lame.c
libavcodec/pcxenc.c
libavcodec/pnmdec.c
libavcodec/pnmenc.c
libavcodec/sgienc.c
libavcodec/utils.c
libavformat/hls.c
libavutil/avutil.h
libswscale/x86/swscale_mmx.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master: (22 commits)
frwu: Employ more meaningful return values.
fraps: Use av_fast_padded_malloc() instead of av_realloc()
mjpegdec: use av_fast_padded_malloc()
eatqi: use av_fast_padded_malloc()
asv1: use av_fast_padded_malloc()
avcodec: Add av_fast_padded_malloc().
swscale: enable dithering in MMX functions.
swscale: make rgb24 function macros slightly smaller.
avcodec.h: Remove some disabled cruft.
swscale: remove obsolete comment.
swscale-test: Drop unused argc and argv arguments from main().
zmbv: Employ more meaningful return values.
zmbvenc: Employ more meaningful return values.
vc1: prevent null pointer dereference on broken files
zmbv: check av_realloc() return values and avoid memleaks on ENOMEM
truespeech: align buffer
ac3: Do not read past the end of ff_ac3_band_start_tab.
dv: Fix small stack overread related to CVE-2011-3929 and CVE-2011-3936.
dv: Fix null pointer dereference due to ach=0
dv: check stype
...
Conflicts:
doc/APIchanges
libavcodec/asv1.c
libavcodec/avcodec.h
libavcodec/eatqi.c
libavcodec/fraps.c
libavcodec/frwu.c
libavcodec/zmbv.c
libavformat/dv.c
libswscale/swscale.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master: (29 commits)
cabac: Move code only used within the CABAC test program into the test program.
vp56: Drop unnecessary cabac.h #include.
h264-test: Initialize AVCodecContext.av_class.
build: Skip compiling network.h and rtsp.h if networking is not enabled.
cosmetics: drop some pointless parentheses
Disable annoying warning without changing behavior
faq: Solutions for common problems with sample paths when running FATE.
avcodec: attempt to clarify the CODEC_CAP_DELAY documentation
avcodec: fix avcodec_encode_audio() documentation.
FATE: xmv-demux test; exercise the XMV demuxer without decoding the perceptual codecs inside.
vqf: recognize more metadata chunks
FATE test: BMV demuxer and associated video and audio decoders.
FATE: indeo4 video decoder test.
FATE: update xxan-wc4 test to a sample with more code coverage.
Change the recent h264_mp4toannexb bitstream filter test to output to an elementary stream rather than a program stream.
g722enc: validate AVCodecContext.trellis
g722enc: set frame_size, and also handle an odd number of input samples
g722enc: split encoding into separate functions for trellis vs. no trellis
mpegaudiodec: Use clearer pointer math
tta: Fix returned error code at EOF
...
Conflicts:
libavcodec/h264.c
libavcodec/indeo3.c
libavcodec/interplayvideo.c
libavcodec/ivi_common.c
libavcodec/libxvidff.c
libavcodec/mpegvideo.c
libavcodec/ppc/mpegvideo_altivec.c
libavcodec/tta.c
libavcodec/utils.c
libavfilter/vsrc_buffer.c
libavformat/Makefile
tests/fate/indeo.mak
tests/ref/acodec/g722
Merged-by: Michael Niedermayer <michaelni@gmx.at>
* qatar/master:
mxfdec: Include FF_INPUT_BUFFER_PADDING_SIZE when allocating extradata.
H.264: tweak some other x86 asm for Atom
probe: Fix insane flow control.
mpegts: remove invalid error check
s302m: use nondeprecated audio sample format API
lavc: use designated initialisers for all codecs.
x86: cabac: add operand size suffixes missing from 6c32576
Conflicts:
libavcodec/ac3enc_float.c
libavcodec/flacenc.c
libavcodec/frwu.c
libavcodec/pictordec.c
libavcodec/qtrleenc.c
libavcodec/v210enc.c
libavcodec/wmv2dec.c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
None of these symbols should be accessed directly, so declare them as
hidden.
Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit d36beb3f6902b1217beda576aa18abf7eb72b03c)