Certain alpha run lengths (for SHQ1/SHQ3/SHQ5) could be stored in
both long and short versions, and we would only accept the short version,
returning -1 (invalid code) for the others. This could cause an
out-of-bounds write on malicious input, as discovered by
Andreas Cadhalpun during fuzzing.
Fix by simply allowing both versions, leaving no invalid codes
in the alpha VLC.
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
This decoder can decode all existing SpeedHQ formats (SHQ0–5, 7, and 9),
including correct decoding of the alpha channel.
1080p is decoded in 142 fps on one core of my i7-4600U (2.1 GHz Haswell),
about evenly split between bitstream reader and IDCT. There is currently
no attempt at slice or frame threading, even though the format trivially
supports both.
NewTek very helpfully provided a full set of SHQ samples, as well as
source code for an SHQ2 encoder (not included) and assistance with
understanding some details of the format.
Use the newly created vlc.h directly instead of including get_bits when needed.
The VLC and RL_VLC_ELEM structures are independent from the bitreader.
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
