If we fill with black then the generated palette will have one color more
than what the user requested. This also resulted in unwanted black specks in
the output of paletteuse, especially when generating small palettes.
FFDIFFSIGN was created explicitly for this purpose, since the common
return a - b idiom is unsafe regarding overflow on signed integers. It
optimizes to branchless code on common compilers.
FFDIFFSIGN also has the subjective benefit of being easier to read due
to lack of ternary operators.
Tested with FATE.
Things not covered by this are unsigned integers, for which overflows
are well defined, and also places where overflow is clearly impossible,
e.g an instance where the a - b was being done on 24 bit values.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Reviewed-by: Clément Bœsch <u@pkh.me>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
Many of the functions from avfilter/formats can return errors, usually AVERROR(ENOMEM).
This propagates the return values.
All of these were found by using av_warn_unused_result, demonstrating its utility.
Tested with FATE. I am least sure of the changes to avfilter/filtergraph,
since I don't know what/how reduce_format is intended to behave and how it should
react to errors.
Fixes: CID 1325680, 1325679, 1325678.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Previous version Reviewed-by: Nicolas George <george@nsup.org>
Previous version Reviewed-by: Clément Bœsch <u@pkh.me>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
This makes the sorting of the colors along an axis (r, g or b)
predictible, and thus testable under FATE. The performance is not really
an issue here since the function is called only once at the end and will
need to sort very small number of entries, so an alternative would be to
make the sorting functions (see DECLARE_CMP_FUNC()) fallback on another
axis in case of equality. This approach was actually simpler.
I don't know if there is any advantage in using a multidimensional sort,
but it will affect the final palette one way or another.
