The code above this does a whitelist on desc->flags, which now includes
the (disallowed) AV_PIX_FMT_FLAG_XYZ for XYZ formats. So there is no
more need for a separate check, here.
There are already several places in the codebase that match desc->name
against "xyz", and many downstream clients replicate this behavior.
I have no idea why this is not just a flag.
Motivated by my desire to add yet another check for XYZ to the codebase,
and I'd rather not keep copy/pasting a string comparison hack.
These are not supported by the drawing functions at all, and were
incorrectly advertised as supported in the past.
Note: This check is added only to separate the logic change from the API
change in the following commit, and will be removed again after it
becomes redundant.
This logic only covers the case of yuv420p. Extend this logic to cover
*all* vertically subsampled YUV formats, which require the same
interlaced scaling logic.
Fortunately, we can get away with re-using the same code for both JPEG
and MPEG range YUV, because the only difference here is the horizontal
alignment. (Which I omit touching for now, to avoid introducing possibly
unintended changes in default behavior)
Buggy ICCv4 profiles are unfortunately used in the wild, and it's quite
easy to work around them by just forcing the white point to the correct
value. Display a warning just in case.
See-Also: https://trac.ffmpeg.org/ticket/9673
This is mathematically equivalent to what we were doing before, but
gives subtly different results due to rounding (rows first vs columns
first). Doing it this way makes our film grain database generation match
reference implementation and now produces bit-exact outputs in my
testing.
Rename the transposed variables to be a bit less confusing.
The spec specified indices in the order [x][y], but our code follows the
traditional C convention of [y][x]. This was not correctly account for
when calculating the base index of the grain database access.
The spec specifies x^31 + x^3 + 1 as the polynomial, but the diagram in
Figure 1-1 omits the +1 offset. The initial implementation was based on
the diagram, but this is wrong (produces subtly incorrect results).
Can be used to configure libplacebo's underlying raw options, which
sometimes includes new or advanced / in-depth settings not (yet) exposed
by vf_libplacebo.
This new upstream struct simplifies params struct management by allowing
them to all be contained in a single dynamically allocated struct. This
commit switches to the new API in a backwards-compatible way.
The only nontrivial change that was required was to handle
`sigmoid_params` in a way consistent with the rest of the params
structs, instead of setting it directly to the upstream default.
Requires a new upstream function to test not for *import* support on a
given output pixel format, but also whether we can render to it.
Fixes: https://github.com/haasn/libplacebo/issues/173
The old logic was trying to be excessively clever in "deducing" that the
user wanted to stretch/scale the image when ow/oh differed from iw/ih
aspect ratio. But this is almost surely unintended except in
pathological cases, and in those cases users should simply disable
normalize_sar and do all the stretching/scaling logic themselves. This
is especially important in multi-input mode, where the canvas may be
vastly different from the input dimensions of any stream. Also, passing
through input 0 SAR in multi-input mode is arbitrary and nearly useless,
so again force output SAR to 1:1 here.
Use the gcd of all input timebases to ensure PTS accuracy. For the
framerate, just pick the highest of all the inputs, under the assumption
that we will render frames with approximately this frequency. Of course,
this is not 100% accurate, in particular if the input frames are badly
misaligned. But this field is informational to begin with.
Importantly, it covers the "common" case of combining high FPS and low
FPS streams with aligned frames.
In the event that some frame mixes are OK while others are not, the
priority goes:
1. Errors in updating any frame -> return error
2. Any input incomplete -> request frames and return
3. Any inputs OK -> ignore EOF streams and render remaining inputs
4. No inputs OK -> set output to most recent status
This logic ensures that we can continue rendering the remaining streams,
no matter which streams reach their end of life, until we have no
streams left at which point we forward the last EOF.
When combining multiple inputs, the output PTS may be less than the PTS
of the input. In this case, the current's code assumption of always
draining one value from the FIFO is incorrect. Replace by a smarter
function which drains only those PTS values that were actually consumed.
When combining multiple inputs with different PTS and durations, in
input-timed mode, we emit one output frame for every input frame PTS,
from *any* input. So when combining a low FPS stream with a high FPS
stream, the output framerate would match the higher FPS, independent of
which order they are specified in.
Subsequent inputs require frame blending to be enabled, in order to not
overwrite the existing frame contents.
For output metadata, we implicitly copy the metadata of the *first*
available stream (falling back to the second stream if the first has
already reached EOF, and so on). This is done to resolve any conflicts
between inputs with differing metadata. So when e.g. input 1 is HDR and
output 2 is SDR, the output will be HDR, and vice versa. This logic
could probablly be improved by dynamically determining some "superior"
set of metadata, but I don't want to handle that complexity in this
series.
Instead of finding the ref frame in output_frame() and then passing its
signature to update_crops(), pull out the logic and invoke it a second
time inside update_crops().
This may seem wasteful at present, but will actually become required in
the future, since update_crops() runs on *every* input, and needs values
specific to that input (which the signature isn't), while output_frame()
is only interested in a single input. It's much easier to just split the
logic cleanly.
Including the queue status, because these will need to be re-queried
inside output_frame_mix when that function is refactored to handle
multiple inputs.
In anticipation of a refactor which will enable multiple input support.
Note: the renderer is also input-specific because it maintains a frame
cache, HDR peak detection state and mixing cache, all of which are tied
to a specific input stream.
If the input queue is EOF, then the s->status check should already have
covered it, and prevented the code from getting this far.
If we still hit this case for some reason, it's probably a bug. Better
to hit the AVERROR_BUG branch.
Importing Vulkan device on older versions no longer works due to the
lavu vulkan API changes (specifically, the switch to planar textures by
default). Additionally, importing on versions that don't suppirt
lock/unlock_queue is unsafe with the advent of the threaded vulkan
hwaccel. As a plus, saves us some annoying #ifdef boilerplate.
I will raise the minimum vf_libplacebo version globally on the next
stable release of libplacebo, and remove all of these checks.
mix->timestamps is expressed relative to the source timebase, which is
possibly a different timescale from `base_pts`. We can't mix-and-match
here. The only reason this worked in my previous testing was because I
was testing on a source file which had an exactly matching timebase.
Fix it by always using the exact PTS as tagged on the AVFrame.
This algorithm has once again been refactored, this time leading to a
dropping of the old `tone_mapping_mode` field, to be replaced by a
single tunable hybrid mode with configurable strength.
We can approximately map the old modes onto the new API for backwards
compatibility. Replace deprecated enums by their integer equivalents to
safely preserve this API until the next bump.
Upstream deprecated the old ad-hoc, enum/intent-based gamut mapping API
and added a new API based on colorimetrically accurate gamut mapping
functions.
The relevant change for us is the addition of several new modes, as well
as deprecation of the old options. Update the documentation accordingly.
The current code relied on pl_queue eventually returning EOF back to the
caller, which didn't work in all situations (e.g. single frame input).
Also, the current code assumed that ff_inlink_acknowledge_status only
fired once, which was patently not true, as the above edge cases
demonstrated.
Solve both issues by keeping track of the acknowledged link status and
forwarding it (instead of trying to probe the pl_queue again) in the
event that we run out of queued input frames, as well as (in CFR mode)
when we pass the indicated status PTS.
This exposes libplacebo's frame mixing functionality to vf_libplacebo,
by allowing users to specify a desired target fps to output at. Incoming
frames will be smoothly resampled (in a manner determined by the
`frame_mixer` option, to be added in the next commit).
To generate a consistently timed output stream, we directly use the
desired framerate as the timebase, and simply output frames in
sequential order (tracked by the number of frames output so far).
To present compatibility with the current behavior, we keep track of a
FIFO of exact frame timestamps that we want to output to the user. In
practice, this is essentially equivalent to the current filter_frame()
code, but this design allows us to scale to more complicated use cases
in the future - for example, insertion of intermediate frames
(deinterlacing, frame doubling, conversion to fixed fps, ...)
This does not leverage any immediate benefits, but refactors and
prepares the codebase for upcoming changes, which will include the
ability to do deinterlacing and resampling (frame mixing).
This commit contains no functional change. The goal is merely to
separate the highly intertwined `filter_frame` and `process_frames`
functions into their separate concerns, specifically to separate frame
uploading (which is now done directly in `filter_frame`) from emitting a
frame (which is now done by a dedicated function `output_frame`).
The overall idea here is to be able to ultimately call `output_frame`
multiple times, to e.g. emit several output frames for a single input
frame.
Motivated by a desire to use vf_libplacebo as a GPU-accelerated
cropping/padding/zooming filter. This commit adds support for setting
the `input/target.crop` fields as dynamic expressions.
Re-use the same generic variables available to other scale and crop type
filters, and also add some more that we can afford as a result of being
able to set these properties dynamically.
It's worth pointing out that `out_t/ot` is currently redundant with
`in_t/t` since it will always contain the same PTS values, but I plan on
changing this in the near future.
I decided to also expose `crop_w/crop_h` and `pos_w/pos_h` as variables
in the expression parser itself, since this enables the fairly common
use case of determining dimensions first and then placing the image
appropriately, such as is done in the default behavior (which centers
the cropped/placed region by default).
In some circumstances, libplacebo will clear the background as a result
of cropping/padding. Currently, this uses the hard-coded default fill
color of black. This option makes this behavior configurable.
Not doing so is an obvious oversight - the ICC profile is tied to the
original colorspace, so if we change it, we should definitely strip this
information.
We should probably also have an extra option to control whether the ICC
profile should be stripped, ignored, or applied, but for now this fixes
an existing bug.
Some testing revealed this to be a very efficient and reliable method of
ingesting GPU frames into libplacebo, so it's a good idea to give as an
example.
This example being first is now misleading because round-tripping
through hwdownload/hwupload is neither required nor recommended. Also,
the comment about avoiding format conversion is unnecessary because
`libplacebo` will now inherit the input frame format by default.
This has not been functional since a year ago, including in our current
minimum dependency of libplacebo (v4.192.0). It also causes build errors
against libplacebo v6, so it needs to be removed from the code. We can
keep the option around for now, but it should also be removed soon.
Signed-off-by: Niklas Haas <git@haasn.dev>
Signed-off-by: James Almer <jamrial@gmail.com>
We need to construct the output format list separatedly from the input
format list, because we need to adhere to two extra requirements:
1. Big-endian output formats are always unsupported (runtime error)
2. Combining 'vulkan' with an explicit out_format that is not supported
by the vulkan frame allocation code is illegal and will crash (abort)
As a free side benefit, this rewrite fixes a possible memory leak in the
`fail` path that was present in the old code.
Signed-off-by: Niklas Haas <git@haasn.dev>
This filter, when used in the "pad" mode, currently makes the
distinction between limited and full range solely by testing for YUVJ
pixel formats at link setup time. This is deprecated and should be
improved to perform the detection based on the per-frame metadata.
In order to make this distinction based on color range metadata, which
is only known at the time of filtering frames, for simplicity, we simply
allocate two copies of the "black" frame - one for limited range and the
other for full range metadata. This could be done more dynamically (e.g.
as-needed or simply by blitting the appropriate pixel value directly),
but this change is relatively simple and preserves the structure of the
existing code.
This commit actually fixes a bug in FATE - the new output is correct for
the first time. The previous md5 ref was of a frame that incorrectly
combined full-range pixel data with limited-range black fields. The
corresponding result has been updated.
Signed-off-by: Niklas Haas <git@haasn.dev>
This filter currently makes the distinction between limited and full
range by testing for the deprecated YUVJ pixel formats at link setup
time. This is deprecated and should be improved to perform the detection
based on the per-frame metadata.
Rewrite it to calculate the black pixel threshold at the time of
filtering a frame, when metadata about the frame's color range is known.
Doing it this way has the small side benefit of being able to handle
streams with variable metadata, and is not a meaningful performance
cost.
Signed-off-by: Niklas Haas <git@haasn.dev>
Rather than hard-coding AV_PIX_FMT_VULKAN, expand this to the full list
of formats supported by <libplacebo/utils/libav.h>. We re-use the
existing `format` option to allow selecting specific software formats in
addition to specific vulkan hwframe formats.
Some minor changes are necessary to account for the fact that
`ff_vk_filter_config_output` is now only called optionally, the fact
that the output format must now be parsed before `query_format` gets
called, and the fact that we need to call a different function to
retrieve data from the `pl_frame` in the non-hwaccel case.
Signed-off-by: Niklas Haas <git@haasn.dev>
This previous expression multiplied a constant (outlink->h) that was
guaranteed to be 0 at this point, thus making it always a no-op.
Fix the calculation, and also properly reset the SAR to 1:1 as is now
necessary (the failure to do so previously hid this bug's existence).
As a result of a typo in the source code, this option was completely
non-functional. In order to fix it, without breaking the current default
behavior, explicitly change this default to 0.
This behavior is also consistent with how other scale filters behave by
default, so it's probably best to enshrine it anyways.
After commit c0b93, it's possible that `ff_vk_filter_config_input` never
gets called, leading to `s->vkctx.input_format` being left unset. This
broke the format auto-selection logic in `libplacebo_config_output`,
resulting in a default to yuv420p, instead of defaulting to the input
format as intended.
Fixes: c0b93c4f8b
This is done only to the inputs, not the outputs, because we always
output vulkan hwframes.
Doing so also requires keeping track of backing textures for non-hwdec
formats, but is otherwise a mostly straightforward change to the format
query function. Special care needs to be taken to avoid crashing on
older libplacebo due to AV_PIX_FMT_RGBF32LE et al.
Instead of doing it ad-hoc in `filter_frame`. This is not a huge change
on its own, but paves the way for adding support for more formats in the
future, in particular formats other than AV_PIX_FMT_VULKAN.
This tests the new "-flags2 icc_profiles" option by making sure the
embedded ICC profile gets correctly detected as sRGB.
Signed-off-by: Niklas Haas <git@haasn.dev>
Only if requested, and only if the codec signals support for ICC
profiles. Implementation roughly matches the functionality of the
existing vf_iccgen filter, albeit with some reduced flexibility and no
caching.
Ideally, we'd also only do this on the first frame (e.g. mjpeg, apng),
but there's no meaningful way for us to distinguish between this case
and e.g. somebody using the image2 muxer, in which case we'd want to
attach ICC profiles to every frame in the stream.
Closes: #9672
Signed-off-by: Niklas Haas <git@haasn.dev>
Implementation for the decode side of the ICC profile API, roughly
matching the behavior of the existing vf_iccdetect filter.
Closes: #9673
Signed-off-by: Niklas Haas <git@haasn.dev>
Handling this in general code makes more sense than handling it in
individual codec files, because it would be a lot of unnecessary code
duplication for the plenty of formats that support exporting ICC
profiles (jpg, png, tiff, webp, jxl, ...).
encode.c and decode.c will be in charge of initializing this state as
needed, so we merely need to make sure to uninit it afterwards from the
common destructor path.
Signed-off-by: Niklas Haas <git@haasn.dev>
This functionally already exists, but as pointed out in #9672 and #9673,
requiring users to manually include filters is clumsy, error-prone and
hard to use together with tools like ffplay.
To streamline ICC profile support, add a new AVCodecContext flag to
globally enable reading and writing ICC profiles, automatically, for all
appropriate media types.
Note that this commit only includes the new API. The implementation is
split off to separate commits for readability.
Signed-off-by: Niklas Haas <git@haasn.dev>
Codecs that can read/write ICC profiles deserve a special capability so
the common logic in encode.c/decode.c can decide whether or not there
needs to be any special handling for ICC profiles. The motivation here
is to be able to use it to decide whether or not an ICC profile needs to
be generated in the encode path, but it might as well get added to
decoders as well for purely informative reasons.
It's not entirely clear to me whether the "thp" and "smvjpeg" variants
of "mjpeg" should have this capability set or not, given that the code
technically supports it but I somehow doubt these files may contain
them. In either case, this cap is purely informative for decoders so it
doesn't matter too much either way.
It's also not entirely clear whether the "amv" encoder should signal ICC
profile support, but again erring on the side of caution, we probably
*shouldn't* be generating (and encoding!) ICC profiles for this type of
media file.
Signed-off-by: Niklas Haas <git@haasn.dev>
We will need this helper inside libavcodec in the future, so move it
there, leaving behind an #include to the raw source file in its old
location in libvfilter. This approach is inspired by the handling of
vulkan.c, and avoids us needing to expose any of it publicly (or
semi-publicly) in e.g. libavutil, thus avoiding any ABI headaches.
It's debatable whether the actual code belongs in libavcodec or
libavfilter, but I decided to put it into libavcodec because it
conceptually deals with encoding and decoding ICC profiles, and will be
used to decode embedded ICC profiles in image files.
Signed-off-by: Niklas Haas <git@haasn.dev>
GL and Metal cache the state at time of texture creation. GLES2 and
Direct3D 11 use the state at time of the render copy call.
So the only way we can get the correct behavior consistently is by
making sure the state is set for both the upload *and* the draw call.
This probably isn't our bug to fix (upstream should make itself behave
consistently and also document its functions), but as it stands,
`ffplay` is misrendering BT.709 as BT.601 on my stock Linux system, and
that leaves a bad taste in my mouth.
Signed-off-by: Niklas Haas <git@haasn.dev>
This filter is designed to parse embedded ICC profiles and attempt
extracting colorspace tags from them, updating the AVFrame metadata
accordingly.
This is intentionally made a separate filter, rather than being part of
libavcodec itself, so that it's an opt-in behavior for the time being.
This also gives the user more flexibility to e.g. first attach an ICC
profile and then also set the colorspace tags from it.
This makes #9673 possible, though not automatic.
Signed-off-by: Niklas Haas <git@haasn.dev>
This filter is designed to specifically cover the task of generating ICC
profiles (and attaching them to output frames) on demand. Other tasks,
such as ICC profile loading/stripping, or ICC profile application, are
better left to separate filters (or included into e.g. vf_setparams).
Signed-off-by: Niklas Haas <git@haasn.dev>
This introduces an optional dependency on lcms2 into FFmpeg. lcms2 is a
widely used library for ICC profile handling, which apart from being
used in almost all major image processing programs and video players,
has also been deployed in browsers. As such, it's both widely available
and well-tested.
Add a few helpers to cover our major use cases. This commit merely
introduces the helpers (and configure check), even though nothing uses
them yet.
It's worth pointing out that the reason the cmsToneCurves for each
AVCOL_TRC are cached inside the context, is because constructing a
cmsToneCurve requires evaluating the curve at 4096 (by default) grid
points and constructing a LUT. So, we ideally only want to do this once
per curve. This matters for e.g. ff_icc_profile_detect_transfer, which
essentially compares a profile against all of these generated LUTs.
Re-generating the LUTs for every iteration would be unnecessarily
wasteful.
The same consideration does not apply to e.g. cmsCreate*Profile, which
is a very lightweight operation just involving struct allocation and
setting a few pointers.
The cutoff value of 0.01 was determined by experimentation. The lowest
"false positive" delta I saw in practice was 0.13, and the largest
"false negative" delta was 0.0008. So a value of 0.01 sits comfortaby
almost exactly in the middle.
Signed-off-by: Niklas Haas <git@haasn.dev>
Related to #9673, this helper exists to facilitate "guessing" the right
primary tags from a given set of raw primary coefficients.
The cutoff value of 0.001 was chosen by experimentation. The smallest
"false positive" delta observed in practice was 0.023329, while the
largest "false negative" delta was 0.00016. So, a value of 0.001 sits
comfortably in the middle.
Signed-off-by: Niklas Haas <git@haasn.dev>
Niklas Haas