mirror of
https://github.com/FFmpeg/FFmpeg.git
synced 2024-12-07 11:13:41 +02:00
289 lines
11 KiB
Plaintext
289 lines
11 KiB
Plaintext
Filter design
|
|
=============
|
|
|
|
This document explains guidelines that should be observed (or ignored with
|
|
good reason) when writing filters for libavfilter.
|
|
|
|
In this document, the word “frame” indicates either a video frame or a group
|
|
of audio samples, as stored in an AVFrame structure.
|
|
|
|
|
|
Format negotiation
|
|
==================
|
|
|
|
The query_formats method should set, for each input and each output links,
|
|
the list of supported formats.
|
|
|
|
For video links, that means pixel format. For audio links, that means
|
|
channel layout, sample format (the sample packing is implied by the sample
|
|
format) and sample rate.
|
|
|
|
The lists are not just lists, they are references to shared objects. When
|
|
the negotiation mechanism computes the intersection of the formats
|
|
supported at each end of a link, all references to both lists are replaced
|
|
with a reference to the intersection. And when a single format is
|
|
eventually chosen for a link amongst the remaining list, again, all
|
|
references to the list are updated.
|
|
|
|
That means that if a filter requires that its input and output have the
|
|
same format amongst a supported list, all it has to do is use a reference
|
|
to the same list of formats.
|
|
|
|
query_formats can leave some formats unset and return AVERROR(EAGAIN) to
|
|
cause the negotiation mechanism to try again later. That can be used by
|
|
filters with complex requirements to use the format negotiated on one link
|
|
to set the formats supported on another.
|
|
|
|
|
|
Frame references ownership and permissions
|
|
==========================================
|
|
|
|
Principle
|
|
---------
|
|
|
|
Audio and video data are voluminous; the frame and frame reference
|
|
mechanism is intended to avoid, as much as possible, expensive copies of
|
|
that data while still allowing the filters to produce correct results.
|
|
|
|
The data is stored in buffers represented by AVFrame structures.
|
|
Several references can point to the same frame buffer; the buffer is
|
|
automatically deallocated once all corresponding references have been
|
|
destroyed.
|
|
|
|
The characteristics of the data (resolution, sample rate, etc.) are
|
|
stored in the reference; different references for the same buffer can
|
|
show different characteristics. In particular, a video reference can
|
|
point to only a part of a video buffer.
|
|
|
|
A reference is usually obtained as input to the filter_frame method or
|
|
requested using the ff_get_video_buffer or ff_get_audio_buffer
|
|
functions. A new reference on an existing buffer can be created with
|
|
av_frame_ref(). A reference is destroyed using
|
|
the av_frame_free() function.
|
|
|
|
Reference ownership
|
|
-------------------
|
|
|
|
At any time, a reference “belongs” to a particular piece of code,
|
|
usually a filter. With a few caveats that will be explained below, only
|
|
that piece of code is allowed to access it. It is also responsible for
|
|
destroying it, although this is sometimes done automatically (see the
|
|
section on link reference fields).
|
|
|
|
Here are the (fairly obvious) rules for reference ownership:
|
|
|
|
* A reference received by the filter_frame method belongs to the
|
|
corresponding filter.
|
|
|
|
* A reference passed to ff_filter_frame is given away and must no longer
|
|
be used.
|
|
|
|
* A reference created with av_frame_ref() belongs to the code that
|
|
created it.
|
|
|
|
* A reference obtained with ff_get_video_buffer or ff_get_audio_buffer
|
|
belongs to the code that requested it.
|
|
|
|
* A reference given as return value by the get_video_buffer or
|
|
get_audio_buffer method is given away and must no longer be used.
|
|
|
|
Link reference fields
|
|
---------------------
|
|
|
|
The AVFilterLink structure has a few AVFrame fields.
|
|
|
|
partial_buf is used by libavfilter internally and must not be accessed
|
|
by filters.
|
|
|
|
fifo contains frames queued in the filter's input. They belong to the
|
|
framework until they are taken by the filter.
|
|
|
|
Reference permissions
|
|
---------------------
|
|
|
|
Since the same frame data can be shared by several frames, modifying may
|
|
have unintended consequences. A frame is considered writable if only one
|
|
reference to it exists. The code owning that reference it then allowed
|
|
to modify the data.
|
|
|
|
A filter can check if a frame is writable by using the
|
|
av_frame_is_writable() function.
|
|
|
|
A filter can ensure that a frame is writable at some point of the code
|
|
by using the ff_inlink_make_frame_writable() function. It will duplicate
|
|
the frame if needed.
|
|
|
|
A filter can ensure that the frame passed to the filter_frame() callback
|
|
is writable by setting the needs_writable flag on the corresponding
|
|
input pad. It does not apply to the activate() callback.
|
|
|
|
|
|
Frame scheduling
|
|
================
|
|
|
|
The purpose of these rules is to ensure that frames flow in the filter
|
|
graph without getting stuck and accumulating somewhere.
|
|
|
|
Simple filters that output one frame for each input frame should not have
|
|
to worry about it.
|
|
|
|
There are two design for filters: one using the filter_frame() and
|
|
request_frame() callbacks and the other using the activate() callback.
|
|
|
|
The design using filter_frame() and request_frame() is legacy, but it is
|
|
suitable for filters that have a single input and process one frame at a
|
|
time. New filters with several inputs, that treat several frames at a time
|
|
or that require a special treatment at EOF should probably use the design
|
|
using activate().
|
|
|
|
activate
|
|
--------
|
|
|
|
This method is called when something must be done in a filter; the
|
|
definition of that "something" depends on the semantic of the filter.
|
|
|
|
The callback must examine the status of the filter's links and proceed
|
|
accordingly.
|
|
|
|
The status of output links is stored in the frame_wanted_out, status_in
|
|
and status_out fields and tested by the ff_outlink_frame_wanted()
|
|
function. If this function returns true, then the processing requires a
|
|
frame on this link and the filter is expected to make efforts in that
|
|
direction.
|
|
|
|
The status of input links is stored by the status_in, fifo and
|
|
status_out fields; they must not be accessed directly. The fifo field
|
|
contains the frames that are queued in the input for processing by the
|
|
filter. The status_in and status_out fields contains the queued status
|
|
(EOF or error) of the link; status_in is a status change that must be
|
|
taken into account after all frames in fifo have been processed;
|
|
status_out is the status that have been taken into account, it is final
|
|
when it is not 0.
|
|
|
|
The typical task of an activate callback is to first check the backward
|
|
status of output links, and if relevant forward it to the corresponding
|
|
input. Then, if relevant, for each input link: test the availability of
|
|
frames in fifo and process them; if no frame is available, test and
|
|
acknowledge a change of status using ff_inlink_acknowledge_status(); and
|
|
forward the result (frame or status change) to the corresponding input.
|
|
If nothing is possible, test the status of outputs and forward it to the
|
|
corresponding input(s). If still not possible, return FFERROR_NOT_READY.
|
|
|
|
If the filters stores internally one or a few frame for some input, it
|
|
can consider them to be part of the FIFO and delay acknowledging a
|
|
status change accordingly.
|
|
|
|
Example code:
|
|
|
|
ret = ff_outlink_get_status(outlink);
|
|
if (ret) {
|
|
ff_inlink_set_status(inlink, ret);
|
|
return 0;
|
|
}
|
|
if (priv->next_frame) {
|
|
/* use it */
|
|
return 0;
|
|
}
|
|
ret = ff_inlink_consume_frame(inlink, &frame);
|
|
if (ret < 0)
|
|
return ret;
|
|
if (ret) {
|
|
/* use it */
|
|
return 0;
|
|
}
|
|
ret = ff_inlink_acknowledge_status(inlink, &status, &pts);
|
|
if (ret) {
|
|
/* flush */
|
|
ff_outlink_set_status(outlink, status, pts);
|
|
return 0;
|
|
}
|
|
if (ff_outlink_frame_wanted(outlink)) {
|
|
ff_inlink_request_frame(inlink);
|
|
return 0;
|
|
}
|
|
return FFERROR_NOT_READY;
|
|
|
|
The exact code depends on how similar the /* use it */ blocks are and
|
|
how related they are to the /* flush */ block, and needs to apply these
|
|
operations to the correct inlink or outlink if there are several.
|
|
|
|
Macros are available to factor that when no extra processing is needed:
|
|
|
|
FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink);
|
|
FF_FILTER_FORWARD_STATUS_ALL(outlink, filter);
|
|
FF_FILTER_FORWARD_STATUS(inlink, outlink);
|
|
FF_FILTER_FORWARD_STATUS_ALL(inlink, filter);
|
|
FF_FILTER_FORWARD_WANTED(outlink, inlink);
|
|
|
|
filter_frame
|
|
------------
|
|
|
|
For filters that do not use the activate() callback, this method is
|
|
called when a frame is pushed to the filter's input. It can be called at
|
|
any time except in a reentrant way.
|
|
|
|
If the input frame is enough to produce output, then the filter should
|
|
push the output frames on the output link immediately.
|
|
|
|
As an exception to the previous rule, if the input frame is enough to
|
|
produce several output frames, then the filter needs output only at
|
|
least one per link. The additional frames can be left buffered in the
|
|
filter; these buffered frames must be flushed immediately if a new input
|
|
produces new output.
|
|
|
|
(Example: frame rate-doubling filter: filter_frame must (1) flush the
|
|
second copy of the previous frame, if it is still there, (2) push the
|
|
first copy of the incoming frame, (3) keep the second copy for later.)
|
|
|
|
If the input frame is not enough to produce output, the filter must not
|
|
call request_frame to get more. It must just process the frame or queue
|
|
it. The task of requesting more frames is left to the filter's
|
|
request_frame method or the application.
|
|
|
|
If a filter has several inputs, the filter must be ready for frames
|
|
arriving randomly on any input. Therefore, any filter with several inputs
|
|
will most likely require some kind of queuing mechanism. It is perfectly
|
|
acceptable to have a limited queue and to drop frames when the inputs
|
|
are too unbalanced.
|
|
|
|
request_frame
|
|
-------------
|
|
|
|
For filters that do not use the activate() callback, this method is
|
|
called when a frame is wanted on an output.
|
|
|
|
For a source, it should directly call filter_frame on the corresponding
|
|
output.
|
|
|
|
For a filter, if there are queued frames already ready, one of these
|
|
frames should be pushed. If not, the filter should request a frame on
|
|
one of its inputs, repeatedly until at least one frame has been pushed.
|
|
|
|
Return values:
|
|
if request_frame could produce a frame, or at least make progress
|
|
towards producing a frame, it should return 0;
|
|
if it could not for temporary reasons, it should return AVERROR(EAGAIN);
|
|
if it could not because there are no more frames, it should return
|
|
AVERROR_EOF.
|
|
|
|
The typical implementation of request_frame for a filter with several
|
|
inputs will look like that:
|
|
|
|
if (frames_queued) {
|
|
push_one_frame();
|
|
return 0;
|
|
}
|
|
input = input_where_a_frame_is_most_needed();
|
|
ret = ff_request_frame(input);
|
|
if (ret == AVERROR_EOF) {
|
|
process_eof_on_input();
|
|
} else if (ret < 0) {
|
|
return ret;
|
|
}
|
|
return 0;
|
|
|
|
Note that, except for filters that can have queued frames and sources,
|
|
request_frame does not push frames: it requests them to its input, and
|
|
as a reaction, the filter_frame method possibly will be called and do
|
|
the work.
|