1
0
mirror of https://github.com/FFmpeg/FFmpeg.git synced 2024-12-12 19:18:44 +02:00
Commit Graph

32 Commits

Author SHA1 Message Date
Shubhanshu Saxena
60b4d07cf6 libavfilter: Unify Execution Modes in DNN Filters
This commit unifies the async and sync mode from the DNN filters'
perspective. As of this commit, the Native backend only supports
synchronous execution mode.

Now the user can switch between async and sync mode by using the
'async' option in the backend_configs. The values can be 1 for
async and 0 for sync mode of execution.

This commit affects the following filters:
1. vf_dnn_classify
2. vf_dnn_detect
3. vf_dnn_processing
4. vf_sr
5. vf_derain

This commit also updates the filters vf_dnn_detect and vf_dnn_classify
to send only the input frame and send NULL as output frame instead of
input frame to the DNN backends.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-28 16:19:07 +08:00
Ting Fu
f02928eb5a dnn: add DCO_RGB color order to enum DNNColorOrder
Adding DCO_RGB color order to DNNColorOrder, since tensorflow model
needs this kind of color oder as input.

Signed-off-by: Ting Fu <ting.fu@intel.com>
2021-05-11 10:28:35 +08:00
Guo, Yejun
fc26dca64e lavfi/dnn: add classify support with openvino backend
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2021-05-06 10:50:44 +08:00
Guo, Yejun
a3b74651a0 lavfi/dnn: refine dnn interface to add DNNExecBaseParams
Different function type of model requires different parameters, for
example, object detection detects lots of objects (cat/dog/...) in
the frame, and classifcation needs to know which object (cat or dog)
it is going to classify.

The current interface needs to add a new function with more parameters
to support new requirement, with this change, we can just add a new
struct (for example DNNExecClassifyParams) based on DNNExecBaseParams,
and so we can continue to use the current interface execute_model just
with params changed.
2021-05-06 10:50:44 +08:00
Guo, Yejun
13bf797ced lavfi/dnn: add post process for detection 2021-04-08 09:23:02 +08:00
Guo, Yejun
59021d79a2 lavfi/dnn: refine code for frame pre/proc processing 2021-04-08 09:23:02 +08:00
Guo, Yejun
0884063f88 dnn_interface.h: add enum DNNColorOrder
the data type and order together decide the color format, we could
not use AVPixelFormat directly because not all the possible formats
are covered by it.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2021-02-18 09:59:37 +08:00
Guo, Yejun
76fc6879e2 dnn: add function type for model
So the backend knows the usage of model is for frame processing,
detect, classify, etc. Each function type has different behavior
in backend when handling the input/output data of the model.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2021-02-18 09:59:37 +08:00
Guo, Yejun
64ea15f050 libavfilter/dnn: add batch mode for async execution
the default number of batch_size is 1

Signed-off-by: Xie, Lin <lin.xie@intel.com>
Signed-off-by: Wu Zhiwen <zhiwen.wu@intel.com>
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2021-01-15 08:59:54 +08:00
Guo, Yejun
477dd2df60 dnn_interface.h: fix redefining typedefs
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2020-12-31 09:21:31 +08:00
Guo, Yejun
5024286465 dnn_interface: change from 'void *userdata' to 'AVFilterContext *filter_ctx'
'void *' is too flexible, since we can derive info from
AVFilterContext*, so we just unify the interface with this data
structure.

Signed-off-by: Xie, Lin <lin.xie@intel.com>
Signed-off-by: Wu Zhiwen <zhiwen.wu@intel.com>
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2020-12-29 09:31:06 +08:00
Guo, Yejun
e67b5d0a24 dnn: add async execution support for openvino backend
Signed-off-by: Xie, Lin <lin.xie@intel.com>
Signed-off-by: Wu Zhiwen <zhiwen.wu@intel.com>
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2020-12-29 09:31:06 +08:00
Guo, Yejun
39f5cb4bd1 dnn_interface: add interface to support async execution
Signed-off-by: Xie, Lin <lin.xie@intel.com>
Signed-off-by: Wu Zhiwen <zhiwen.wu@intel.com>
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2020-12-29 09:31:06 +08:00
Guo, Yejun
e71d73b096 dnn: add a new interface DNNModel.get_output
for some cases (for example, super resolution), the DNN model changes
the frame size which impacts the filter behavior, so the filter needs
to know the out frame size at very beginning.

Currently, the filter reuses DNNModule.execute_model to query the
out frame size, it is not clear from interface perspective, so add
a new explict interface DNNModel.get_output for such query.
2020-09-21 21:26:56 +08:00
Guo, Yejun
fce3e3e137 dnn: put DNNModel.set_input and DNNModule.execute_model together
suppose we have a detect and classify filter in the future, the
detect filter generates some bounding boxes (BBox) as AVFrame sidedata,
and the classify filter executes DNN model for each BBox. For each
BBox, we need to crop the AVFrame, copy data to DNN model input and do
the model execution. So we have to save the in_frame at DNNModel.set_input
and use it at DNNModule.execute_model, such saving is not feasible
when we support async execute_model.

This patch sets the in_frame as execution_model parameter, and so
all the information are put together within the same function for
each inference. It also makes easy to support BBox async inference.
2020-09-21 21:26:56 +08:00
Guo, Yejun
2003e32f62 dnn: change dnn interface to replace DNNData* with AVFrame*
Currently, every filter needs to provide code to transfer data from
AVFrame* to model input (DNNData*), and also from model output
(DNNData*) to AVFrame*. Actually, such transfer can be implemented
within DNN module, and so filter can focus on its own business logic.

DNN module also exports the function pointer pre_proc and post_proc
in struct DNNModel, just in case that a filter has its special logic
to transfer data between AVFrame* and DNNData*. The default implementation
within DNN module is used if the filter does not set pre/post_proc.
2020-09-21 21:26:56 +08:00
Guo, Yejun
6918e240d7 dnn: add userdata for load model parameter
the userdata will be used for the interaction between AVFrame and DNNData
2020-09-21 21:26:56 +08:00
Guo, Yejun
0f7a99e37a dnn: move output name from DNNModel.set_input_output to DNNModule.execute_model
currently, output is set both at DNNModel.set_input_output and
DNNModule.execute_model, it makes sense that the output name is
provided at model inference time so all the output info is set
at a single place.

and so DNNModel.set_input_output is renamed to DNNModel.set_input

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2020-08-25 09:02:59 +08:00
Guo, Yejun
0a51abe8ab dnn: add backend options when load the model
different backend might need different options for a better performance,
so, add the parameter into dnn interface, as a preparation.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2020-08-12 15:43:40 +08:00
Guo, Yejun
ff37ebaf30 dnn: add openvino as one of dnn backend
OpenVINO is a Deep Learning Deployment Toolkit at
https://github.com/openvinotoolkit/openvino, it supports CPU, GPU
and heterogeneous plugins to accelerate deep learning inferencing.

Please refer to https://github.com/openvinotoolkit/openvino/blob/master/build-instruction.md
to build openvino (c library is built at the same time). Please add
option -DENABLE_MKL_DNN=ON for cmake to enable CPU path. The header
files and libraries are installed to /usr/local/deployment_tools/inference_engine/
with default options on my system.

To build FFmpeg with openvion, take my system as an example, run with:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/deployment_tools/inference_engine/lib/intel64/:/usr/local/deployment_tools/inference_engine/external/tbb/lib/
$ ../ffmpeg/configure --enable-libopenvino --extra-cflags=-I/usr/local/deployment_tools/inference_engine/include/ --extra-ldflags=-L/usr/local/deployment_tools/inference_engine/lib/intel64
$ make

Here are the features provided by OpenVINO inference engine:
- support more DNN model formats
It supports TensorFlow, Caffe, ONNX, MXNet and Kaldi by converting them
into OpenVINO format with a python script. And torth model
can be first converted into ONNX and then to OpenVINO format.

see the script at https://github.com/openvinotoolkit/openvino/tree/master/model-optimizer/mo.py
which also does some optimization at model level.

- optimize at inference stage
It optimizes for X86 CPUs with SSE, AVX etc.

It also optimizes based on OpenCL for Intel GPUs.
(only Intel GPU supported becuase Intel OpenCL extension is used for optimization)

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2020-07-02 09:36:34 +08:00
Guo, Yejun
f4b3c0e55c avfilter/dnn: add a new interface to query dnn model's input info
to support dnn networks more general, we need to know the input info
of the dnn model.

background:
The data type of dnn model's input could be float32, uint8 or fp16, etc.
And the w/h of input image could be fixed or variable.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-10-30 11:07:06 -03:00
Guo, Yejun
e1b45b8596 avfilter/dnn: get the data type of network output from dnn execution result
so,  we can make a filter more general to accept different network
models, by adding a data type convertion after getting data from network.

After we add dt field into struct DNNData, it becomes the same as
DNNInputData, so merge them with one struct: DNNData.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-10-30 11:00:41 -03:00
Guo, Yejun
83e0b71f66 dnn: export operand info in python script and load in c code
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-08-30 11:41:30 -03:00
Guo, Yejun
c636dc9819 libavfilter/dnn: add more data type support for dnn model input
currently, only float is supported as model input, actually, there
are other data types, this patch adds uint8.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-05-08 12:33:00 -03:00
Guo, Yejun
25c1cd909f libavfilter/dnn: support multiple outputs for tensorflow model
some models such as ssd, yolo have more than one output.

the clean up code in this patch is a little complex, it is because
that set_input_output_tf could be called for many times together
with ff_dnn_execute_model_tf, we have to clean resources for the
case that the two interfaces are called interleaved.

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-05-08 12:33:00 -03:00
Guo, Yejun
e2b92896c4 libavfilter/dnn: determine dnn output during execute_model instead of set_input_output
Currently, within interface set_input_output, the dims/memory of the tensorflow
dnn model output is determined by executing the model with zero input,
actually, the output dims might vary with different input data for networks
such as object detection models faster-rcnn, ssd and yolo.

This patch moves the logic from set_input_output to execute_model which
is suitable for all the cases. Since interface changed, and so dnn_backend_native
also changes.

In vf_sr.c, it knows it's srcnn or espcn by executing the model with zero input,
so execute_model has to be called in function config_props

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-05-08 12:33:00 -03:00
Guo, Yejun
05f86f05bb libavfilter/dnn: remove limit for the name of DNN model input/output
remove the requirment that the name of DNN model input/output
should be "x"/"y",

Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2019-05-08 12:33:00 -03:00
Sergey Lavrushkin
bd10c1e9a8 libavfilter: Removes stored DNN models. Adds support for native backend model file format in tf backend.
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2018-09-17 19:44:42 -03:00
Sergey Lavrushkin
9d87897ba8 libavfilter: Code style fixes for pointers in DNN module and sr filter.
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2018-08-07 11:58:34 -03:00
Sergey Lavrushkin
575b718990 Adds ESPCN super resolution filter merged with SRCNN filter.
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2018-07-02 10:47:14 -03:00
Sergey Lavrushkin
d8c0bbb0aa Adds TensorFlow backend for dnn inference module.
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2018-06-05 10:16:50 -03:00
Sergey Lavrushkin
bdf1bbdbb4 Adds dnn inference module for simple convolutional networks. Reimplements srcnn filter based on it.
Signed-off-by: Pedro Arthur <bygrandao@gmail.com>
2018-05-29 10:02:30 -03:00