1
0
mirror of https://github.com/FFmpeg/FFmpeg.git synced 2024-11-26 19:01:44 +02:00
Commit Graph

192 Commits

Author SHA1 Message Date
Andreas Rheinhardt
1ea3650823 Replace all occurences of av_mallocz_array() by av_calloc()
They do the same.

Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
2021-09-20 01:03:52 +02:00
Shubhanshu Saxena
660a205b05 lavfi/dnn: Rename InferenceItem to LastLevelTaskItem
This patch renames the InferenceItem to LastLevelTaskItem in the
three backends to avoid confusion among the meanings of these structs.

The following are the renames done in this patch:

1. extract_inference_from_task -> extract_lltask_from_task
2. InferenceItem -> LastLevelTaskItem
3. inference_queue -> lltask_queue
4. inference -> lltask
5. inference_count -> lltask_count

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-28 16:19:07 +08:00
Shubhanshu Saxena
1544d6fa0a libavfilter: Remove Async Flag from DNN Filter Side
Remove async flag from filter's perspective after the unification
of async and sync modes in the DNN backend.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-28 16:19:07 +08:00
Shubhanshu Saxena
60b4d07cf6 libavfilter: Unify Execution Modes in DNN Filters
This commit unifies the async and sync mode from the DNN filters'
perspective. As of this commit, the Native backend only supports
synchronous execution mode.

Now the user can switch between async and sync mode by using the
'async' option in the backend_configs. The values can be 1 for
async and 0 for sync mode of execution.

This commit affects the following filters:
1. vf_dnn_classify
2. vf_dnn_detect
3. vf_dnn_processing
4. vf_sr
5. vf_derain

This commit also updates the filters vf_dnn_detect and vf_dnn_classify
to send only the input frame and send NULL as output frame instead of
input frame to the DNN backends.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-28 16:19:07 +08:00
Shubhanshu Saxena
d39580ac11 lavfi/dnn: Task-based Inference in Native Backend
This commit rearranges the code in Native Backend to use the TaskItem
for inference.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-28 16:19:07 +08:00
Shubhanshu Saxena
2063745a93 lavfi/dnn: DNNAsyncExecModule Execution Failure Handling
This commit adds the case handling if the asynchronous execution
of a request fails by checking the exit status of the thread when
joining before starting another execution. On failure, it does the
cleanup as well.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
371e5672f3 lavfi/dnn_backend_tf: Error Handling for tf_create_inference_request
This commit includes the check for the case when the newly created
TFInferRequest is NULL.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
009b2e5b5e lavfi/dnn: Extract Common Parts from get_output functions
The frame allocation and filling the TaskItem with execution
parameters is common in the three backends. This commit shifts
this logic to dnn_backend_common.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
4d627acefa lavfi/dnn_backend_tf: Add TF_Status to TFRequestItem
Since requests are running in parallel, there is inconsistency in
the status of the execution. To resolve it, we avoid using mutex
as it would result in single TF_Session running at a time. So add
TF_Status to the TFRequestItem

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
a3db9b5405 lavfi/dnn_backend_tf: Error Handling for execute_model_tf
This patch adds error handling for cases where the execute_model_tf
fails, clears the used memory in the TFRequestItem and finally pushes
it back to the request queue.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
0985e9283c lavfi/dnn: Async Support for TensorFlow Backend
This commit enables async execution in the TensorFlow backend
and adds function to flush extra frames.

The async execution mechanism executes the TFInferRequests on
a separate thread which is joined before the next execution of
same TFRequestItem/while freeing the model.

The following is the comparison of this mechanism with the existing
sync mechanism on TensorFlow C API 2.5 CPU variant.

Async Mode: 4m32.846s
Sync Mode: 5m17.582s

The above was performed on super resolution filter using SRCNN model.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
e6ae8fc18e lavfi/dnn_backend_tf: TFInferRequest Execution and Documentation
This commit adds a function for execution of TFInferRequest and documentation
for functions related to TFInferRequest.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
c716578588 lavfi/dnn: Common Function to Get Async Result in DNN Backends
This commits refactors the get async result function for common
use in all three backends.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Shubhanshu Saxena
86f0a4f9de lavfi/dnn: Add Async Execution Mechanism and Documentation
This commit adds an async execution mechanism for common use
in the TensorFlow and Native backends.
This commit also adds the documentation of typedefs and functions in
the async module for common use in DNN backends.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-08-10 22:27:27 +08:00
Andreas Rheinhardt
ef54590f83 avfilter/internal: Don't include libavcodec/(avcodec|internal).h
The reasons for including them don't exist any longer: ff_tlog() has
been moved to libavutil/internal.h and FF_QSCALE_TYPE_* has been moved
to qp_table.h.

Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
2021-08-05 21:15:25 +02:00
Andreas Rheinhardt
2934a4b9a5 Remove unnecessary avassert.h inclusions
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
2021-07-22 15:02:30 +02:00
Andreas Rheinhardt
69f120ead7 avcodec/avcodec: Don't include cpu.h
It is not used here at all; instead, add it where it is used without
including it or any of the arch-specific CPU headers.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
2021-07-22 12:59:07 +02:00
Shubhanshu Saxena
0bc7ddc460 lavfi/dnn_backend_ov: Rename RequestItem to OVRequestItem
Rename RequestItem to OVRequestItem in the OpenVINO backend
to avoid confusion.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-22 08:13:14 +08:00
Shubhanshu Saxena
429954822c lavfi/dnn_backend_openvino.c: Fix Memory Leak in execute_model_ov
In cases where the execution inside the function execute_model_ov fails,
the OVRequestItem must be pushed back to the request_queue before returning
the error. In case pushing back fails, release the allocated memory.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-22 08:13:14 +08:00
Shubhanshu Saxena
6f9570a633 lavfi/dnn_backend_tf: Error Handling
This commit adds handling for cases where an error may occur, clearing
the allocated memory resources.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-11 20:12:27 +08:00
Shubhanshu Saxena
84e4e60fdc lavfi/dnn_backend_tf: Separate function for Completion Callback
This commit rearranges the existing code to create a separate function
for the completion callback in execute_model_tf.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-11 20:12:27 +08:00
Shubhanshu Saxena
b849228ae0 lavfi/dnn_backend_tf: Separate function for filling RequestItem
This commit rearranges the existing code to create separate function
for filling request with execution data.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-11 20:12:27 +08:00
Shubhanshu Saxena
08d8b3b631 lavfi/dnn_backend_tf: Request-based Execution
This commit uses TFRequestItem and the existing sync execution
mechanism to use request-based execution. It will help in adding
async functionality to the TensorFlow backend later.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-11 20:12:27 +08:00
Shubhanshu Saxena
a4de605110 lavfi/dnn_backend_tf: Add TFInferRequest and TFRequestItem
This commit introduces a typedef TFInferRequest to store
execution parameters for a single call to the TensorFlow C API.
This typedef is used in the TFRequestItem.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-11 20:12:27 +08:00
Shubhanshu Saxena
68cf14d2b1 lavfi/dnn_backend_tf: TaskItem Based Inference
This commit uses the common TaskItem and InferenceItem typedefs
for execution in TensorFlow backend.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-11 20:12:27 +08:00
Shubhanshu Saxena
f73943d514 lavfi/dnn_backend_openvino.c: Fix Memory Leak in execute_model_ov
In cases where the execution inside the function execute_model_ov fails,
push the RequestItem back to the request_queue before returning the error.
In case pushing back fails, release the allocated memory.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-07-04 18:56:17 +08:00
Guo Yejun
2cf95f2dd9 lavfi/dnn_backend_openvino.c: fix crash when target is not specified 2021-06-19 19:17:56 +08:00
Shubhanshu Saxena
2df963b5fa lavfi/dnn_backend_openvino.c: Fix Memory Leak for RequestItem
Fix memory leak for RequestItem upon error while pushing to the
request_queue in the completion callback.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-06-18 21:26:50 +08:00
Shubhanshu Saxena
5509235818 lavfi/dnn: Fill Task using Common Function
This commit adds a common function for filling the TaskItems
in all three backends.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-06-12 15:18:58 +08:00
Shubhanshu Saxena
6b961f7409 lavfi/dnn: Use uint8_t for async and do_ioproc in TaskItems
These properties have values either 0 or 1, so using uint8_t
is a better option as compared to int.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-06-12 15:18:58 +08:00
Shubhanshu Saxena
9675ebbb91 lavfi/dnn: Add nb_output to TaskItem
Add nb_output property to TaskItem for use in TensorFlow backend
and Native backend.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-06-12 15:18:58 +08:00
Shubhanshu Saxena
446b4f77c1 lavfi/dnn: Convert output_name to char** in TaskItem
Convert output_name to char **output_names in TaskItem and use it as
a pointer to array of output names in the DNN backend.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-06-12 15:18:58 +08:00
Shubhanshu Saxena
f5ab8905fd lavfi/dnn: Extract TaskItem and InferenceItem from OpenVino Backend
Extract TaskItem and InferenceItem from OpenVino backend and convert
ov_model to void in TaskItem.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-06-12 15:18:58 +08:00
Guo Yejun
3ac7d99428 lavfi/dnn/dnn_io_proc.c: fix CID 1484955
CID 1484955:  Memory - corruptions  (ARRAY_VS_SINGLETON)
2021-06-05 17:15:29 +08:00
Shubhanshu Saxena
e41255cddb lavfi/dnn_backend_openvino.c: Correct Pointer Type while Freeing
This commit corrects the type of pointer of elements from the
inference queue in ff_dnn_free_model_ov.

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-05-28 08:40:07 +08:00
Guo, Yejun
4c705a2775 lavfi/dnn: refine code to separate processing and detection in backends 2021-05-24 09:09:34 +08:00
Guo, Yejun
3fb1d2e71c lavfi/dnn/dnn_io_proc.c: fix Improper use of negative value (NEGATIVE_RETURNS)
fix coverity CID 1473511 and 1473566
2021-05-18 09:20:08 +08:00
Guo, Yejun
bd6ea9ed1d lavfi/dnn/dnn_io_proc.c: Fix Out-of-bounds access (ARRAY_VS_SINGLETON)
fix coverity CID 1473571, 1473577 and 1482089
2021-05-18 09:20:08 +08:00
Shubhanshu Saxena
11b489d592 lavfi/dnn_backend_native_layer_mathunary.h: Documentation
Add documentation for Unary Math Layer

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-05-17 09:33:40 +08:00
Shubhanshu Saxena
57fe5c1412 lavfi/dnn_backend_native_layer_depth2space.h: Documentation
Add documentation for Depth to Space Layer

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-05-17 09:33:40 +08:00
Shubhanshu Saxena
58de2b9eb3 lavfi/dnn_backend_native_layer_dense.h: Documentation
Add documentation for Dense Layer

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-05-17 09:33:40 +08:00
Shubhanshu Saxena
a61b7654a2 lavfi/dnn_backend_native_layer_conv2d.h: Documentation
Add documentation for 2D Convolution Layer

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-05-17 09:33:40 +08:00
Shubhanshu Saxena
0bdd677c5f lavfi/dnn_backend_native_layer_avgpool.h: Documentation
Add documentation for Average Pool Layer

Signed-off-by: Shubhanshu Saxena <shubhanshu.e01@gmail.com>
2021-05-14 10:21:15 +08:00
Limin Wang
2899fb61d2 avfilter/dnn/dnn_backend_tf: fix cross library usage
duplicate ff_hex_to_data() function from avformat and rename it to
hex_to_data() as static function.

Reviewed-by: Guo, Yejun <yejun.guo@intel.com>
Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
2021-05-11 18:46:14 +08:00
Ting Fu
e42125edab lavfi/dnn_backend_tensorflow: support detect model
Signed-off-by: Ting Fu <ting.fu@intel.com>
2021-05-11 10:28:35 +08:00
Ting Fu
1b1064054c lavfi/dnn_backend_tensorflow: add multiple outputs support
Signed-off-by: Ting Fu <ting.fu@intel.com>
2021-05-11 10:28:35 +08:00
Ting Fu
f02928eb5a dnn: add DCO_RGB color order to enum DNNColorOrder
Adding DCO_RGB color order to DNNColorOrder, since tensorflow model
needs this kind of color oder as input.

Signed-off-by: Ting Fu <ting.fu@intel.com>
2021-05-11 10:28:35 +08:00
Guo, Yejun
fc26dca64e lavfi/dnn: add classify support with openvino backend
Signed-off-by: Guo, Yejun <yejun.guo@intel.com>
2021-05-06 10:50:44 +08:00
Guo, Yejun
a3b74651a0 lavfi/dnn: refine dnn interface to add DNNExecBaseParams
Different function type of model requires different parameters, for
example, object detection detects lots of objects (cat/dog/...) in
the frame, and classifcation needs to know which object (cat or dog)
it is going to classify.

The current interface needs to add a new function with more parameters
to support new requirement, with this change, we can just add a new
struct (for example DNNExecClassifyParams) based on DNNExecBaseParams,
and so we can continue to use the current interface execute_model just
with params changed.
2021-05-06 10:50:44 +08:00
Guo, Yejun
7eb9accc37 lavfi/dnn_backend_openvino.c: move the logic for batch mode earlier 2021-05-06 10:50:44 +08:00