fill_model_input_tf() pops a LastLevelTaskItem from lltask_queue and
stores it in request->lltask, but nothing frees it once the inference
has completed. The next fill_model_input_tf() on the same request just
overwrites the pointer, so only the last lltask of each request is
released, by destroy_request_item() when the model is freed. Every
inference therefore leaks one lltask, apart from the last one of each
of the nireq requests, with both async=0 and async=1.
Free it in infer_completion_callback() before the request is pushed
back to request_queue, as the OpenVINO backend does in its completion
callback. Also free it on the failure paths of execute_model_tf() and
dnn_flush_tf(), where the popped lltask is dropped in the same way. No
inference thread is running on any of these paths. av_freep() resets
the pointer, so the av_freep() in destroy_request_item() is still safe.
Signed-off-by: Jake Fineman <jake@wave.online>
With input_resizable enabled, get_output_ov() created partial_shape with
ov_partial_shape_create() and then passed the same variable to
ov_shape_to_partial_shape(), which allocates a new dims array and
overwrites the struct. The first allocation was never freed, so every
call leaked it, on the success path and on the two error returns in
between. The dims it was built from were never used.
Drop the ov_partial_shape_create() call and the dims array; the partial
shape is built from the model's input shape by ov_shape_to_partial_shape()
and freed after ov_model_reshape_single_input() as before.
The leak was introduced with the OpenVINO API 2.0 support in e79bd1f1b1.
Signed-off-by: Jake Fineman <jake@wave.online>
Check the request before reading its fields and use a context-free log for
the NULL case. This makes the documented EINVAL return reachable.
Assisted-by: Fairy
This is not a security issue.
Fixes: 6HSeekmc6mV7
Fixes: 8YwigyLd85Gn
Fixes: swarm-Memory-leak-of-input-output-frames-on-ff_dnn_execute_model-failure-3a5cbe
Fixes: memleak
Memory leak Replicated through UnModified FFmpeg with LSan
Found-by: zhang xingxing <2388969553@qq.com>
Fixes: O0Cm42dvrP4l
Fixes: NULL pointer dereference
NULL pointer dereference Replicated through UnModified FFmpeg with UBSAN and gdb
Found during the review of https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24617
Fixes: nC3u8khGyx61
Fixes: hang on teardown
Hang on teardown Replicated through UnModified FFmpeg with gdb
Hang on teardown Replicated through UnModified FFmpeg with gdb, a crafted 10240x10240 PNG, input_resizable=1 and vm.overcommit_memory=2
Found during the review of https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24617
Fixes: cE2BMN6svM0X
Fixes: success reported after a failure
Success reported after a failure Replicated through UnModified FFmpeg with LSan
Found during the review of https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24617
Fixes: o7gnt6oof6Qd
Fixes: use after free
Fixes: double free
Heap use after free after an allocation failure Replicated through UnModified FFmpeg with ASAN, the allocation failure injected through an LD_PRELOAD shim
Double free Replicated through UnModified FFmpeg with gdb, a crafted 8192x8192 PNG, input_resizable=1 and vm.overcommit_memory=2
Found during triage of the review of https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24617
Fixes: NULL pointer dereference
Fixes: gJMHsT6GErl5
Regression since: 22652b576c
Found during triage of the review of https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24617
NULL pointer dereference Replicated through UnModified FFmpeg with ASAN
This commit enables the DNN video filter to run entirely on the GPU
for the full D3D12 pipeline (decode, AI inference, and encode)
without ever copying frames back to system memory, making it
significantly faster on D3D12 hardware.
Benchmarked the new zero-copy D3D12/DirectML DNN pipeline against
the existing CPU round-trip pipeline on three major GPU vendors; up
to 10x throughput improvement was measured across every vendor.
Add dnn_onnx_d3d12, a helper that binds the DirectML execution
provider to a D3D12VA input frame's own device and does NV12
texture<->tensor conversion on the GPU (CopyTextureRegion + compute
shaders), avoiding a host round-trip.
dnn_processing negotiates this automatically for D3D12 NV12 input
with device=dml. Software-frame input retains the host I/O path.
D3D12 input requires successful DirectML negotiation; unsupported
hardware-input configurations fail during configuration. Currently
limited to single-channel FLOAT/NCHW models; chroma bypasses the
model and is copied/upscaled on the GPU. At runtime, DirectML.dll
and d3dcompiler_47.dll must be loadable from the application
directory or System32.
Example usage:
# Zero-copy pipeline
ffmpeg.exe -y -loglevel info \
-hwaccel d3d12va -hwaccel_output_format d3d12 \
-i input-1920x1080.mp4 \
-vf "dnn_processing=dnn_backend=onnx:device=dml:model=ecbsrnet_y_fused.onnx:input=input:output=output" \
-c:v h264_d3d12va -bf 0 -c:a copy \
output_2x_sr_zero_copy.mp4
# CPU round-trip pipeline
ffmpeg.exe -y -loglevel info \
-init_hw_device d3d12va=dev -filter_hw_device dev -hwaccel d3d12va \
-i input-1920x1080.mp4 \
-vf "dnn_processing=dnn_backend=onnx:device=dml:model=ecbsrnet_y_fused.onnx:input=input:output=output,hwupload" \
-c:v h264_d3d12va -bf 0 -c:a copy \
output_2x_sr_round_trip.mp4
Signed-off-by: younengxiao <steven.xiao@amd.com>
ff_proc_from_dnn_to_frame() and ff_proc_from_frame_to_dnn() each declare a
single `void *middle_data` local and, on the DL_NCHW layout branch, take its
address (`dst_data = &middle_data;` / `src_data = &middle_data;`) and pass
that as sws_scale()'s dst/src plane-array argument after casting to
`uint8_t *const *`. sws_scale()'s dst/src parameters are a 4-element array
(uint8_t *const dst[4]/src[4]), and scale_internal() copies all four
pointers (memcpy of srcSlice/dstSlice into src2/dst2) regardless of how many
planes the actual conversion uses. Since `&middle_data` only ever points at
one pointer-sized stack slot, that copy reads past it into whatever is next
on the stack: an out-of-bounds stack read on
every dnn_processing call whose model uses NCHW layout output or input into
an RGB24/BGR24 frame.
Fix both by giving dst_data/src_data real 4-element backing storage
(dst_data_buf/src_data_buf, zero-initialized) with the allocated buffer in
slot 0, instead of aliasing the single pointer.
Tested on arm64 macOS 27.0 (M1 Ultra), clang-asan toolchain, with
--enable-libopenvino (2026.4.0), --enable-libonnxruntime (1.30.0) and
--enable-libtensorflow, against unmodified upstream master (ff2059afd9) and
against this commit applied on top of the same base:
- Reproduced the stack-buffer-overflow on unmodified master two ways:
- ff_proc_from_dnn_to_frame(): `ffmpeg -threads 1 -i puppies.ogv -an -vf
"fps=2,format=rgb24,dnn_processing=dnn_backend=openvino:
model=proc_valid_720.onnx:input=input:output=output" -f framemd5 out.md5`
aborts under ASan on the first frame (offset-72 read into the
`middle_data`/`planar_data` stack slots, from the OpenVINO async
inference callback thread).
- ff_proc_from_frame_to_dnn(): the same dnn_processing filter with two real
ONNX Model Zoo fast-neural-style models (candy-9.onnx,
sha256 9d11a3529d1e547da6ae07201d93484dbab2ec0a3614535752c8f40f0fe2968a;
mosaic-9.onnx,
sha256 fa646dedade881243f8d5a2ceb7de2b93675b21fc24f7482894ac4851a9a0a47;
both NCHW 1x3x224x224 in/out) via dnn_backend=onnx aborts under ASan the
same way, this time with the overflow located directly in
ff_proc_from_frame_to_dnn's `middle_data` frame (offset 72), confirming
the sibling instance flagged at review is independently live, not just
structurally identical.
With this commit applied, all three of the above runs are clean of the
stack-buffer-overflow report (an unrelated, pre-existing OpenVINO/TBB
at-exit SEGV under ASan remains on this machine/toolchain/OpenVINO version;
it reproduces identically before and after this commit, and even on plain
`ffmpeg -version` with no DNN filter at all, so it is unrelated to this fix
and not addressed here).
- Whole-clip (puppies.ogv, 589 frames) framemd5 output from a release
(non-ASan) build is byte-for-byte identical before/after this commit for:
the NCHW OpenVINO identity model (proc_valid_720.onnx) into rgb24, and for
both candy-9.onnx and mosaic-9.onnx into rgb24 via dnn_backend=onnx. This
is expected: the aliasing read pulls garbage bytes that sws_scale never
writes to a packed destination/never reads for these plane counts, so the
visible output is unaffected; only ASan's redzone instrumentation makes the
read observable.
- Unchanged paths are also byte-for-byte identical before/after: the NHWC
TensorFlow path (tf_rgb_valid.pb into rgb24, which never takes the
DL_NCHW/middle_data branch) and the planar/gray output path (tf_y_valid.pb
into gray8, which uses frame->data directly and never allocates
middle_data).
- `make fate-source` passes with this commit. This tree has no fate-dnn*
tests to run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jake Fineman <jake@wave.online>
ff_proc_from_dnn_to_frame()'s AV_PIX_FMT_RGB24/BGR24 case assumes the
model's output has exactly 3 channels, but never checks it.
The first sws_scale() in that case reads from output->data with a
hardcoded source stride of frame->width * 3 * src_datatype_size over
frame->height rows, i.e. frame->width * 3 * frame->height *
src_datatype_size bytes. For a model whose output has fewer channels
than that -- e.g. the single-channel alpha matte an RGB-in
background-removal model produces -- the real tensor only holds
frame->width * frame->height * channels * src_datatype_size bytes, so
the call reads roughly 3x past the end of it. This affects both
DL_NCHW and DL_NHWC output, since it happens before the layout-specific
planar-to-packed branch.
For DL_NCHW there is a second, independent overflow on the write side:
middle_data is allocated plane_size * output->dims[1] bytes, but
linesize[0] is set to frame->width * 3 unconditionally, so the same
sws_scale() also writes past that allocation. This reproduces as a
SIGABRT (glibc "free(): invalid pointer", or a malloc.c assertion,
depending on allocator state) inside dnn_processing, after inference
has already completed successfully in the backend.
The input side is guarded -- check_modelinput_inlink() in
vf_dnn_processing.c -- but the output side is not. dnn_processing only
supports channel-preserving models for these pixel formats by design,
so reject a mismatched channel count with a clear error instead of
reading and writing out of bounds. The channel dimension is resolved
via dnn_get_channel_idx_by_layout() so the check covers both layouts.
This does not add support for rendering a 1- or 2-channel model output
through this filter; that would need a different pixel-format target
and is left as future work.
Signed-off-by: Jake Fineman <jake@wave.online>
get_output_tf() (dnn_backend_tf.c) and get_output_ov()
(dnn_backend_openvino.c) both declare `TaskItem task;` uninitialized
and then unconditionally free task.in_frame/task.out_frame on their
error path:
ret = ff_dnn_fill_gettingoutput_task(&task, ...);
if (ret != 0)
goto err;
...
err:
av_frame_free(&task.out_frame);
av_frame_free(&task.in_frame);
ff_dnn_fill_gettingoutput_task() (dnn_backend_common.c) allocates
in_frame first and, only if that succeeds, allocates out_frame and
assigns both pointers into exec_params (and from there into the
task via ff_dnn_fill_task()). If the first av_frame_alloc() fails,
the function returns AVERROR(ENOMEM) immediately, before task.in_frame
or task.out_frame is ever set. The two probes then reach `err:` with
task's frame fields holding whatever garbage was on the stack, and
av_frame_free() dereferences those pointers.
Fix by zero-initializing the declaration, matching get_output_onnx()
(dnn_backend_onnx.c) and get_output_th() (dnn_backend_torch.cpp),
which already use `TaskItem task = { 0 };` and are not affected.
Reported by Forgejo_Fairy, review 8971 on #24761, which also
confirmed the defect predates that series and is already present at
6246f7a4cd.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jake Fineman <jake@wave.online>
Add dnn_classify filter support to the ONNX Runtime DNN backend so
classification inference can run with ONNX models.
Classifier outputs of rank 2 [N,C] and rank 1 [C] are promoted to
[N,C,1,1], since common classification models produce these shapes.
The output batch dimension is validated to equal 1 before
post-processing.
Fix a deadlock when a frame has no bounding box to classify: the
request is returned to the queue instead of being destroyed, and the
lltask queue is checked before popping a request. Frames with multiple
bounding boxes are handled by looping through all queued lltasks in
execute_model_onnx.
Bounding box boundary checks use > instead of >=, so a box ending
exactly at the frame edge is no longer rejected.
ff_dnn_async_module_cleanup and ff_dnn_start_inference_async
unconditionally pthread_join() thread_id, which is zeroed by av_mallocz
until pthread_create runs. The first call to either helper joins a
never-created handle, which strict_pthread_join turns into abort().
HAVE_PTHREAD_CANCEL checks whether pthread_cancel() is available, not
whether thread support exists at all. It was used in dnn to guard pthread
API, none of which depend on pthread_cancel. Platforms can have a working
pthread implementation without pthread_cancel (e.g., Android), so the
guard could disable the async path on systems that actually support
threads.
Now that threads is a hard dependency of dnn, remove the guards and the
sync fallback path they protected.
The DNNCond / dnn_cond_* macros in safe_queue.c were guarded by
HAVE_PTHREAD_CANCEL, but none of them call pthread_cancel().
The guard was semantically wrong. AVCond provides both a threads
and a no-thread fallback path, so the conditional compilation is
no longer needed here.
The ONNX Runtime backend only ever runs one synchronous request, so
clamp nireq to 1 at load time (matching the actual allocation)
Signed-off-by: younengxiao <steven.xiao@amd.com>
When a CUDA frame with a supported RGB sw_format is received, map the
GPU device pointer directly into a LibTorch tensor using
torch::from_blob() with CUDA device options, bypassing cudaMemcpy
entirely.
Changes:
- Add fill_model_input_cuda() that extracts the CUdeviceptr from the
AVHWFramesContext and wraps it in a PyTorch GPU tensor with a no-op
deleter (memory owned by FFmpeg AVBuffer ref-counting).
- Handle GPU memory alignment padding via custom strides derived from
AVFrame linesize.
- Permute the mapped NHWC tensor to NCHW (PyTorch convention).
- Manage CUDA context explicitly with cuCtxPushCurrent/cuCtxPopCurrent
for thread safety in async filter graphs.
- Implement zero-copy Device-to-Device output mapping that writes the
model result directly into the output frame's VRAM, bypassing
ff_proc_from_dnn_to_frame / sws_scale entirely for CUDA frames.
- Add torch::cuda::synchronize() to prevent async race conditions
with downstream encoders (e.g. NVENC reading uninitialized frames).
- Add format validation in ff_dnn_zero_copy_supported_cuda() to reject
unsupported sw_formats and batching with CUDA zero-copy early.
- Register AV_PIX_FMT_CUDA in vf_dnn_processing pixel format list.
- Gracefully fall back to CPU path when sw_format is not a supported
RGB variant or when CONFIG_CUDA is not enabled.
Signed-off-by: Raja-89 <imraja729@gmail.com>
update ctx->ov_option.layout to the explict value which will be
used in function init_model_ov.
also move the code before input_resizable which uses the layout
When loading a TorchScript model that does not contain any learnable
parameters (e.g., a purely functional model), the Torch backend would
crash during inference. This occurred because the code attempted to
dereference the first iterator of the model's parameter list
`parameters().begin()` to determine the device, which results in
Undefined Behavior when the parameter list is empty.
This commit fixes the issue by determining the inference device directly
from the user-configured `ctx->device` string instead of probing the
model parameters, allowing parameterless models to execute safely.
Testing:
1. Generate a parameterless model:
cat << 'EOF' > generate_model.py
import torch
class DummyModel(torch.nn.Module):
def forward(self, x):
return x
scripted_model = torch.jit.script(DummyModel())
scripted_model.save("dummy_model.pt")
EOF
python3 generate_model.py
2. Run inference (previously crashed, now succeeds):
./ffmpeg -y -i input.mp4 -vf 'format=rgb24,dnn_processing=dnn_backend=torch:model=dummy_model.pt' -frames:v 5 -f null -
Signed-off-by: Raja Rathour <imraja729@gmail.com>
When the filter graph is torn down early (e.g. at EOF with -frames:v),
the main thread calls dnn_free_model and destroys the request queue
while async inference threads may still be running. When a detached
thread finishes and tries to push its result back to the destroyed
queue, it triggers a segmentation fault.
Add ff_dnn_wait_requests() to dnn_backend_common which blocks
until all allocated request items (ctx->nireq) have been safely
returned to the request_queue. Call it from dnn_free_model in the
torch, tensorflow, openvino, and onnx backends before destroying the queue.
Add batch processing support to the LibTorch DNN backend, following
the same pattern used by the OpenVINO backend.
Key changes:
- Add batch_size AVOption (range 1-1000, default 1) to DnnContext
- Accumulate lltasks in the queue, trigger inference when batch_size
lltasks are ready (matching the OpenVINO batch execution pattern)
- Pre-allocate a single contiguous memory buffer for the entire batch
in fill_model_input_th() to avoid unnecessary tensor copies
- Split batched output in infer_completion_callback() and dispatch
each slice to its corresponding task
Tested with:
ffmpeg -f lavfi -i testsrc=duration=5:size=640x480:rate=25 -vf format=rgb24,dnn_processing=dnn_backend=torch:model=dummy_model.pt:batch_size=4 -f null -
Signed-off-by: Raja Rathour <imraja729@gmail.com>
Initialize ctx->nireq with av_cpu_count() / 2 + 1 when
unset, matching the TensorFlow and OpenVINO backends.
Create ctx->nireq THRequestItems in a loop instead of
hardcoding a single request, enabling concurrent async
inference requests for improved throughput.
Signed-off-by: Raja Rathour <imraja729@gmail.com>
ff_dnn_start_inference_async() return value was not stored in ret,
causing execute_model_th() to return success on async startup failure.
This left vf_dnn_processing stuck in its flush loop waiting on a
task that was never started.
This patch adds ONNX Runtime as a new DNN backend for FFmpeg's dnn_processing
filter, enabling hardware-accelerated neural network inference on multiple
GPU and NPU platforms.
Execution Providers Supported:
- CPU execution provider (default)
- CUDA execution provider (NVIDIA GPUs)
- DirectML execution provider (AMD/Intel/NVIDIA GPUs on Windows)
- VitisAI execution provider (AMD Ryzen AI NPU)
The options for dnn_processing with dnn_backend=onnx:
- device: execution provider — cpu, cuda, dml, or vitisai (default: cpu)
- device_id: GPU device index (default: 0)
- threads_per_operation: inference thread count for CPU EP (default: 0, auto)
- input: input tensor name. When omitted the backend resolves it from loaded session
- output: output tensor name. When omitted the backend resolves it from loaded session
Example usage:
# CPU inference
ffmpeg -i input.mp4 -vf "format=rgb24,dnn_processing=dnn_backend=onnx:model=model.onnx:input=image_in:output=image_out" output.mp4
# CUDA GPU inference
ffmpeg -i input.mp4 -vf "dnn_processing=dnn_backend=onnx:model=model.onnx:device=cuda:device_id=0" output.mp4
# DirectML GPU inference (Windows)
ffmpeg -i input.mp4 -vf "dnn_processing=dnn_backend=onnx:model=model.onnx:device=dml:device_id=0" output.mp4
# VitisAI NPU inference
ffmpeg -i input.mp4 -vf "dnn_processing=dnn_backend=onnx:model=model.onnx:device=vitisai" output.mp4
Note: depending on the model, you may need a format filter (e.g. format=rgb24 or format=grayf32) before dnn_processing to convert the frames to the pixel format the model's input tensor expects.
Signed-off-by: younengxiao <steven.xiao@amd.com>
Reviewed-by: Guo Yejun <yejun.guo@intel.com>
Add support for CUDA and ROCm (AMD GPU) devices in the LibTorch DNN
backend.
This works for both NVIDIA CUDA and AMD ROCm, as PyTorch exposes ROCm
through the CUDA-compatible API.
Usage:
./ffmpeg -i input.mp4 -vf scale=224:224,format=rgb24,dnn_processing=dnn_backend=torch:model=sr_model_torch.pt:device=cuda output.mp4
Reviewed-by: Guo Yejun <yejun.guo@intel.com>
Signed-off-by: younengxiao <steven.xiao@amd.com>
ctx->options.async does not exist on DnnContext; the correct
field is ctx->async directly on the context struct.
Signed-off-by: younengxiao <steven.xiao@amd.com>
There are two options which use non-zero default value: async and
batch_size of openvino. init_model_ov checks and set batch_size to
one when batch_size is equal to zero, so the only option affected
by missing default value is async. Now async works as expected.
This patch implements the DNNAsyncExecModule for the LibTorch backend,
enabling non-blocking inference using the common infrastructure instead
of custom threading (th_async_module_submit) to align with the
TensorFlow and OpenVINO backends.
The implementation uses ff_dnn_start_inference_async which provides
unified async logic across all DNN backends, eliminating the need for
backend-specific threading code.
Verified with:
ffmpeg -f lavfi -i testsrc=duration=5:size=320x240:rate=30 -vf dnn_processing=dnn_backend=torch:model=model.pt -y output.mp4
Signed-off-by: Raja Rathour <imraja729@gmail.com>
Starting with PyTorch 2.6, the hooks API was redesigned so it no longer depends on the device type.
As part of this change, the XPU initialization function was renamed from initXPU() to init().
Add version check to support both old and new LibTorch versions.
Signed-off-by: younengxiao <steven.xiao@amd.com>
When using `dnn_processing` filter with torch backend, FFmpeg hangs indefinitely because no inference is actually performed.
Resolve this problem by add "else" branch for synchronous execution path.
Usage:
ffmpeg -i input.mp4 -vf scale=224:224,format=rgb24,dnn_processing=dnn_backend=torch:model=sr_model_torch.pt:device=cpu output.mp4
Update execute_model_th to support asynchronous execution by pushing tasks to the pending queue and notifying the worker thread.
Signed-off-by: Raja Rathour <imraja729@gmail.com>
Reviewed-by: Wenbin Chen <wenbin.chen@intel.com>
Reviewed-by: Guo Yejun <yejun.guo@intel.com>
Remove redundant av_freep() to avoid double free since task will be freed in dnn_free_model_tf() after the success of ff_queue_push_back().
Fixes: af052f9066 ("lavfi/dnn: fix mem leak in TF backend error handle")
Signed-off-by: Jiasheng Jiang <jiashengjiangcool@gmail.com>
Add xpu device support to libtorch backend.
To enable xpu support you need to add
"-Wl,--no-as-needed -lintel-ext-pt-gpu -Wl,--as-needed" to
"--extra-libs" when configure ffmpeg.
Signed-off-by: Wenbin Chen <wenbin.chen@intel.com>
Jake Fineman