diff --git a/README.md b/README.md index 184e7ec..65182e7 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ - [Capabilities of iVSR](#14-capabilities-of-ivsr) - [Video Super Resolution (VSR)](#141-video-super-resolution-vsr) - [Smart Video Processing (SVP)](#142-smart-video-processing-svp) + - [VideoSeal Invisible Watermarking](#143-videoseal-invisible-watermarking) 2. [Setup iVSR env on linux](#2-setup-ivsr-env-on-linux) - [Install GPU kernel packages(Optional)](#21-optional-install-gpu-kernel-packages) - [Install dependencies and build iVSR manually](#22-install-dependencies-and-build-ivsr-manually) @@ -64,7 +65,10 @@ We've also included a `vsr_sample` as a demonstration of its usage. In order to support the widely-used media processing solution FFmpeg, we've provided an iVSR SDK plugin to simplify integration.
This plugin is integrated into FFmpeg's `dnn_processing` filter in the [FFmpeg documentation](https://ffmpeg.org/ffmpeg-filters.html#dnn_005fprocessing-1) in the libavfilter library, serving as a new `ivsr` backend to this filter. The patches provided in this project target **FFmpeg n8.1**.
-### 1.3.2 OpenVINO patches and extension +### 1.3.2 JSON Model Configuration +As of patch 0005, a **JSON config dispatch system** is included. Enhanced BasicVSR is the only built-in model type (`model_type=0`). All other models — SVP/VideoProc, Enhanced EDSR, Custom VSR, TSENet, RIFE, and VideoSeal — are configured at runtime by passing a JSON descriptor file via the `model_config` AVOption. This means new models can be supported without C code changes or recompilation. Canonical config files are provided in `ivsr_ffmpeg_plugin/model_configs/`.
+ +### 1.3.3 OpenVINO patches and extension In [this folder](./ivsr_ov/based_on_openvino_2022.3/patches), you'll find patches for OpenVINO that enable the Enhanced BasicVSR model. These patches utilize OpenVINO's [Custom OpenVINO™ Operations](https://docs.openvino.ai/2024/documentation/openvino-extensibility/custom-openvino-operations.html) feature, which allows users to support models with custom operations not inherently supported by OpenVINO.
These patches are specifically for OpenVINO 2022.3, meaning the Enhanced BasicVSR model will only work on OpenVINO 2022.3 with these patches applied.
@@ -72,7 +76,7 @@ These patches are specifically for OpenVINO 2022.3, meaning the Enhanced BasicVS Currently, iVSR offers two AI media processing functionalities: Video Super Resolution (VSR) and Smart Video Processing (SVP) for bandwidth optimization. Both functionalities can be run on Intel CPUs and Intel GPUs (including Flex170, Arc770) via OpenVINO and FFmpeg. ### 1.4.1 Video Super Resolution (VSR) -Video Super Resolution (VSR) is a technique extensively employed in the AI media enhancement domain to upscale low-resolution videos to high-resolution. iVSR supports `Enhanced BasicVSR`, `Enhanced EDSR`, and `TSENet`. It also has the capability to be extended to support additional models. +Video Super Resolution (VSR) is a technique extensively employed in the AI media enhancement domain to upscale low-resolution videos to high-resolution. iVSR supports `Enhanced BasicVSR`, `Enhanced EDSR`, `TSENet`, `RIFE` (frame interpolation), and `SPAN`. It also has the capability to be extended to support additional models via JSON config files. - #### i. Enhanced BasicVSR `BasicVSR` is a publicly available AI-based VSR algorithm. For more details on the public `BasicVSR`, please refer to this [paper](https://arxiv.org/pdf/2012.02181.pdf).

@@ -102,6 +106,23 @@ Video Super Resolution (VSR) is a technique extensively employed in the AI media ``` For each inference, the input data is the `(n-1)th`, `(n)th`, and `(n+1)th` frames combined. The output data is the `(N)th` frame. For the first frame, the input data is `1st`, `1st`, `2nd` frames combined. For the last frame, the input data is the `(n-1)th`, `(n)th`, `(n)th` frames combined. +- #### iv. RIFE (Frame Interpolation) + `RIFE` (Real-Time Intermediate Flow Estimation) is an AI-based frame interpolation model that synthesises intermediate frames between two consecutive input frames, enabling frame-rate upscaling (e.g. 30 fps → 60 fps).

+ RIFE uses a 2-frame sliding window (`nif=2`). Pixel values are normalised to `[0, 1]` before inference and the output is scaled back to `[0, 255]`. The model is configured via the JSON config file `model_configs/rife_config.json` (`model_config` AVOption). Input alignment is 128 pixels (width and height are padded to the nearest multiple of 128 before inference).

+ The input and output shapes are: + ```plaintext + Input shape: [1, (channels * frames)6, H, W] (2 × RGB frames concatenated along the channel axis) + Output shape: [1, (channels)3, H, W] (single interpolated RGB frame) + ``` + +- #### v. SPAN + `SPAN` (Swift Parameter-free Attention Network) is a lightweight single-frame super-resolution model. Like RIFE, it expects pixel values normalised to `[0, 1]` (float32) and produces output in the `[0, 1]` range which is scaled back to `[0, 255]`. The model is configured via the JSON config file `model_configs/span_config.json` (`model_config` AVOption).

+ The input and output shapes are: + ```plaintext + Input shape: [1, (channels)3, H, W] + Output shape: [1, (channels)3, scale×H, scale×W] + ``` + ### 1.4.2. Smart Video Processing (SVP) `SVP` is an AI-based video prefilter that enhances perceptual rate-distortion in video encoding. With `SVP`, encoded video streams maintain the same visual quality while reducing bandwidth usage.
@@ -124,6 +145,15 @@ The input and output shapes are: ```
+### 1.4.3 VideoSeal Invisible Watermarking +`VideoSeal` is an AI-based invisible video watermarking model. It embeds imperceptible watermarks into video frames for content protection, provenance tracking, and rights management. The watermark survives common video processing operations and is not visible to the human eye.

+VideoSeal operates as a single-frame passthrough-dimension model: the output resolution matches the input. Pixel values are packed as raw `[0, 255]` float32 (no `/255` normalisation). It is configured via the JSON config file `model_configs/videoseal_config.json` (`model_config` AVOption).

+The input and output shapes are: +```plaintext +Input shape: [1, (channels)3, H, W] (RGB, float32 [0, 255]) +Output shape: [1, (channels)3, H, W] (watermarked RGB frame) +``` + # 2. Setup iVSR env on linux The software was validated on: - Intel Xeon hardware platform @@ -188,6 +218,8 @@ The `vsr_sample` is developed using the iVSR SDK and OpenCV. For detailed instru ## 3.2 Run with FFmpeg After applying the FFmpeg plugin patches and building FFmpeg, refer to [the FFmpeg command line samples](ivsr_ffmpeg_plugin/README.md#how-to-run-inference-with-ffmpeg-plugin) for instructions on running inference with FFmpeg. +> **Model dispatch:** Enhanced BasicVSR uses `model_type=0` (built-in). All other models — SVP/VideoProc, Enhanced EDSR, Custom VSR, TSENet, RIFE, SPAN, and VideoSeal — are selected by passing a JSON descriptor file via the `model_config` AVOption (e.g. `model_config=/ivsr_ffmpeg_plugin/model_configs/rife_config.json`). Canonical config files are provided in `ivsr_ffmpeg_plugin/model_configs/`. Custom models can be added without code changes by copying and editing `ivsr_model_config.template.json`. + # 4. Model files iVSR supports only models in OpenVINO IR format. Contact your Intel representative to obtain the model files, as they are not included in the repo. diff --git a/build.sh b/build.sh index ed0b00b..828f672 100755 --- a/build.sh +++ b/build.sh @@ -99,7 +99,7 @@ build_install_ivsr_sdk() { ivsr_sdk_dir=${base_dir}/ivsr_sdk/ cd ${ivsr_sdk_dir} - rm -rf build + sudo rm -rf build mkdir -p build && cd build && cmake \ -DENABLE_LOG=OFF -DENABLE_PERF=OFF -DENABLE_THREADPROCESS=ON \ -DENABLE_IRGUARD=${enable_irguard} \ @@ -138,11 +138,15 @@ build_ffmpeg() { ffmpeg_tag=n8.1 ffmpeg_repo=https://github.com/FFmpeg/FFmpeg.git + git config --global --add safe.directory ${ffmpeg_dir} + if [ ! -d "${ffmpeg_dir}/.git" ]; then git clone --depth 1 --branch ${ffmpeg_tag} ${ffmpeg_repo} ${ffmpeg_dir} - git config --global --add safe.directory ${ffmpeg_dir} fi + # Fix ownership of root-owned files from previous sudo builds + sudo chown -R $(id -u):$(id -g) "${ffmpeg_dir}" + cd ${ffmpeg_dir} git am --abort 2>/dev/null || true # Ensure the target tag is locally reachable (handles a prior clone at a different tag) @@ -150,6 +154,9 @@ build_ffmpeg() { git fetch --depth 1 origin "refs/tags/${ffmpeg_tag}:refs/tags/${ffmpeg_tag}" fi git checkout -f "${ffmpeg_tag}" + # Remove untracked files left by previous patch applications so git apply + # can create new files (e.g. dnn_backend_ivsr.c) without conflicts. + git clean -fd -- libavfilter libswscale # --------------------------------------------------------------- # Apply all iVSR patches for n8.1. @@ -176,6 +183,9 @@ build_ffmpeg() { git apply --3way --whitespace=fix 0002-*.patch git apply --3way --whitespace=fix 0003-*.patch + git apply --3way --whitespace=fix 0005-*.patch + git apply --3way --whitespace=fix 0006-*.patch + git apply --3way --whitespace=fix 0007-*.patch ./configure \ --enable-gpl \ @@ -205,7 +215,7 @@ install_openvino_from_apt() { echo "ERROR: Failed to download Intel GPG key from ${key_url}" >&2 exit 1 fi - sudo gpg --output "${keyring}" --dearmor /tmp/intel-sw-products.pub + sudo gpg --yes --output "${keyring}" --dearmor /tmp/intel-sw-products.pub rm -f /tmp/intel-sw-products.pub if [ ! -s "${keyring}" ]; then diff --git a/ivsr_ffmpeg_plugin/README.md b/ivsr_ffmpeg_plugin/README.md index 5083efd..8a6bdbd 100644 --- a/ivsr_ffmpeg_plugin/README.md +++ b/ivsr_ffmpeg_plugin/README.md @@ -14,6 +14,17 @@ For full instructions on building the Docker image, see [docs/docker_image_build To run inference with iVSR SDK, you need to specify `ivsr` as the backend for the `dnn_processing` filter. Here is an example of how to do it: `dnn_processing=dnn_backend=ivsr`.
Additionally, there are other parameters that you can use. These parameters are listed in the table below:
+> **Model dispatch (patches 0005-0007):** Two dispatch paths are supported for backward compatibility: +> +> 1. **Legacy integer path** (`model_type=0..8`): Use integer values to select built-in model descriptors. Production code using `model_type=1..8` continues to work unchanged. +> 2. **JSON config path** (`model_config=.json`): Use JSON descriptor files for runtime configuration. Provides flexibility for model customization without code changes. +> +> **Priority**: When both are set, `model_config` takes priority over `model_type`. This enables custom JSON configs to override built-in descriptors. +> +> Canonical JSON config files are provided in `ivsr_ffmpeg_plugin/model_configs/`. Both paths produce identical output for the same model. + +> **OpenVINO version:** Enhanced BasicVSR requires OpenVINO **2022.3** with custom patches. All other models (SVP, EDSR, CustVSR, TSENet, RIFE, SPAN, VideoSeal, HDRTVNet-LE) require OpenVINO **2026.1** (installed via apt). The `build.sh` script in the project root defaults to OV 2026.1. + |AVOption name|Description|Default value|Recommended value(s)| |:--|:--|:--|:--| |dnn_backend|DNN backend framework name|native|ivsr| @@ -21,44 +32,204 @@ Additionally, there are other parameters that you can use. These parameters are |input|input name of the model|NULL|input| |output|output name of the model|NULL|output| |device|device for inference task|CPU|CPU or GPU| -|model_type|type for models|0|0 for Enhanced BasicVSR, 1 for SVP models, 2 for Enhanced EDSR, 3 for one CUSTOM VSR, 4 for TSENet| -|normalize_factor|normalizing factor for models that do not require input normalization to [0, 1]|1.0|255.0 for Enhanced EDSR, 1.0 for all other models| +|model_type|Built-in model selector using integer values: `0`=BasicVSR, `1`=VideoProc, `2`=EDSR, `3`=CustVSR, `4`=TSENet, `5`=RIFE, `6`=SPAN, `7`=VideoSeal, `8`=HDRTVNet-LE. **Legacy option**: For backward compatibility with existing production code. New projects should use `model_config` instead.|0|0-8 (see table below)| +|model_config|Path to a JSON model descriptor file. Takes priority over `model_type` when both are set. Enables runtime model customization without code changes. Canonical configs: `model_configs/{videoproc,edsr,custvsr,tsenet,rife,span,videoseal,hdrtvnet_le}_config.json`. See `ivsr_model_config.template.json` for all fields.|NULL|Full path to the appropriate `*_config.json`| +|normalize_factor|Legacy normalizing factor for models that do not require input normalization to [0, 1]. For JSON-config models this is handled inside the config.|1.0|255.0 for Enhanced EDSR (legacy), 1.0 for all other models| |num_streams|number of execution streams for throughput mode (valid only for GPU devices)|1|use `benchmark_app` to determine the best value| |extension|extension lib file full path, required for Enhanced BasicVSR|—|—| |op_xml|custom op xml file full path, required for Enhanced BasicVSR|—|—| -|nif|number of input frames in batch sent to the DNN backend|1|3 for Enhanced BasicVSR, 1 for other models| +|nif|number of input frames in batch sent to the DNN backend|1|3 for Enhanced BasicVSR; for JSON-config models this is set inside the config| |nireq|number of infer requests|0|leave as default or set to match CPU core count| -Here are some examples of FFmpeg command lines to run inference with the supported models using the `ivsr` backend.
+### Model Type Integer Mapping + +| `model_type` | Model Name | Equivalent JSON Config | Input Format | Notes | +|:-------------|:-----------|:-----------------------|:-------------|:------| +| 0 | Enhanced BasicVSR | N/A (built-in only) | rgb24 | Requires OV 2022.3 + patches | +| 1 | VideoProc (SVP) | `videoproc_config.json` | yuv420p or rgb24 | Y-channel SR, auto-detects color | +| 2 | Enhanced EDSR | `edsr_config.json` | rgb24 | Single-frame RGB SR | +| 3 | Custom VSR | `custvsr_config.json` | yuv420p | Y-channel only | +| 4 | TSENet | `tsenet_config.json` | rgb24 | 3-frame temporal SR | +| 5 | RIFE | `rife_config.json` | rgb24 | Frame interpolation | +| 6 | SPAN | `span_config.json` | rgb24 | Single-frame RGB SR | +| 7 | VideoSeal | `videoseal_config.json` | rgb24 | Invisible watermarking | +| 8 | HDRTVNet-LE | `hdrtvnet_le_config.json` | rgb24 | HDR local enhancement | -- Command sample to run Enhanced BasicVSR inference, the input pixel format supported by the model is `rgb24`. +Here are examples of FFmpeg command lines to run inference with the supported models using the `ivsr` backend. Both **legacy integer** (`model_type`) and **JSON config** (`model_config`) approaches are shown for comparison.
+ +### Enhanced BasicVSR (built-in, `model_type=0`) +Input pixel format: `rgb24`. Requires OpenVINO 2022.3 with patches applied. ``` cd /ivsr_ffmpeg_plugin/ffmpeg -./ffmpeg -i -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=3:device=:extension=/ivsr_ov/based_on_openvino_2022.3/openvino/bin/intel64/Release/libcustom_extension.so:op_xml=/ivsr_ov/based_on_openvino_2022.3/openvino/flow_warp_cl_kernel/flow_warp.xml test_out.mp4 +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=3:device=:model_type=0:extension=/ivsr_ov/based_on_openvino_2022.3/openvino/bin/intel64/Release/libcustom_extension.so:op_xml=/ivsr_ov/based_on_openvino_2022.3/openvino/flow_warp_cl_kernel/flow_warp.xml \ + test_out.mp4 ``` -Please note that for the Enhanced BasicVSR model, you need to set the `extension` and `op_xml` options (with `backend_configs`) in the command line. After applying OpenVINO's patches and building OpenVINO, the extension lib file is located in `/openvino/bin/intel64/Release/libcustom_extension.so`, and the op xml file is located in `/openvino/flow_warp_cl_kernel/flow_warp.xml`.
+The `extension` and `op_xml` options are required for Enhanced BasicVSR. After applying OpenVINO's patches and building OpenVINO, the extension lib is at `/openvino/bin/intel64/Release/libcustom_extension.so` and the op xml is at `/openvino/flow_warp_cl_kernel/flow_warp.xml`.
-- Command sample to run SVP models inference. If the supported input pixel format of the model variance is `rgb24`, set the preceeding format as is to avoid unnecessary layout conversion: -``` +--- + +> **All models below support both dispatch paths (patches 0005-0007).** You can use either: +> - **Legacy**: `model_type=N` (where N is 1-8) for built-in descriptors +> - **JSON**: `model_config=.json` for custom configurations +> +> Both produce identical output. Canonical JSON configs are in `ivsr_ffmpeg_plugin/model_configs/`. + +### SVP / VideoProc (`model_type=1`) +Input pixel format: `rgb24` (RGB model variant) or `yuv420p` (Y-channel model variant). Raw pixel values `[0, 255]` are passed to the model as float32 without normalisation. `color_format_auto: 1` enables automatic I420/RGB colour-format selection based on the input pixel format. + +**Legacy integer path** (`model_type=1`): +```bash cd /ivsr_ffmpeg_plugin/ffmpeg -./ffmpeg -i -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=1:device=:model_type=1 -pix_fmt yuv420p test_out.mp4 +# Y-channel variant +./ffmpeg -i \ + -vf format=yuv420p,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=1:device= \ + -pix_fmt yuv420p test_out.mp4 ``` -If the model variance supports Y-input, set the preceeding format as YUV: + +**JSON config path** (equivalent): +```bash +# Y-channel variant +./ffmpeg -i \ + -vf format=yuv420p,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_config=/ivsr_ffmpeg_plugin/model_configs/videoproc_config.json:device= \ + -pix_fmt yuv420p test_out.mp4 ``` -./ffmpeg -i -vf format=yuv420p,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=1:device=:model_type=1 -pix_fmt yuv420p test_out.mp4 + +### Enhanced EDSR (`model_type=2`) +Input pixel format: `rgb24`. Single-frame RGB super-resolution. Output precision is fixed at `fp32`. Both FP32 and INT8 model variants are supported. + +**Legacy integer path** (`model_type=2`): +```bash +cd /ivsr_ffmpeg_plugin/ffmpeg +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=2:device= \ + -pix_fmt yuv420p test_out.mp4 ``` -- Command sample to run Enhanced EDSR inference, the input pixel format supported by the model is `rgb24`. + +**JSON config path** (equivalent): +```bash +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_config=/ivsr_ffmpeg_plugin/model_configs/edsr_config.json:device= \ + -pix_fmt yuv420p test_out.mp4 ``` + +### Custom VSR (`model_type=3`) +Input pixel format: `yuv420p` (Y-channel only). Single-frame inference on the luma plane. + +**Legacy integer path** (`model_type=3`): +```bash cd /ivsr_ffmpeg_plugin/ffmpeg -./ffmpeg -i -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=1:device=:model_type=2:normalize_factor=255.0 -pix_fmt yuv420p test_out.mp4 +./ffmpeg -i \ + -vf format=yuv420p,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=3:device= \ + -pix_fmt yuv420p test_out.mp4 ``` -- Command sample to run CUSTOM VSR inference. Note the input pixel format supported by this model is `yuv420p`, and its input shape is `[1, (Y channel)1, H, W]`, output shape is `[1, 1, 2xH, 2xW]`. + +**JSON config path** (equivalent): +```bash +./ffmpeg -i \ + -vf format=yuv420p,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_config=/ivsr_ffmpeg_plugin/model_configs/custvsr_config.json:device= \ + -pix_fmt yuv420p test_out.mp4 ``` + +### TSENet (`model_type=4`) +Input pixel format: `rgb24`. Uses a 3-frame sliding window with first-frame duplication priming. Input is passed as raw `[0, 255]` float32; output is rescaled from `[0, 1]` to `[0, 255]`. + +**Legacy integer path** (`model_type=4`): +```bash cd /ivsr_ffmpeg_plugin/ffmpeg -./ffmpeg -i -vf format=yuv420p,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=1:nireq=1:device=CPU:model_type=3 -pix_fmt yuv420p test_out.mp4 +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=4:device= \ + -pix_fmt yuv420p test_out.mp4 ``` -- Command sample to run TSENet model, the input pixel format supported by the model is `rgb24`. + +**JSON config path** (equivalent): +```bash +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_config=/ivsr_ffmpeg_plugin/model_configs/tsenet_config.json:device= \ + -pix_fmt yuv420p test_out.mp4 ``` -cd /ivsr_ffmpeg_plugin/ffmpeg -./ffmpeg -i -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:nif=1:device=:model_type=4 -pix_fmt yuv420p test_out.mp4 + +### Additional Models + +The following models support both dispatch paths. For brevity, only the **legacy integer path** is shown. To use JSON config, replace `model_type=N` with `model_config=/model_configs/_config.json`. + +#### RIFE - Frame Interpolation (`model_type=5`) +```bash +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=5:device= \ + test_out.mp4 ``` + +#### SPAN - Single-Frame SR (`model_type=6`) +```bash +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=6:device= \ + -pix_fmt yuv420p test_out.mp4 +``` + +#### VideoSeal - Invisible Watermarking (`model_type=7`) +```bash +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=7:device= \ + test_out.mp4 +``` + +#### HDRTVNet-LE - HDR Local Enhancement (`model_type=8`) +```bash +./ffmpeg -i \ + -vf format=rgb24,dnn_processing=dnn_backend=ivsr:model=:input=input:output=output:model_type=8:device= \ + test_out.mp4 +``` + +--- + +## Migration Guide + +### For Existing Production Code +If your code currently uses `model_type=1..8`, **no changes are required**. Patch 0007 restores full backward compatibility. + +### For New Projects +We recommend using JSON config files for better flexibility: +```bash +# Instead of: model_type=2 +# Use: model_config=/path/to/model_configs/edsr_config.json +``` + +### Custom Model Variants +JSON configs enable runtime customization without code changes. Copy a canonical config and modify as needed: +```bash +cp model_configs/edsr_config.json custom_edsr.json +# Edit custom_edsr.json to change align, normalize_input, etc. +./ffmpeg ... model_config=custom_edsr.json ... +``` + +### Generating a `model_config` JSON from a new IR model +For a new OpenVINO IR model (not one of the 8 built-in ones), use `generate_ivsr_model_config.py` (in the project root) to produce a starting `model_config` JSON instead of writing one by hand: +```bash +cd +python3 generate_ivsr_model_config.py .xml -o custom_model_config.json +``` +It parses the IR's `Parameter`/`Result` layers to auto-detect `in_layout`, `nif`, `channel_divisor`, `in_precision`, `out_precision`, SR scale, `window_type`, and `align`, and reads the model's `.bin` weights to detect any baked-in pixel normalization (e.g. a `Divide(255)` or per-channel mean `Subtract`/`Add` sitting directly on the input) to infer `normalize_input`/`normalize_output`. Fields it cannot infer from the IR alone (e.g. `color_format_auto`, or `normalize_input`/`output` when no baked op is found) are flagged as unconfirmed and prompted for interactively — pass `--non-interactive` to accept the safe defaults instead. Always diff the generated JSON against the closest canonical config in `model_configs/` before using it. + +--- + +## Troubleshooting + +### "model_type out of range" Error +- **Cause**: Patch 0007 not applied +- **Fix**: Rebuild with `./build.sh --ov_version 2026.1` (applies all patches including 0007) + +### Different Output Between model_type and model_config +- **Expected**: Both should produce identical output for the same model +- **Debug**: Check that `model_table[N]` matches the canonical JSON config +- **Verify**: `md5sum` the output files from both paths + +--- + +## See Also + +- **Patch Documentation**: `PATCH_0007_README.md` - Details on backward compatibility +- **Build Guide**: `PATCH_APPLIED.md` - Build and verification instructions +- **Model Configs**: `model_configs/*.json` - Canonical configuration files +- **Template**: `ivsr_model_config.template.json` - All available config fields +- **Config Generator**: `../generate_ivsr_model_config.py` - Auto-generates a `model_config` JSON from a new IR model diff --git a/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/Dockerfile b/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/Dockerfile index 7275194..4f22668 100644 --- a/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/Dockerfile +++ b/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/Dockerfile @@ -222,7 +222,10 @@ RUN filterdiff \ git apply --ignore-whitespace 0004-*.patch && \ git add -A && \ git apply --3way --whitespace=fix 0002-*.patch && \ - git apply --3way --whitespace=fix 0003-*.patch + git apply --3way --whitespace=fix 0003-*.patch && \ + git apply --3way --whitespace=fix 0005-*.patch && \ + git apply --3way --whitespace=fix 0006-*.patch && \ + git apply --3way --whitespace=fix 0007-*.patch RUN ./configure \ --extra-cflags=-fopenmp \ diff --git a/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/ov2024.5s.dockerfile b/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/ov2024.5s.dockerfile index 0e4ec73..b441f21 100644 --- a/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/ov2024.5s.dockerfile +++ b/ivsr_ffmpeg_plugin/dockerfiles/rockylinux9/ov2024.5s.dockerfile @@ -135,7 +135,10 @@ RUN filterdiff \ git apply --ignore-whitespace 0004-*.patch && \ git add -A && \ git apply --3way --whitespace=fix 0002-*.patch && \ - git apply --3way --whitespace=fix 0003-*.patch + git apply --3way --whitespace=fix 0003-*.patch && \ + git apply --3way --whitespace=fix 0005-*.patch && \ + git apply --3way --whitespace=fix 0006-*.patch && \ + git apply --3way --whitespace=fix 0007-*.patch RUN ./configure \ --extra-cflags=-fopenmp \ diff --git a/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/Dockerfile b/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/Dockerfile index fe0f98c..7cbded0 100644 --- a/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/Dockerfile +++ b/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/Dockerfile @@ -267,6 +267,7 @@ RUN apt-get update && \ libx265-dev \ libde265-dev \ libva-dev \ + patchutils \ && \ rm -rf /var/lib/apt/lists/* ENV LD_LIBRARY_PATH=${IVSR_SDK_DIR}/lib:/usr/local/lib:$LD_LIBRARY_PATH @@ -275,16 +276,24 @@ ARG FFMPEG_IVSR_SDK_PLUGIN_DIR=${IVSR_DIR}/ivsr_ffmpeg_plugin ARG FFMPEG_DIR=${FFMPEG_IVSR_SDK_PLUGIN_DIR}/ffmpeg ARG FFMPEG_REPO=https://github.com/FFmpeg/FFmpeg.git -ARG FFMPEG_VERSION=n7.1 +ARG FFMPEG_VERSION=n8.1 WORKDIR ${FFMPEG_DIR} RUN git clone ${FFMPEG_REPO} ${FFMPEG_DIR} && \ git checkout ${FFMPEG_VERSION} COPY ./ivsr_ffmpeg_plugin/patches/*.patch ${FFMPEG_DIR}/ -RUN { set -e; \ - for patch_file in $(find -iname "*.patch" | sort -n); do \ - echo "Applying: ${patch_file}"; \ - git am --whitespace=fix ${patch_file}; \ - done; } +RUN filterdiff \ + -x '*/configure' \ + -x '*/dnn_interface.c' \ + -x '*/swscale_unscaled.c' \ + 0001-*.patch | \ + git apply --3way --ignore-whitespace - && \ + git apply --ignore-whitespace 0004-*.patch && \ + git add -A && \ + git apply --3way --whitespace=fix 0002-*.patch && \ + git apply --3way --whitespace=fix 0003-*.patch && \ + git apply --3way --whitespace=fix 0005-*.patch && \ + git apply --3way --whitespace=fix 0006-*.patch && \ + git apply --3way --whitespace=fix 0007-*.patch RUN if [ -f "${CUSTOM_OV_INSTALL_DIR}/setvars.sh" ]; then \ . ${CUSTOM_OV_INSTALL_DIR}/setvars.sh ; \ diff --git a/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5.dockerfile b/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5.dockerfile index 980f3b0..efaae99 100644 --- a/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5.dockerfile +++ b/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5.dockerfile @@ -147,21 +147,30 @@ RUN apt-get update && \ apt-get install -y --no-install-recommends \ ca-certificates tar g++ wget pkg-config nasm yasm libglib2.0-dev flex bison gobject-introspection libgirepository1.0-dev \ python3-dev libx11-dev libxv-dev libxt-dev libasound2-dev libpango1.0-dev libtheora-dev libvisual-0.4-dev libgl1-mesa-dev \ - libcurl4-gnutls-dev librtmp-dev libx264-dev libx265-dev libde265-dev libva-dev && \ + libcurl4-gnutls-dev librtmp-dev libx264-dev libx265-dev libde265-dev libva-dev patchutils && \ rm -rf /var/lib/apt/lists/* # FFmpeg setup and build ARG FFMPEG_REPO=https://github.com/FFmpeg/FFmpeg.git -ARG FFMPEG_VERSION=n7.1 +ARG FFMPEG_VERSION=n8.1 ARG FFMPEG_IVSR_SDK_PLUGIN_DIR=${WORKSPACE}/ivsr/ivsr_ffmpeg_plugin WORKDIR ${FFMPEG_IVSR_SDK_PLUGIN_DIR}/ffmpeg RUN git clone ${FFMPEG_REPO} . && \ git checkout ${FFMPEG_VERSION} COPY ./ivsr_ffmpeg_plugin/patches/*.patch ./ -RUN for patch_file in $(find -iname "*.patch" | sort -n); do \ - echo "Applying: ${patch_file}"; \ - git am --whitespace=fix ${patch_file}; \ - done +RUN filterdiff \ + -x '*/configure' \ + -x '*/dnn_interface.c' \ + -x '*/swscale_unscaled.c' \ + 0001-*.patch | \ + git apply --3way --ignore-whitespace - && \ + git apply --ignore-whitespace 0004-*.patch && \ + git add -A && \ + git apply --3way --whitespace=fix 0002-*.patch && \ + git apply --3way --whitespace=fix 0003-*.patch && \ + git apply --3way --whitespace=fix 0005-*.patch && \ + git apply --3way --whitespace=fix 0006-*.patch && \ + git apply --3way --whitespace=fix 0007-*.patch RUN sed -i 's|-L${prefix}/runtime/3rdparty/tbb|-L${prefix}/runtime/3rdparty/tbb/lib|' \ ${CUSTOM_IE_DIR}/lib/intel64/pkgconfig/openvino.pc diff --git a/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5s.dockerfile b/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5s.dockerfile index cad8416..9e40b3c 100644 --- a/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5s.dockerfile +++ b/ivsr_ffmpeg_plugin/dockerfiles/ubuntu22/ov2024.5s.dockerfile @@ -79,7 +79,7 @@ RUN apt-get update && \ apt-get install -y --no-install-recommends \ ca-certificates tar g++ wget pkg-config nasm yasm libglib2.0-dev flex bison gobject-introspection libgirepository1.0-dev \ python3-dev libx11-dev libxv-dev libxt-dev libasound2-dev libpango1.0-dev libtheora-dev libvisual-0.4-dev libgl1-mesa-dev \ - libcurl4-gnutls-dev librtmp-dev libx264-dev libx265-dev libde265-dev libva-dev libtbb-dev && \ + libcurl4-gnutls-dev librtmp-dev libx264-dev libx265-dev libde265-dev libva-dev libtbb-dev patchutils && \ rm -rf /var/lib/apt/lists/* # Build iVSR SDK @@ -94,7 +94,7 @@ RUN mkdir -p build && cd build && \ # Build and install FFmpeg with libopenvino support # FFmpeg setup and build ARG FFMPEG_REPO=https://github.com/FFmpeg/FFmpeg.git -ARG FFMPEG_VERSION=n7.1 +ARG FFMPEG_VERSION=n8.1 ARG FFMPEG_IVSR_SDK_PLUGIN_DIR=${WORKSPACE}/ivsr/ivsr_ffmpeg_plugin WORKDIR ${FFMPEG_IVSR_SDK_PLUGIN_DIR}/ffmpeg RUN git config --global user.email "noname@example.com" && \ @@ -103,10 +103,19 @@ RUN git config --global user.email "noname@example.com" && \ git checkout ${FFMPEG_VERSION} COPY ./ivsr_ffmpeg_plugin/patches/*.patch ./ -RUN for patch_file in $(find -iname "*.patch" | sort -n); do \ - echo "Applying: ${patch_file}"; \ - git am --whitespace=fix ${patch_file}; \ - done +RUN filterdiff \ + -x '*/configure' \ + -x '*/dnn_interface.c' \ + -x '*/swscale_unscaled.c' \ + 0001-*.patch | \ + git apply --3way --ignore-whitespace - && \ + git apply --ignore-whitespace 0004-*.patch && \ + git add -A && \ + git apply --3way --whitespace=fix 0002-*.patch && \ + git apply --3way --whitespace=fix 0003-*.patch && \ + git apply --3way --whitespace=fix 0005-*.patch && \ + git apply --3way --whitespace=fix 0006-*.patch && \ + git apply --3way --whitespace=fix 0007-*.patch RUN ./configure \ --enable-gpl \ diff --git a/ivsr_ffmpeg_plugin/generate_ivsr_model_config.py b/ivsr_ffmpeg_plugin/generate_ivsr_model_config.py new file mode 100644 index 0000000..31cdd05 --- /dev/null +++ b/ivsr_ffmpeg_plugin/generate_ivsr_model_config.py @@ -0,0 +1,1273 @@ +#!/usr/bin/env python3 +""" +generate_ivsr_model_config.py +───────────────────────────── +Auto-generate an iVSR JSON model config file from an OpenVINO IR .xml file. + +The tool reads the Parameter (input) and Result (output) layers from the IR, +infers as many config fields as possible from the tensor shapes and types, +always displays a full IR analysis summary with warnings, and interactively +prompts only for the fields that cannot be determined from the model structure. + +Usage +----- + # Recommended: interactive mode + python3 generate_ivsr_model_config.py model.xml + + # Fully automatic (for CI / scripted pipelines) + python3 generate_ivsr_model_config.py model.xml --non-interactive + + # Override output path and model name + python3 generate_ivsr_model_config.py model.xml -o configs/my_model.json -n "MyModel" + + # Validate the generated config + jsonschema -i my_model_config.json ivsr_model_config.schema.json + +JSON fields produced +-------------------- + name Human-readable label (log messages) + nif Frames consumed per inference call + align Input W/H alignment padding (0 = none) + in_layout Input tensor layout: NCHW | NHWC | NFHWC + in_precision Input element type: f32 | u8 | u16 | null + out_layout Output tensor layout + out_precision Output element type: fp32 | u8 | u16 | null + model_color SDK color space: RGB | I420_Three_Planes | null + out_order Output channel order: RGB | BGR | NONE + window_type Frame queuing: single | sliding | in_queue + window_init_dup Duplicate first frame for sliding window prime + normalize_input Divide uint8 pixels by 255 before float packing + normalize_output Multiply float output by 255 after inference + output_passthrough_dims Output W/H == input W/H (passthrough models) + out_precision_depth_derived Derive output precision from frame bit depth +""" + +import argparse +import json +import os +import struct +import sys +import xml.etree.ElementTree as ET +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + + +# ───────────────────────────────────────────────────────────────────────────── +# Section 1: OpenVINO IR XML parsing +# ───────────────────────────────────────────────────────────────────────────── + +def _parse_dims(port_el: ET.Element) -> List[int]: + """Return a list of ints from children; -1 for dynamic/unknown dims.""" + dims = [] + for d in port_el.findall("dim"): + try: + dims.append(int(d.text.strip())) + except (ValueError, AttributeError): + dims.append(-1) + return dims + + +def parse_ir(xml_path: str) -> Tuple[List[Dict], List[Dict]]: + """ + Parse an OpenVINO IR .xml and return: + inputs: list of {"name", "shape", "element_type"} + outputs: list of {"name", "shape", "precision"} + + Works with OpenVINO IR version 10 and 11 (opset-based IR). + """ + tree = ET.parse(xml_path) + root = tree.getroot() # + + # Build layer-id → element map for cross-referencing edges + layers_by_id: Dict[str, ET.Element] = {} + for layer in root.iter("layer"): + lid = layer.get("id") + if lid is not None: + layers_by_id[lid] = layer + + # Build edge map: (to_layer_id, to_port_id) → (from_layer_id, from_port_id) + edges: Dict[Tuple[str, str], Tuple[str, str]] = {} + for edge in root.iter("edge"): + key = (edge.get("to-layer"), edge.get("to-port")) + edges[key] = (edge.get("from-layer"), edge.get("from-port")) + + inputs: List[Dict] = [] + outputs: List[Dict] = [] + + for layer in root.iter("layer"): + ltype = layer.get("type") + lid = layer.get("id", "") + lname = layer.get("name", "") + + # ── Input layer ────────────────────────────────────────────────────── + if ltype == "Parameter": + data_el = layer.find("data") + # OV IR stores shape/element_type as attributes on + shape_str = data_el.get("shape", "") if data_el is not None else "" + element_type = ( + data_el.get("element_type", "f32") if data_el is not None else "f32" + ) + + # Parse shape from the attribute (most reliable) + if shape_str: + shape = [] + for s in shape_str.replace(" ", "").split(","): + try: + shape.append(int(s)) + except ValueError: + shape.append(-1) + else: + # Fallback: read from output port dims + out_port = None + output_el = layer.find("output") + if output_el is not None: + out_port = output_el.find("port") + shape = _parse_dims(out_port) if out_port is not None else [] + + # Extract the tensor name from the output port's "names" attribute + out_port = None + output_el = layer.find("output") + if output_el is not None: + out_port = output_el.find("port") + tensor_name = ( + out_port.get("names", lname) if out_port is not None else lname + ) + + inputs.append( + {"name": tensor_name, "shape": shape, "element_type": element_type} + ) + + # ── Output layer ───────────────────────────────────────────────────── + elif ltype == "Result": + inp_el = layer.find("input") + port = inp_el.find("port") if inp_el is not None else None + precision = port.get("precision", "FP32") if port is not None else "FP32" + shape = _parse_dims(port) if port is not None else [] + + out_names = layer.get("output_names", lname) + + # If Result port dims are all dynamic, trace back through edges to get + # the actual shape from the feeding layer's output port. + if not shape or all(d <= 0 for d in shape): + from_info = edges.get((lid, "0")) + if from_info: + from_layer = layers_by_id.get(from_info[0]) + if from_layer is not None: + out_el = from_layer.find("output") + if out_el is not None: + for p in out_el.findall("port"): + if p.get("id") == from_info[1]: + candidate = _parse_dims(p) + if candidate and any(d > 0 for d in candidate): + shape = candidate + precision = p.get("precision", precision) + break + + outputs.append({"name": out_names, "shape": shape, "precision": precision}) + + return inputs, outputs + + +def scan_early_ops(xml_path: str, max_ops: int = 20) -> Dict[str, bool]: + """ + Fast heuristic scan of the IR's Parameter/Result layers. + Returns a dict of clues: + "multi_input" — model has more than one Parameter (conditioning inputs) + "multi_output" — model has more than one Result layer + + NOTE: an earlier version of this scan also flagged "has_early_divide" / + "has_early_subtract" whenever ANY Divide/Multiply/Subtract/Add op appeared + in the first `max_ops` layers, and used that to guess normalize_input. + That signal is unreliable: it matches unrelated ops (Conv bias-Add, + floor_divide shape arithmetic) and was empirically wrong for SPAN/HDRTVNet. + See detect_input_normalization_constant() for the constant-value-based + replacement, which only fires on the exact op feeding the Parameter. + """ + tree = ET.parse(xml_path) + root = tree.getroot() + + clues: Dict[str, bool] = { + "multi_input": False, + "multi_output": False, + } + + param_count = sum(1 for layer in root.iter("layer") if layer.get("type") == "Parameter") + result_count = sum(1 for layer in root.iter("layer") if layer.get("type") == "Result") + + clues["multi_input"] = param_count > 1 + clues["multi_output"] = result_count > 1 + return clues + + +# Element types (as they appear on for Const layers) +# that we know how to unpack as native floats via struct, plus (struct_char, byte_size). +_CONST_FLOAT_FORMATS = { + "f32": ("f", 4), "fp32": ("f", 4), + "f16": ("e", 2), "fp16": ("e", 2), + "f64": ("d", 8), "fp64": ("d", 8), +} + +# Ops whose second operand (if a small Const) reveals the input's expected +# pixel-value range. +_NORMALIZATION_OP_TYPES = {"Divide", "Multiply", "Subtract", "Add"} +_PASSTHROUGH_OP_TYPES = {"Convert", "Reshape", "Squeeze", "Unsqueeze"} + + +def _resolve_to_const( + layers_by_id: Dict[str, ET.Element], + edges: Dict[Tuple[str, str], Tuple[str, str]], + layer_id: str, + port: str, + max_hops: int = 3, +) -> Optional[str]: + """Follow a (layer_id, port) input back through Convert/Reshape ops to find + the id of the Const layer that ultimately feeds it, or None if it doesn't + terminate in a Const within max_hops.""" + cur_layer, cur_port = layer_id, port + for _ in range(max_hops): + from_info = edges.get((cur_layer, cur_port)) + if not from_info: + return None + from_layer, _from_port = from_info + layer = layers_by_id.get(from_layer) + if layer is None: + return None + ltype = layer.get("type") + if ltype == "Const": + return from_layer + if ltype in _PASSTHROUGH_OP_TYPES: + cur_layer, cur_port = from_layer, "0" + continue + return None + return None + + +def detect_input_normalization_constant( + xml_path: str, +) -> Tuple[Optional[str], Optional[List[float]]]: + """ + Find the literal constant baked into any Divide/Multiply/Subtract/Add op + that sits directly on the input path (Parameter -> [Convert/Reshape]* -> + op), and return (op_type, values) read straight from the sibling .bin + weights file. Returns (None, None) if no such op/constant is found. + + This targets the *exact* op consuming the model's raw pixel tensor, unlike + a generic "any Divide/Subtract in the first N layers" scan — which also + matches unrelated ops (conv bias-Add, floor_divide shape math) and gives + false positives. Verified empirically against shipped models: + VideoSeal: Parameter -> Divide(255.0) -> raw [0,255] scale + SPAN: Parameter -> Add(-0.449,-0.437,-0.404) -> ImageNet-style + per-channel mean + """ + try: + tree = ET.parse(xml_path) + except (ET.ParseError, OSError): + return None, None + root = tree.getroot() + + layers_by_id = {l.get("id"): l for l in root.iter("layer")} + edges: Dict[Tuple[str, str], Tuple[str, str]] = {} + consumers: Dict[str, List[Tuple[str, str]]] = {} + for e in root.iter("edge"): + to_layer, to_port = e.get("to-layer"), e.get("to-port") + from_layer, from_port = e.get("from-layer"), e.get("from-port") + edges[(to_layer, to_port)] = (from_layer, from_port) + consumers.setdefault(from_layer, []).append((to_layer, to_port)) + + param_id = next((l.get("id") for l in root.iter("layer") if l.get("type") == "Parameter"), None) + if param_id is None: + return None, None + + # BFS from the Parameter through passthrough ops (Convert/Reshape/...) only, + # looking for the first real math op that could be a normalization step. + frontier = [param_id] + visited = set() + op_id = op_type = None + for _ in range(4): + next_frontier = [] + for lid in frontier: + if lid in visited: + continue + visited.add(lid) + for to_layer, _to_port in consumers.get(lid, []): + clayer = layers_by_id.get(to_layer) + if clayer is None: + continue + ctype = clayer.get("type") + if ctype in _NORMALIZATION_OP_TYPES: + op_id, op_type = to_layer, ctype + break + if ctype in _PASSTHROUGH_OP_TYPES: + next_frontier.append(to_layer) + if op_id: + break + if op_id: + break + frontier = next_frontier + + if not op_id: + return None, None + + const_layer_id = _resolve_to_const(layers_by_id, edges, op_id, "0") or \ + _resolve_to_const(layers_by_id, edges, op_id, "1") + if not const_layer_id: + return None, None + + data_el = layers_by_id[const_layer_id].find("data") + if data_el is None or data_el.get("offset") is None: + return None, None + try: + offset = int(data_el.get("offset")) + size = int(data_el.get("size")) + except (TypeError, ValueError): + return None, None + et = (data_el.get("element_type") or "f32").lower() + + fmt = _CONST_FLOAT_FORMATS.get(et) + # Only scalar/small per-channel constants are normalization values; a huge + # tensor here means we followed the wrong operand (e.g. conv weights). + if not fmt or size <= 0 or size > 256 or size % fmt[1] != 0: + return None, None + + bin_path = Path(xml_path).with_suffix(".bin") + if not bin_path.is_file(): + return None, None + try: + with open(bin_path, "rb") as f: + f.seek(offset) + raw = f.read(size) + n = size // fmt[1] + values = list(struct.unpack(f"<{n}{fmt[0]}", raw)) + except (OSError, struct.error): + return None, None + + return op_type, values + + +def classify_input_normalization(op_type: Optional[str], values: Optional[List[float]]) -> Optional[str]: + """ + Classify a detected input-side constant as: + "raw_255_scale" — Divide by ~255 (or Multiply by ~1/255): the model + internally rescales raw [0,255] pixels itself, so + the FFmpeg plugin must NOT pre-divide -> normalize_input=False + "meanstd_normalize" — Subtract/Add/Divide/Multiply by a small (<3) value: + ImageNet-style per-channel mean/std centering, which + only makes sense on already-[0,1] data -> normalize_input=True + None — no confident classification (value out of range, or + nothing detected) + """ + if op_type is None or not values: + return None + abs_vals = [abs(v) for v in values] + vmin, vmax = min(abs_vals), max(abs_vals) + + if op_type == "Divide" and 200.0 <= vmin <= vmax <= 300.0: + return "raw_255_scale" + if op_type == "Multiply" and (1 / 300.0) <= vmin <= vmax <= (1 / 200.0): + return "raw_255_scale" + if op_type in ("Subtract", "Add", "Divide", "Multiply") and 0.01 <= vmin <= vmax <= 3.0: + return "meanstd_normalize" + return None + + + +# ───────────────────────────────────────────────────────────────────────────── +# Section 2: Heuristic analysis of tensor shapes +# ───────────────────────────────────────────────────────────────────────────── + +def detect_layout(shape: List[int]) -> Tuple[str, int]: + """ + Given a tensor shape [N, ...], return (layout_string, channels_per_frame). + + Layout detection heuristic: + - 4-D NCHW: shape[1] (C) is small (≤16), shape[2]/shape[3] (H/W) are large + - 4-D NHWC: shape[3] (C) is small (≤16), shape[1]/shape[2] (H/W) are large + - 5-D NFHWC: shape[4] (C) is small; shape[1] (F) is the frame count + - 5-D NFCHW: shape[2] (C) is small; shape[1] (F) is the frame count + + Returns layout string and the channel-per-frame count (may be -1 if dynamic). + """ + CHANNEL_THRESHOLD = 16 # channels are always ≤ 16 for RGB/YUV/RGBA frames + + if len(shape) == 4: + c_pos1 = shape[1] # candidate C for NCHW + c_pos3 = shape[3] # candidate C for NHWC + + # Both dynamic: can't distinguish — default to NCHW (more common for SR) + if c_pos1 <= 0 and c_pos3 <= 0: + return "NCHW", -1 + + # Clear NHWC signal: last dim is small and positive + if c_pos3 > 0 and c_pos3 <= CHANNEL_THRESHOLD: + if c_pos1 <= 0 or c_pos1 > CHANNEL_THRESHOLD: + return "NHWC", c_pos3 + + # Clear NCHW signal: second dim is small and positive + if c_pos1 > 0 and c_pos1 <= CHANNEL_THRESHOLD: + return "NCHW", c_pos1 + + # Ambiguous (e.g., square shape) — default to NCHW + return "NCHW", c_pos1 + + elif len(shape) == 5: + # NFHWC: [N, F, H, W, C] — shape[4] is channels + # NFCHW: [N, F, C, H, W] — shape[2] is channels + c_pos4 = shape[4] + c_pos2 = shape[2] + if c_pos4 > 0 and c_pos4 <= CHANNEL_THRESHOLD: + return "NFHWC", c_pos4 + if c_pos2 > 0 and c_pos2 <= CHANNEL_THRESHOLD: + return "NFCHW", c_pos2 + return "NFHWC", c_pos4 + + elif len(shape) == 3: + # N, H, W (grayscale single frame) or N, C, HW — rare + return "NHW", 1 + + # Fallback + return "NCHW", -1 + + +def detect_nif(layout: str, shape: List[int]) -> Tuple[int, int]: + """ + Return (nif, channels_per_frame) based on layout and input shape. + + For NCHW: total_channels = shape[1]; nif = total_channels / 3 (or /1 for gray) + For NHWC: total_channels = shape[3]; same logic + For NFHWC: shape[1] = F = nif directly; shape[4] = channels_per_frame + For NFCHW: shape[1] = F = nif; shape[2] = channels_per_frame + """ + if layout in ("NFHWC",): + f = shape[1] if len(shape) > 1 and shape[1] > 0 else 1 + c = shape[4] if len(shape) > 4 and shape[4] > 0 else 3 + return f, c + + if layout in ("NFCHW",): + f = shape[1] if len(shape) > 1 and shape[1] > 0 else 1 + c = shape[2] if len(shape) > 2 and shape[2] > 0 else 3 + return f, c + + if layout == "NCHW": + total_ch = shape[1] if len(shape) > 1 and shape[1] > 0 else 3 + elif layout == "NHWC": + total_ch = shape[3] if len(shape) > 3 and shape[3] > 0 else 3 + else: + total_ch = 3 + + # Probe common per-frame channel counts (RGB=3, grayscale=1, RGBA=4) + for ch_per_frame in (3, 1, 4, 2): + if total_ch % ch_per_frame == 0: + return total_ch // ch_per_frame, ch_per_frame + + return 1, total_ch + + +def detect_channel_divisor(layout: str, nif: int) -> int: + """ + Return the channel_divisor value required by the JSON config dispatch + (dnn_backend_ivsr.c: parse_model_config_json / fill_model_input_ivsr). + + When multiple frames are stacked along the channel axis (NCHW/NHWC with + nif > 1, e.g. RIFE nif=2 -> 6ch, TSENet nif=3 -> 9ch), the SDK reports the + *total* stacked channel count to FFmpeg unless channel_divisor tells it to + divide back down to the per-frame count (3 for RGB). Omitting this field + silently defaults to 1 in the C parser, which breaks tensor/channel + reporting for any stacked-channel multi-frame model. + + NFHWC/NFCHW layouts already carry frames on a separate axis (BasicVSR- + style, window_type=in_queue), so no channel division is needed there. + """ + if nif > 1 and layout in ("NCHW", "NHWC"): + return nif + return 1 + + +def map_element_type(et: str) -> str: + """ + Map OV IR element_type string → iVSR in_precision string. + + OV uses lowercase element_type on : "f32", "u8", "i64", "f16", etc. + iVSR uses "f32", "u8", "u16" (or null for depth-derived). + """ + et = et.lower().replace(" ", "") + if et in ("f32", "fp32", "float32"): + return "f32" + if et in ("u8", "uint8"): + return "u8" + if et in ("u16", "uint16"): + return "u16" + if et in ("f16", "fp16", "float16"): + # f16 models run as f32 in the iVSR/OV runtime + return "f32" + if et in ("bf16", "bfloat16"): + return "f32" + # Integer types (i32, i64) are not standard model inputs — fallback safely + return "f32" + + +def map_output_precision(prec: str) -> str: + """Map OV port precision string → iVSR out_precision string.""" + p = prec.upper().replace(" ", "") + if p in ("FP32", "F32", "FLOAT32"): + return "fp32" + if p in ("U8", "UINT8"): + return "u8" + if p in ("U16", "UINT16"): + return "u16" + if p in ("FP16", "F16"): + return "fp32" # cast to fp32 at runtime + return "fp32" + + +def detect_scale_from_pixel_shuffle(xml_path: str) -> Tuple[int, int]: + """ + Detect SR upscale factor from the pixel_shuffle Reshape_1 pattern in OV IR. + + PyTorch pixel_shuffle(x, r) is lowered to: + Reshape_1: [N, C*r*r, H, W] → [N, C, r, r, H, W] + Transpose: [N, C, r, r, H, W] → [N, C, H, r, W, r] + Reshape_2: [N, C, H, r, W, r] → [N, C, H*r, W*r] + + The scale r is always a literal integer in Reshape_1's output dims at + positions [2] and [3] (the two hardcoded 'r' dims). It is readable + directly from the XML without touching the .bin weight file. + + Returns (scale_h, scale_w) — typically (r, r) — or (1, 1) if not found. + """ + try: + tree = ET.parse(xml_path) + root = tree.getroot() + for layer in root.iter("layer"): + name = layer.get("name", "") + if "pixel_shuffle" not in name: + continue + if layer.get("type") != "Reshape": + continue + # Reshape_1 has 6-D output: [N, C, r, r, H, W] + out_port = layer.find(".//output/port") + if out_port is None: + continue + dims = [int(d.text) for d in out_port.findall("dim")] + if len(dims) == 6 and dims[2] > 0 and dims[3] > 0: + return dims[2], dims[3] + except Exception: + pass + return 1, 1 + + +def detect_sr_scale(in_shape: List[int], out_shape: List[int]) -> Tuple[int, int]: + """ + Detect super-resolution upscale factor by comparing spatial dims. + Returns (scale_h, scale_w); both 1 means passthrough / same-size. + Handles dynamic (-1) shapes conservatively. + """ + if not in_shape or not out_shape: + return 1, 1 + + def _spatial_dims(shape): + """Return (H_idx, W_idx) candidates for NCHW and NHWC.""" + if len(shape) >= 4: + return [(2, 3), (1, 2)] # NCHW first, then NHWC + return [] + + for hi, wi in _spatial_dims(in_shape): + if hi >= len(in_shape) or wi >= len(in_shape): + continue + if hi >= len(out_shape) or wi >= len(out_shape): + continue + ih, iw = in_shape[hi], in_shape[wi] + oh, ow = out_shape[hi], out_shape[wi] + if ih > 0 and iw > 0 and oh > 0 and ow > 0: + sh = oh // ih if oh > ih else 1 + sw = ow // iw if ow > iw else 1 + return sh, sw + + return 1, 1 + + +def infer_window_type(nif: int, layout: str) -> Tuple[str, bool]: + """ + Infer window_type and window_init_dup from nif and layout. + + Rules: + nif == 1 → "single" + nif == 2, NCHW → "sliding" (RIFE-style, no dup needed) + nif >= 3, NFHWC/NFCHW → "in_queue" (BasicVSR-style batched read) + nif == 3, NCHW → "sliding" + init_dup=True (TSENet-style) + nif >= 3, other → "sliding" + init_dup=True (conservative) + """ + if nif <= 1: + return "single", False + if nif == 2: + return "sliding", False + if "NFHWC" in layout or "NFCHW" in layout: + return "in_queue", False + # nif ≥ 3 with stacked-channel layout (TSENet-style) + return "sliding", True + + +def infer_align(nif: int, in_precision: str, layout: str) -> int: + """ + Infer alignment padding requirement from known model patterns. + + RIFE (nif=2, f32 NCHW) → 128 px (grid_sample requires power-of-2 resolution) + BasicVSR (NFHWC/NFCHW) → 32 px + All others → 0 (no padding) + + Single-channel Y-plane models (VideoProc, CustVSR) cannot be told apart + from tensor shape alone: canonical VideoProc needs align=64, canonical + CustVSR needs align=0. This field is left at 0 by default for those and + MUST be confirmed manually (see the interactive 'align' prompt). + """ + if nif == 2 and in_precision == "f32" and "NCHW" in layout: + return 128 + if "NFHWC" in layout or "NFCHW" in layout: + return 32 + return 0 + + +def infer_normalize(in_precision: str, norm_signal: Optional[str]) -> Tuple[bool, bool]: + """ + Infer normalize_input and normalize_output. + + If in_precision is "f32", the C code packs pixels as float and the question + is whether it should divide by 255 (RIFE/SPAN/HDRTVNet-LE convention, model + trained on [0,1]) or pass raw [0,255] (VideoProc/EDSR/CustVSR/TSENet-in/ + VideoSeal convention, model trained on [0,255]). + + norm_signal comes from classify_input_normalization() — the literal constant + baked into whatever Divide/Multiply/Subtract/Add op sits directly on the + Parameter's input path (read from the .bin weights, not just op-type + pattern matching): + "raw_255_scale" — confirmed Divide/Multiply by ~255: the model itself + rescales raw [0,255] pixels -> normalize_input=False + "meanstd_normalize" — confirmed small (<3) mean/std constant: only valid + on already-[0,1] data -> normalize_input=True + None — no op sits on the raw input at all (goes straight + into e.g. Convolution/Slice). This is genuinely + indistinguishable from the IR: HDRTVNet-LE/RIFE + (True) and EDSR/TSENet-input (False) have an + identical "no preprocessing op" structure. Default + to False (matches the corrected C-side default, + patch 0006) and require manual confirmation. + If in_precision is "u8" or "u16": + - The generic SDK path handles normalization; normalize_input applies only + to the pack_input_window() custom float path. Irrelevant, but we keep + it False to avoid confusion. + """ + if in_precision in ("u8", "u16"): + return False, False + + if norm_signal == "raw_255_scale": + return False, False + if norm_signal == "meanstd_normalize": + return True, True + # No confident signal in the IR — safe majority default, must be confirmed. + return False, False + + +# ───────────────────────────────────────────────────────────────────────────── +# Section 3: Interactive prompts +# ───────────────────────────────────────────────────────────────────────────── + +def _ask(prompt: str, default: Any, choices: Optional[List[str]] = None) -> str: + """Print a prompt, return user input or default on empty/EOF.""" + if choices: + choices_display = "/".join( + f"[{c}]" if str(c) == str(default) else str(c) for c in choices + ) + full_prompt = f" {prompt} ({choices_display}): " + else: + full_prompt = f" {prompt} [default: {default}]: " + + try: + ans = input(full_prompt).strip() + except (EOFError, KeyboardInterrupt): + print() + return str(default) + + return ans if ans else str(default) + + +def _ask_bool(prompt: str, default: bool) -> bool: + ans = _ask(prompt, "true" if default else "false", choices=["true", "false"]) + return ans.lower() in ("true", "1", "yes", "y") + + +def _ask_int(prompt: str, default: int) -> int: + while True: + ans = _ask(prompt, str(default)) + try: + return int(ans) + except ValueError: + print(f" ↳ '{ans}' is not an integer — please try again.") + + +# ───────────────────────────────────────────────────────────────────────────── +# Section 4: Parameter-by-parameter documentation (deep dive) +# ───────────────────────────────────────────────────────────────────────────── + +PARAM_DOCS = { + "name": ( + "Human-readable label embedded in FFmpeg log messages.\n" + " Used by: av_log() calls inside ff_dnn_load_model_ivsr() and\n" + " fill_model_input_ivsr() for error/debug output.\n" + " No functional effect on inference." + ), + "nif": ( + "Number of input frames consumed per inference call.\n" + " Stored in IVSRModel.nif (overrides SDK-reported value when > 0).\n" + " Used by: fill_model_input_ivsr() — controls how many frames are\n" + " read from task->in_queue (in_queue mode) or frame_queue (sliding).\n" + " Also passed to pack_input_* functions as m->nif.\n" + " Set to 1 for single-frame models, 2 for RIFE, 3 for TSENet/BasicVSR." + ), + "channel_divisor": ( + "Divides the SDK-reported tensor channel count so FFmpeg sees a\n" + " standard 3-channel (RGB) frame instead of nif stacked frames.\n" + " Parsed in parse_model_config_json() (MKEY(\"channel_divisor\")) and\n" + " applied in fill_model_input_ivsr() / get_input_ivsr():\n" + " int div = get_model_desc(ivsr_model)->channel_divisor;\n" + " if (div > 1) { input.dims[channel_idx] /= div; input.channels /= div; }\n" + " Required whenever nif > 1 frames are stacked on the channel axis\n" + " (NCHW/NHWC layout): RIFE (nif=2) uses 2, TSENet (nif=3) uses 3.\n" + " Defaults to 1 in the C parser when omitted — leaving it out for a\n" + " stacked-channel model silently reports the wrong channel count.\n" + " 1 for all single-frame or NFHWC/NFCHW (in_queue) models." + ), + "align": ( + "Rounds input W and H up to the nearest multiple of this value.\n" + " Applied in ff_dnn_load_model_ivsr():\n" + " if (a > 0) { frame_h = (frame_h + a-1)/a*a; ... }\n" + " The aligned resolution is passed to ivsr_init() as RESHAPE_SETTINGS.\n" + " The difference (padded_size - actual_size) is zero-filled by\n" + " set_padding_value() in fill_model_input_ivsr().\n" + " 0 = no padding. RIFE needs 128 (grid_sample). BasicVSR needs 32." + ), + "in_layout": ( + "Tensor memory layout passed to the iVSR SDK as INPUT_TENSOR_DESC_SETTING.\n" + " Applied in ff_dnn_load_model_ivsr():\n" + " strcpy(input_tensor_desc_set.layout, md->in_layout);\n" + " The SDK uses this to configure the OV inference request input.\n" + " Also used by set_dnndata_info() to populate DNNData.layout,\n" + " which controls NCHW↔NHWC conversion in fill_model_input_ivsr().\n" + " NCHW: standard for CNN models (PyTorch default)\n" + " NHWC: TensorFlow-style; also used by some ONNX exports\n" + " NFHWC: multi-frame batched layout (BasicVSR)" + ), + "in_precision": ( + "Input tensor element type passed to the iVSR SDK.\n" + " Applied in ff_dnn_load_model_ivsr():\n" + " if (md->in_precision) strcpy(input_tensor_desc_set.precision, md->in_precision);\n" + " null → depth-derived: u8 for 8-bit frames, u16 for 10/16-bit.\n" + " f32 → model expects float32 input; C code uses pack_input_window()\n" + " which optionally divides by 255 (normalize_input flag).\n" + " u8/u16 → SDK handles normalization via scale field in tensor_desc." + ), + "out_layout": ( + "Output tensor layout passed to the iVSR SDK as OUTPUT_TENSOR_DESC_SETTING.\n" + " Applied in ff_dnn_load_model_ivsr():\n" + " if (md->out_layout) strcpy(output_tensor_desc_set.layout, md->out_layout);\n" + " Used by set_dnndata_info() in infer_completion_callback() to populate\n" + " DNNData.layout for the output tensor.\n" + " If NCHW and the generic path is used, convert_nchw_to_nhwc() is called\n" + " before ff_proc_from_dnn_to_frame()." + ), + "out_precision": ( + "Output tensor element type passed to the iVSR SDK.\n" + " Applied in ff_dnn_load_model_ivsr():\n" + " if (md->out_precision) strcpy(output_tensor_desc_set.precision, md->out_precision);\n" + " null → defaults to fp32.\n" + " Overridden by out_precision_depth_derived when that flag is true." + ), + "model_color": ( + "Color space string set on input_tensor_desc_set.model_color_format.\n" + " Applied in ff_dnn_load_model_ivsr():\n" + " if (md->model_color) strcpy(input_tensor_desc_set.model_color_format, ...);\n" + " Tells the iVSR SDK what color space the model was trained on, so it\n" + " can perform any needed color conversion internally.\n" + " RGB → standard RGB (RIFE, SPAN, EDSR, VideoSeal)\n" + " I420_Three_Planes → YUV 4:2:0 planar (VideoProc, CustVSR)\n" + " null → color_format_auto logic kicks in:\n" + " 0=no override, 1=auto from pixel format, 2=always YUV" + ), + "out_order": ( + "DNNColorOrder for DNNData.order in infer_completion_callback().\n" + " Set as: output.order = get_model_desc(ivsr_model)->out_order;\n" + " Used by ff_proc_from_dnn_to_frame() (generic output path) to determine\n" + " the channel order when writing to the AVFrame.\n" + " RGB → output channels are R,G,B (most models)\n" + " BGR → output channels are B,G,R\n" + " NONE → no channel reordering (grayscale, or custom unpack_output handles it)" + ), + "window_type": ( + "Frame queuing strategy — controls how frames are fed to the model.\n" + " Dispatched in pack_input_window() via a switch statement:\n" + "\n" + " single:\n" + " No queue. One AVFrame in → immediate inference.\n" + " Used by SPAN, VideoSeal, EDSR, HDRTVNet++ LE.\n" + " pack_input_window() WINDOW_SINGLE branch runs.\n" + "\n" + " sliding:\n" + " Maintains an AVFifo (m->frame_queue) of the last nif frames.\n" + " Returns DNN_MORE_FRAMES until nif frames are queued.\n" + " After each inference, the oldest frame is popped (slide forward).\n" + " Used by RIFE (nif=2) and TSENet (nif=3).\n" + " window_init_dup=true duplicates the first frame to prime the queue\n" + " (so output starts at frame 1, not frame nif).\n" + "\n" + " in_queue:\n" + " Reads nif frames directly from task->in_queue (filled by the filter).\n" + " Routes to pack_input_basicvsr() internally.\n" + " Used by BasicVSR (nif=3, NFHWC layout)." + ), + "window_init_dup": ( + "Only meaningful when window_type == 'sliding'.\n" + " Checked in pack_input_window() WINDOW_SLIDING branch:\n" + " if (md->sliding_window_init_dup && m->sliding_window_frame_num == 0) {\n" + " // duplicate first frame to prime the window\n" + " }\n" + " true → frame 1 is duplicated so the window fills immediately,\n" + " and inference starts from the very first input frame.\n" + " TSENet behaviour: the [prev, curr] window is [frame1, frame1].\n" + " false → window fills naturally; first output is produced after nif\n" + " distinct frames have been received.\n" + " RIFE behaviour: first output is the interpolated frame between\n" + " frame 1 and frame 2." + ), + "normalize_input": ( + "Whether to divide uint8 pixel values by 255.0 before packing into float32.\n" + " Checked in pack_input_window() WINDOW_SINGLE branch:\n" + " if (md->normalize_input) {\n" + " dst[...] = row[...] / 255.0f; // [0,1] range\n" + " } else {\n" + " dst[...] = (float)row[...]; // [0,255] range\n" + " }\n" + " Only active when in_precision == f32 (custom pack_input_window path).\n" + " Default is False in the C parser (patch 0006) — the earlier True\n" + " default silently corrupted VideoProc/EDSR/CustVSR/VideoSeal output.\n" + " Detection: detect_input_normalization_constant() reads the actual\n" + " Divide/Multiply/Subtract/Add constant (if any) baked directly onto\n" + " the Parameter's input path from the .bin weights file:\n" + " constant ≈ 255 → raw [0,255] scale → false (confident)\n" + " constant ≈ 0.01-3.0 → mean/std normalize → true (confident)\n" + " no such op on raw input → no signal, defaults false (unconfirmed)\n" + " The 'no op' case is genuinely ambiguous from the IR alone: HDRTVNet-LE\n" + " (true) and EDSR (false) have an identical 'no preprocessing op' graph\n" + " shape, because the [0,1]/[0,255] convention is usually applied by\n" + " external preprocessing (e.g. torchvision.ToTensor()) that never gets\n" + " traced into the exported ONNX/IR graph — confirm manually in that case.\n" + " true → model trained on [0,1] input (RIFE, SPAN, HDRTVNet-LE)\n" + " false → model trained on [0,255] input (VideoProc, EDSR, CustVSR,\n" + " TSENet input, VideoSeal — majority of shipped models)" + ), + "normalize_output": ( + "Whether to multiply float32 output by 255 before clipping to uint8.\n" + " Checked in unpack_output_window():\n" + " if (md->normalize_output) {\n" + " pixel = (uint8_t)av_clip((int)(val * 255.0f + 0.5f), 0, 255);\n" + " } else {\n" + " pixel = (uint8_t)av_clip((int)(val + 0.5f), 0, 255);\n" + " }\n" + " Default is False in the C parser (patch 0006).\n" + " Auto-set alongside normalize_input from the same input-side constant\n" + " detection (meanstd_normalize implies both true); there is no separate\n" + " output-side constant scan, so this is only as confident as the input\n" + " detection was — always confirm output range against the canonical\n" + " shipped config for your model family when in doubt (e.g. TSENet needs\n" + " normalize_input=false / normalize_output=true, which no input-side\n" + " scan alone could ever derive).\n" + " true → model output is in [0,1] range (RIFE, SPAN, TSENet, HDRTVNet-LE)\n" + " false → model output is already in [0,255] range (VideoProc, EDSR,\n" + " CustVSR, VideoSeal)\n" + " Not required to match normalize_input — e.g. TSENet uses\n" + " normalize_input=false / normalize_output=true." + ), + "output_passthrough_dims": ( + "Report output W/H == input W/H regardless of the model's output tensor size.\n" + " Used in get_output_ivsr():\n" + " if (get_model_desc(ivsr_model)->output_passthrough_dims) {\n" + " *output_height = input_height;\n" + " *output_width = input_width;\n" + " }\n" + " Needed for models where the output tensor is the same resolution as\n" + " the input but the tensor metadata doesn't reflect the padded dimensions.\n" + " VideoProc uses this because its output is aligned/padded like the input.\n" + " false for all super-resolution models (output is larger than input)." + ), + "color_format_auto": ( + "Integer controlling how model_color_format is set when model_color is null.\n" + " Parsed in parse_model_config_json() and consumed in ff_dnn_load_model_ivsr():\n" + " if (md->model_color) → use model_color string directly (ignore this field)\n" + " else if (color_format_auto==1) → VideoProc-auto: 'RGB' or 'I420_Three_Planes'\n" + " based on the input pixel format flags.\n" + " else if (color_format_auto==2) → CustVSR-auto: always 'I420_Three_Planes'.\n" + " 0 (default) — model_color string is used; this field is irrelevant.\n" + " 1 — set for VideoProc (model_color must be null in JSON).\n" + " 2 — set for CustVSR (model_color must be null in JSON).\n" + " For all RGB super-resolution models (SPAN, EDSR, etc.), keep 0." + ), + "out_precision_depth_derived": ( + "Derive output tensor precision from the input frame's bit depth.\n" + " Checked in ff_dnn_load_model_ivsr():\n" + " if (get_model_desc(ivsr_model)->out_precision_depth_derived) {\n" + " if (depth == 8) strcpy(output_tensor_desc_set.precision, 'u8');\n" + " if (depth >= 10) strcpy(output_tensor_desc_set.precision, 'u16');\n" + " }\n" + " This overrides out_precision when true.\n" + " Used by EDSR which supports both 8-bit (u8 output) and 10/16-bit (u16)\n" + " via the same model weights, automatically selected at runtime.\n" + " false for almost all other models." + ), +} + + +def print_deep_dive(): + """Print the complete parameter reference.""" + print("\n" + "=" * 78) + print("iVSR JSON Config — Deep-dive parameter reference") + print("=" * 78) + for field, doc in PARAM_DOCS.items(): + print(f"\n [{field}]") + for line in doc.splitlines(): + print(f" {line}") + print() + + +# ───────────────────────────────────────────────────────────────────────────── +# Section 5: Main config generation +# ───────────────────────────────────────────────────────────────────────────── + +def generate_config( + xml_path: str, + name_override: Optional[str], + non_interactive: bool, +) -> Dict: + # ── Parse IR ───────────────────────────────────────────────────────────── + inputs, outputs = parse_ir(xml_path) + + if not inputs: + print("ERROR: No Parameter (input) layers found in the IR XML.", file=sys.stderr) + sys.exit(1) + + clues = scan_early_ops(xml_path) + norm_op_type, norm_values = detect_input_normalization_constant(xml_path) + norm_signal = classify_input_normalization(norm_op_type, norm_values) + + inp = inputs[0] + out = outputs[0] if outputs else None + + in_shape = inp["shape"] + in_et = inp["element_type"] + in_name = inp["name"] + + out_shape = out["shape"] if out else [] + out_prec_raw = out["precision"] if out else "FP32" + out_name = out["name"] if out else "output" + + # ── Auto-detect fields ──────────────────────────────────────────────────── + layout, _ = detect_layout(in_shape) + nif, ch_per_frame = detect_nif(layout, in_shape) + in_precision = map_element_type(in_et) + out_precision = map_output_precision(out_prec_raw) + + # SR detection — compare spatial dims of input and output tensors + scale_h, scale_w = 1, 1 + output_passthrough_dims = True # default: assume passthrough + if out_shape and in_shape: + is_dyn_in = any(d <= 0 for d in in_shape[1:]) + is_dyn_out = any(d <= 0 for d in out_shape[1:]) + if not is_dyn_in and not is_dyn_out: + scale_h, scale_w = detect_sr_scale(in_shape, out_shape) + output_passthrough_dims = (scale_h <= 1 and scale_w <= 1) + elif is_dyn_in and is_dyn_out: + # Both dynamic — static shape comparison is impossible. + # Try reading scale from the pixel_shuffle Reshape_1 literal dims. + scale_h, scale_w = detect_scale_from_pixel_shuffle(xml_path) + output_passthrough_dims = (scale_h <= 1 and scale_w <= 1) + + window_type, window_init_dup = infer_window_type(nif, layout) + align = infer_align(nif, in_precision, layout) + normalize_input, normalize_output = infer_normalize(in_precision, norm_signal) + channel_divisor = detect_channel_divisor(layout, nif) + + # out_layout: for single-frame models, output layout = input layout. + # For multi-frame batched (NFHWC input), output is typically NCHW or NHWC. + if "NFHWC" in layout or "NFCHW" in layout: + out_layout = "NCHW" + else: + out_layout = layout + + model_color: Optional[str] = "RGB" if ch_per_frame == 3 else None + out_order = "RGB" + out_precision_depth_derived = False + # color_format_auto: 0 = use model_color string; 1 = VideoProc-auto; 2 = CustVSR-auto. + # Only relevant when model_color is None; default 0 for all standard models. + color_format_auto = 0 + + model_name = name_override or Path(xml_path).stem + + # ── IR analysis output (always shown) ────────────────────────────────── + print(f"\n{'='*68}") + print(f" IR Analysis: {xml_path}") + print(f"{'='*68}") + print(f" Input tensors ({len(inputs)} found):") + for i, inp_i in enumerate(inputs): + print(f" [{i}] name={inp_i['name']!r} shape={inp_i['shape']} " + f"element_type={inp_i['element_type']!r}") + print(f" Output tensors ({len(outputs)} found):") + for i, out_i in enumerate(outputs): + print(f" [{i}] name={out_i['name']!r} shape={out_i['shape']} " + f"precision={out_i['precision']!r}") + print() + print(f" Detected in_layout: {layout}") + print(f" Detected nif: {nif} (channels/frame: {ch_per_frame})") + print(f" Detected channel_divisor: {channel_divisor}") + print(f" Detected in_precision: {in_precision}") + print(f" Detected out_precision: {out_precision}") + if scale_h > 1 or scale_w > 1: + print(f" Detected SR scale: {scale_h}×{scale_w} (height×width)") + else: + print(f" Detected SR scale: 1×1 (passthrough or dynamic)") + print(f" Inferred output_passthrough: {output_passthrough_dims}") + print(f" Inferred window_type: {window_type}") + print(f" Inferred align: {align}") + if norm_signal == "raw_255_scale": + vals_str = ", ".join(f"{v:.4g}" for v in norm_values) + print(f" Detected input constant: {norm_op_type}({vals_str}) on the raw input") + print(f" → raw [0,255] scale baked into the graph itself") + print(f" Inferred normalize_input: {normalize_input} (confident)") + print(f" Inferred normalize_output: {normalize_output} (confident)") + elif norm_signal == "meanstd_normalize": + vals_str = ", ".join(f"{v:.4g}" for v in norm_values) + print(f" Detected input constant: {norm_op_type}({vals_str}) on the raw input") + print(f" → mean/std-style normalization; only valid on already-[0,1] data") + print(f" Inferred normalize_input: {normalize_input} (confident)") + print(f" Inferred normalize_output: {normalize_output} (confident)") + else: + print(f" Inferred normalize_input: {normalize_input} (no signal found — unconfirmed, verify below)") + print(f" Inferred normalize_output: {normalize_output} (no signal found — unconfirmed, verify below)") + if clues["multi_input"]: + print(f"\n ⚠ Multiple Parameter layers detected ({len(inputs)}).") + print(f" Consider fusing conditioning inputs before export (see guide §3).") + if clues["multi_output"]: + print(f"\n ⚠ Multiple Result layers detected ({len(outputs)}).") + print(f" The config will target the first output only.") + if channel_divisor > 1: + print(f"\n ⚠ channel_divisor={channel_divisor}: {nif} frames are stacked on the") + print(f" channel axis ({layout}). Omitting channel_divisor from the JSON") + print(f" defaults to 1 in the C parser and reports the wrong channel count") + print(f" to FFmpeg (see patch 0005/parse_model_config_json).") + print() + # ── Interactive refinement (non-IR fields only) ─────────────────────── + if not non_interactive: + print(f"\n{'─'*68}") + print(f" iVSR Config Generator — Interactive Refinement") + print(f" Model: {xml_path}") + print(f" Fields derived from the IR above are used as-is.") + print(f" Press Enter to accept the suggested value in [brackets].") + print(f"{'─'*68}\n") + + model_name = _ask("name", model_name) + align = _ask_int( + "align (0=none | 32/64/128 — heuristic, verify for new architectures)", + align, + ) + if channel_divisor > 1 or nif > 1: + channel_divisor = _ask_int( + "channel_divisor (stacked frames on channel axis; nif frames / this = 3ch)", + channel_divisor, + ) + normalize_input = _ask_bool( + "normalize_input (divide uint8 by 255 before packing float32? True=[0,1] input)", + normalize_input, + ) + normalize_output = _ask_bool( + "normalize_output (multiply float32 output by 255? True=[0,1] output)", + normalize_output, + ) + mc_default = model_color or "null" + mc_ans = _ask( + "model_color (training color space; RGB for all standard SR models)", + mc_default, + choices=["RGB", "I420_Three_Planes", "null"], + ) + model_color = None if mc_ans == "null" else mc_ans + + out_order = _ask( + "out_order (output channel order)", + out_order, + choices=["RGB", "BGR", "NONE"], + ) + + _, wid_default = infer_window_type(nif, layout) + window_type = _ask( + "window_type (frame queuing strategy; IR-inferred above)", + window_type, + choices=["single", "sliding", "in_queue"], + ) + if window_type == "sliding": + window_init_dup = _ask_bool( + "window_init_dup (duplicate first frame to prime the sliding window?)", + wid_default, + ) + else: + window_init_dup = False + # ── Build config dict ───────────────────────────────────────────────────── + cfg: Dict[str, Any] = { + "_comment_generated": ( + f"Auto-generated by generate_ivsr_model_config.py " + f"from {Path(xml_path).name}" + ), + "_comment_input_tensor": ( + f"Input: name={in_name!r} shape={in_shape} element_type={in_et!r}" + ), + "_comment_output_tensor": ( + f"Output: name={out_name!r} shape={out_shape} precision={out_prec_raw!r}" + ), + "name": model_name, + "nif": nif, + "channel_divisor": channel_divisor, + "align": align, + "in_layout": layout, + "in_precision": in_precision, + "out_layout": out_layout, + "out_precision": out_precision, + "model_color": model_color, + "out_order": out_order, + "window_type": window_type, + "window_init_dup": window_init_dup if window_type == "sliding" else False, + "normalize_input": normalize_input, + "normalize_output": normalize_output, + "output_passthrough_dims": output_passthrough_dims, + "out_precision_depth_derived": out_precision_depth_derived, + "color_format_auto": color_format_auto, + } + + return cfg + + +# ───────────────────────────────────────────────────────────────────────────── +# Section 6: Entry point +# ───────────────────────────────────────────────────────────────────────────── + +def main(): + parser = argparse.ArgumentParser( + prog="generate_ivsr_model_config.py", + description=( + "Auto-generate an iVSR JSON model config from an OpenVINO IR .xml file.\n" + "Infers tensor layout, precision, nif, and window strategy from the IR;\n" + "prompts interactively for the few fields that require domain knowledge." + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap_dedent(""" +Examples +-------- + # Interactive (recommended first use): + python3 generate_ivsr_model_config.py models/span_x4.xml + + # Fully automatic (CI / scripted pipelines): + python3 generate_ivsr_model_config.py models/rife.xml --non-interactive + + # Override output path and model name: + python3 generate_ivsr_model_config.py model.xml -o configs/mymodel.json -n "MyModel" + + # Print the complete parameter reference (no IR file needed): + python3 generate_ivsr_model_config.py --deep-dive + + # Validate the output against the JSON schema: + jsonschema -i mymodel_config.json ivsr_model_config.schema.json +"""), + ) + parser.add_argument( + "xml", + nargs="?", + help="Path to the OpenVINO IR .xml file", + ) + parser.add_argument( + "--output", "-o", + default=None, + help="Output JSON path (default: _config.json next to the xml)", + ) + parser.add_argument( + "--name", "-n", + default=None, + help="Model name override (default: derived from the xml filename)", + ) + parser.add_argument( + "--non-interactive", + action="store_true", + help="Suppress all prompts; use auto-detected defaults for every field", + ) + parser.add_argument( + "--deep-dive", + action="store_true", + help="Print the complete field-by-field parameter reference and exit", + ) + + args = parser.parse_args() + + if args.deep_dive: + print_deep_dive() + sys.exit(0) + + if not args.xml: + parser.print_help() + sys.exit(1) + + if not os.path.isfile(args.xml): + print(f"ERROR: File not found: {args.xml}", file=sys.stderr) + sys.exit(1) + + output_path = args.output + if output_path is None: + p = Path(args.xml) + output_path = str(p.parent / (p.stem + "_config.json")) + + cfg = generate_config( + xml_path=args.xml, + name_override=args.name, + non_interactive=args.non_interactive, + ) + + with open(output_path, "w") as f: + json.dump(cfg, f, indent=2) + f.write("\n") + + print(f"\nConfig written to: {output_path}") + print("\n── Generated config ─────────────────────────────────────────────────") + public = {k: v for k, v in cfg.items() if not k.startswith("_comment")} + print(json.dumps(public, indent=2)) + print() + print("── Next steps ──────────────────────────────────────────────────────") + print(f" 1. Validate: jsonschema -i {output_path} ivsr_model_config.schema.json") + print(f" 2. Run:") + xml_name = Path(args.xml).name.replace(".xml", "") + print(f" ./ffmpeg -i input.mp4 \\") + print(f" -vf \"format=rgb24,dnn_processing=dnn_backend=ivsr:\\") + print(f" model={args.xml}:input={cfg.get('_comment_input_tensor','input').split('name=')[1].split(' ')[0].strip(chr(39))}:output={cfg.get('_comment_output_tensor','output').split('name=')[1].split(' ')[0].strip(chr(39))}:\\") + print(f" model_type=-1:model_config={output_path}:device=GPU\" \\") + print(f" -pix_fmt yuv420p output.mp4") + + +def textwrap_dedent(s: str) -> str: + """Minimal textwrap.dedent equivalent to avoid stdlib import.""" + lines = s.split("\n") + # find minimum leading spaces on non-empty lines + min_indent = None + for line in lines: + stripped = line.lstrip() + if stripped: + indent = len(line) - len(stripped) + if min_indent is None or indent < min_indent: + min_indent = indent + if min_indent is None: + return s + return "\n".join( + line[min_indent:] if len(line) >= min_indent else line for line in lines + ) + + +if __name__ == "__main__": + main() diff --git a/ivsr_ffmpeg_plugin/model_configs/custvsr_config.json b/ivsr_ffmpeg_plugin/model_configs/custvsr_config.json new file mode 100644 index 0000000..6b2951d --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/custvsr_config.json @@ -0,0 +1,17 @@ +{ + "_comment": "CustVSR uses single-channel YUV (yuv420p) input/output (Y-plane only). color_format_auto=2 enables the single-channel Y-plane dispatch path so pack_input_window / unpack_output_window handle the luma plane correctly instead of assuming 3-channel RGB.", + "name": "Custom VSR", + "nif": 1, + "align": 0, + "in_layout": "NCHW", + "out_layout": "NCHW", + "out_precision": "fp32", + "out_order": "NONE", + "window_type": "single", + "window_init_dup": false, + "normalize_input": false, + "normalize_output": false, + "output_passthrough_dims": false, + "out_precision_depth_derived": false, + "color_format_auto": 2 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/edsr_config.json b/ivsr_ffmpeg_plugin/model_configs/edsr_config.json new file mode 100644 index 0000000..a718e1e --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/edsr_config.json @@ -0,0 +1,19 @@ +{ + "_comment": "Enhanced EDSR single-frame RGB SR. in_precision=f32/out_precision=fp32: explicit float32 I/O (not bit-depth derived). normalize_input/normalize_output=false: model operates on raw [0,255] float32. Works for both FP32 and INT8 model variants.", + "name": "Enhanced EDSR", + "nif": 1, + "align": 0, + "in_layout": "NCHW", + "in_precision": "f32", + "out_layout": "NCHW", + "out_precision": "fp32", + "model_color": "RGB", + "out_order": "RGB", + "window_type": "single", + "window_init_dup": false, + "normalize_input": false, + "normalize_output": false, + "output_passthrough_dims": false, + "out_precision_depth_derived": false, + "color_format_auto": 0 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/hdrtvnet_le_config.json b/ivsr_ffmpeg_plugin/model_configs/hdrtvnet_le_config.json new file mode 100644 index 0000000..5d5edbf --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/hdrtvnet_le_config.json @@ -0,0 +1,19 @@ +{ + "_comment": "HDRTVNet++ LE (Local Enhancement, Path A). normalize_input/normalize_output=false: raw [0,255] float32 I/O. out_precision_depth_derived=true: output bit depth follows source depth (u16 for 10-bit HDR input). output_passthrough_dims=false: output dimensions are determined by the model (standard SR upscale).", + "name": "HDRTVNet-LE", + "nif": 1, + "align": 0, + "in_layout": "NCHW", + "in_precision": "f32", + "out_layout": "NCHW", + "out_precision": "fp32", + "model_color": "RGB", + "out_order": "RGB", + "window_type": "single", + "window_init_dup": false, + "normalize_input": true, + "normalize_output": true, + "output_passthrough_dims": false, + "out_precision_depth_derived": false, + "color_format_auto": 0 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/ivsr_model_config_README.md b/ivsr_ffmpeg_plugin/model_configs/ivsr_model_config_README.md new file mode 100644 index 0000000..181400b --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/ivsr_model_config_README.md @@ -0,0 +1,247 @@ +# iVSR Model Config Files + +Each non-BasicVSR model is described by a JSON config file placed in this directory. +At runtime FFmpeg reads the file and configures the iVSR backend without any recompilation. + +**FFmpeg usage:** + +```bash +./ffmpeg -i input.mp4 \ + -vf "format=rgb24,dnn_processing=dnn_backend=ivsr:\ +model=mymodel.xml:model_config=../model_configs/mymodel_config.json" \ + output.mp4 +``` + +BasicVSR does not use a JSON config — it is a built-in and uses `model_type=0` directly. + +--- + +## Required fields + +### `name` + +Human-readable label used in FFmpeg log messages. +**Type:** string +**Example:** `"RIFE"`, `"VideoSeal"` + +### `in_layout` + +Memory layout of the input tensor as understood by the iVSR SDK. + +| Value | Meaning | +| --- | --- | +| `"NCHW"` | Batch × Channels × Height × Width (most common for single-frame RGB models) | +| `"NHWC"` | Batch × Height × Width × Channels (VideoProc, CustVSR) | +| `"NFHWC"` | Batch × Frames × Height × Width × Channels (BasicVSR only, built-in) | + +--- + +## Optional fields and their defaults + +### `nif` + +Number of input frames consumed per inference call. +**Type:** integer ≥ 1 +**Default (if absent):** `1` (single-frame) +**When to set:** sliding-window models only — `2` for RIFE, `3` for TSENet. + +--- + +### `channel_divisor` + +Divides the channel count reported to FFmpeg so it sees standard 3-channel frames. +Some models pack multiple frames along the channel axis; this corrects the mismatch. +**Type:** integer ≥ 1 +**Default (if absent):** `1` (no division) +**When to set:** `2` for RIFE (two RGB frames → 6 channels ÷ 2), `3` for TSENet (three frames → 9 channels ÷ 3). + +--- + +### `align` + +Pads input width and height up to the nearest multiple of this value before reshaping. +**Type:** integer ≥ 0 +**Default (if absent):** `0` (no alignment) +**When to set:** `128` for RIFE, `64` for VideoProc. + +--- + +### `in_precision` + +Element type of the input tensor sent to the iVSR SDK. +**Type:** string or absent +**Supported values:** `"f32"`, `"u8"`, `"u16"` +**Default (if absent):** inherited from the input frame bit depth (8-bit → `u8`, 10/16-bit → `u16`) +**When to set:** explicitly `"f32"` for models that always expect float regardless of frame depth (RIFE, SPAN, EDSR, VideoSeal). + +--- + +### `out_layout` + +Memory layout of the output tensor. +**Type:** string +**Supported values:** `"NCHW"`, `"NHWC"`, `"NFHWC"` +**Default (if absent):** `"NHWC"` — or, for JSON-config models, inherited from `in_layout` when `out_layout` is absent. +**NOTE: Must be set explicitly when the model output layout differs from its input layout.** For example, TSENet has `in_layout=NCHW` but `out_layout=NHWC` and must declare both. + +--- + +### `out_precision` + +Element type of the output tensor. +**Type:** string or absent +**Supported values:** `"fp32"`, `"u8"`, `"u16"` +**Default (if absent):** `"fp32"` — confirmed safe; the C struct is explicitly initialized to `"fp32"` before the JSON is applied. +**When to set:** only needed if the model outputs integer data (`"u8"` or `"u16"`). All currently shipped models use `"fp32"` and can omit this field once the comparison table below is read. + +--- + +### `out_precision_depth_derived` + +When `true`, the output precision is derived from the frame bit depth instead of `out_precision`: 8-bit frames → `u8`, 10/16-bit frames → `u16`. +**Type:** boolean +**Default (if absent):** `false` +**When to set:** `true` for legacy EDSR integer-output variants only. All current shipped configs set this to `false`. + +--- + +### `model_color` + +Colour space string sent to the iVSR SDK to describe the input tensor's colour format. +**Type:** string or absent +**Supported values:** `"RGB"`, `"I420_Three_Planes"` + +| Value | Meaning | +| --- | --- | +| `"RGB"` | Model expects interleaved RGB planes | +| `"I420_Three_Planes"` | Model expects YUV 4:2:0 three-plane layout | + +**Default (if absent):** no colour format is written to the SDK if `color_format_auto` is also `0` — in that case the parser default of `"RGB"` is used. Always set either `model_color` or `color_format_auto` when a non-RGB format is needed. + +--- + +### `out_order` + +Output channel order used when writing pixels back into the FFmpeg frame buffer. +**Type:** string +**Supported values:** `"RGB"`, `"BGR"`, `"NONE"` +**Default (if absent):** `"RGB"` +**When to set:** `"NONE"` for single-channel (luma-only) models like CustVSR, where no colour reordering applies. + +--- + +### `window_type` + +Frame queuing strategy used to assemble the input tensor. + +| Value | Behaviour | Used by | +| --- | --- | --- | +| `"single"` | No queue; one frame is fed directly into the tensor per inference call | EDSR, SPAN, VideoSeal, VideoProc, CustVSR | +| `"sliding"` | A sliding window of `nif` frames is maintained in a queue; oldest frame is dropped after each call | RIFE, TSENet | +| `"in_queue"` | `nif` frames are read from the task's input queue (BasicVSR built-in path; not for JSON configs) | BasicVSR only | + +**Default (if absent):** `"single"` + +--- + +### `window_init_dup` + +Only relevant when `window_type` is `"sliding"`. +When `true`, the first frame is duplicated to prime the sliding window so inference can begin immediately without waiting for `nif` distinct frames. +**Type:** boolean +**Default (if absent):** `false` +**When to set:** `true` for TSENet (temporal continuity at stream start). `false` for RIFE (interpolation genuinely requires two distinct frames). + +--- + +### `normalize_input` + +When `true`, uint8 pixel values [0, 255] are divided by 255 before being packed into the float32 input tensor, producing a [0.0, 1.0] range. +When `false`, raw pixel values are packed as-is into float32. +**Type:** boolean +**Default (if absent):** `false` +**When to set:** `true` for RIFE, SPAN and HDRTVNet-LE (normalised [0,1] float input). All other shipped models use raw [0,255] and can omit this field. + +--- + +### `normalize_output` + +When `true`, float32 output values are multiplied by 255 and clipped to [0, 255] before being written into the output frame. +When `false`, float32 values are rounded and clipped directly. +**Type:** boolean +**Default (if absent):** `false` +**When to set:** `true` for RIFE, SPAN, TSENet, VideoProc and HDRTVNet-LE (model outputs [0,1] float that must be scaled back to [0,255]). All other shipped models output raw [0,255] and can omit this field. + +--- + +### `output_passthrough_dims` + +When `true`, the output frame dimensions are forced to match the input frame dimensions, regardless of what the model's output tensor reports. +Used for watermarking and passthrough models where the spatial size never changes. +**Type:** boolean +**Default (if absent):** `false` +**When to set:** `true` for VideoSeal and VideoProc. + +--- + +### `color_format_auto` + +Controls how the colour space string is determined when `model_color` is absent. +Takes priority over `model_color` being set when the value is non-zero. + +| Value | Behaviour | Used by | +| --- | --- | --- | +| `0` | Use the `model_color` string directly | All standard RGB models | +| `1` | Auto-detect from the input pixel format: RGB input → `"RGB"`, YUV input → `"I420_Three_Planes"` | VideoProc | +| `2` | Always `"I420_Three_Planes"` regardless of input format | CustVSR | + +**Default (if absent):** `0` +**When to set:** `1` for models that accept both RGB and YUV input; `2` for YUV-only models. + +--- + +## Model parameter comparison + +| Parameter | RIFE | VideoSeal | TSENet | EDSR | VideoProc | CustVSR | SPAN | HDRTVNet-LE | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **name** | `"RIFE"` | `"VideoSeal"` | `"TSENet"` | `"Enhanced EDSR"` | `"VideoProc"` | `"Custom VSR"` | `"SPAN"` | `"HDRTVNet-LE"` | +| **nif** | `2` | `1` | `3` | `1` | `1` | `1` | `1` | `1` | +| **channel_divisor** | `2` | *(absent→1)* | `3` | *(absent→1)* | *(absent→1)* | *(absent→1)* | *(absent→1)* | *(absent→1)* | +| **align** | `128` | `0` | `0` | `0` | `64` | `0` | `0` | `0` | +| **in_layout** | `NCHW` | `NCHW` | `NCHW` | `NCHW` | `NHWC` | `NCHW` | `NCHW` | `NCHW` | +| **in_precision** | `f32` | `f32` | *(absent)* | `f32` | *(absent)* | *(absent)* | `f32` | `f32` | +| **out_layout** | `NCHW` | `NCHW` | `NHWC` | `NCHW` | `NHWC` | `NCHW` | `NCHW` | `NCHW` | +| **out_precision** | `fp32` | `fp32` | `fp32` | `fp32` | `fp32` | `fp32` | `fp32` | `fp32` | +| **model_color** | `RGB` | `RGB` | `RGB` | `RGB` | *(absent)* | *(absent)* | `RGB` | `RGB` | +| **out_order** | `RGB` | `RGB` | `RGB` | `RGB` | *(absent→RGB)* | `NONE` | `RGB` | `RGB` | +| **window_type** | `sliding` | `single` | `sliding` | `single` | `single` | `single` | `single` | `single` | +| **window_init_dup** | `false` | `false` | `true` | `false` | `false` | `false` | `false` | `false` | +| **normalize_input** | `true` | `false` | `false` | `false` | `false` | `false` | `true` | `true` | +| **normalize_output** | `true` | `false` | `true` | `false` | `true` | `false` | `true` | `true` | +| **output_passthrough_dims** | `false` | `true` | `false` | `false` | `true` | `false` | `false` | `false` | +| **out_precision_depth_derived** | `false` | `false` | `false` | `false` | `false` | `false` | `false` | `false` | +| **color_format_auto** | `0` | `0` | `0` | `0` | `1` | `2` | `0` | `0` | + +--- + +## Summary of safe defaults (current state) + +| Parameter | C default | Safe to omit today? | +| --- | --- | --- | +| `name` | — | required | +| `in_layout` | — | required | +| `nif` | `1` | yes, for single-frame models | +| `channel_divisor` | `1` | yes, when no channel stacking | +| `align` | `0` | yes, when no alignment needed | +| `in_precision` | depth-derived | yes, when depth-derived is correct | +| `out_layout` | `in_layout` (inherited) | yes, when output layout matches input layout; must set when layouts differ (e.g. TSENet: in=NCHW, out=NHWC) | +| `out_precision` | `"fp32"` | yes, for all currently shipped models | +| `model_color` | `"RGB"` | yes, for all standard RGB models | +| `out_order` | `"RGB"` | yes, except `"NONE"` for luma-only models | +| `window_type` | `"single"` | yes, for single-frame models | +| `window_init_dup` | `false` | yes, when false | +| `normalize_input` | `false` | yes, for most models; set `true` only for RIFE, SPAN and HDRTVNet-LE | +| `normalize_output` | `false` | yes, for most models; set `true` only for RIFE, SPAN, TSENet, VideoProc and HDRTVNet-LE | +| `output_passthrough_dims` | `false` | yes, when false | +| `out_precision_depth_derived` | `false` | yes, always for current models | +| `color_format_auto` | `0` | yes, when using `model_color` | diff --git a/ivsr_ffmpeg_plugin/model_configs/rife_config.json b/ivsr_ffmpeg_plugin/model_configs/rife_config.json new file mode 100644 index 0000000..63e69fd --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/rife_config.json @@ -0,0 +1,20 @@ +{ + "_comment": "RIFE frame interpolation. nif=2/channel_divisor=2: two RGB frames stacked along channel axis; window_init_dup=false: no priming (interpolation starts on frame 2). normalize_input=true: divide by 255 before inference; normalize_output=true: multiply by 255 after inference. align=128: pad H/W to multiples of 128.", + "name": "RIFE", + "nif": 2, + "channel_divisor": 2, + "align": 128, + "in_layout": "NCHW", + "in_precision": "f32", + "out_layout": "NCHW", + "out_precision": "fp32", + "model_color": "RGB", + "out_order": "RGB", + "window_type": "sliding", + "window_init_dup": false, + "normalize_input": true, + "normalize_output": true, + "output_passthrough_dims": false, + "out_precision_depth_derived": false, + "color_format_auto": 0 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/span_config.json b/ivsr_ffmpeg_plugin/model_configs/span_config.json new file mode 100644 index 0000000..f0f1c26 --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/span_config.json @@ -0,0 +1,19 @@ +{ + "_comment": "SPAN single-frame RGB SR. normalize_input=true: divide by 255; normalize_output=true: multiply by 255. Works for both x2 and x4 upscale model variants.", + "name": "SPAN", + "nif": 1, + "align": 0, + "in_layout": "NCHW", + "in_precision": "f32", + "out_layout": "NCHW", + "out_precision": "fp32", + "model_color": "RGB", + "out_order": "RGB", + "window_type": "single", + "window_init_dup": false, + "normalize_input": true, + "normalize_output": true, + "output_passthrough_dims": false, + "out_precision_depth_derived": false, + "color_format_auto": 0 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/tsenet_config.json b/ivsr_ffmpeg_plugin/model_configs/tsenet_config.json new file mode 100644 index 0000000..44c62e9 --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/tsenet_config.json @@ -0,0 +1,19 @@ +{ + "_comment": "TSENet 3-frame temporal SR. nif=3/channel_divisor=3: three consecutive frames stacked along the channel axis; window_init_dup=true duplicates the first frame to prime the sliding window. normalize_input=false: raw [0,255] float32 input; normalize_output=true: model outputs [0,1] scaled back to [0,255] by unpack. in_layout NCHW / out_layout NHWC matches the model's actual tensor order.", + "name": "TSENet", + "nif": 3, + "channel_divisor": 3, + "align": 0, + "in_layout": "NCHW", + "out_layout": "NHWC", + "out_precision": "fp32", + "model_color": "RGB", + "out_order": "RGB", + "window_type": "sliding", + "window_init_dup": true, + "normalize_input": false, + "normalize_output": true, + "output_passthrough_dims": false, + "out_precision_depth_derived": false, + "color_format_auto": 0 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/videoproc_config.json b/ivsr_ffmpeg_plugin/model_configs/videoproc_config.json new file mode 100644 index 0000000..65261b8 --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/videoproc_config.json @@ -0,0 +1,16 @@ +{ + "_comment": "SVP/VideoProc Y-channel SR. align=64 pads H/W to multiples of 64 for model reshape. color_format_auto=1 auto-selects I420_Three_Planes for yuv420p input or RGB for rgb24. normalize_input=false: u8 input is passed as-is; the iVSR SDK divides by scale=255 so the model receives [0,1] float. normalize_output=true: model outputs [0,1] float which is multiplied by 255 (8-bit) or 2^bits-1 (10/16-bit) to restore the full pixel range. output_passthrough_dims=true: output H/W are forced to match input H/W (no scale change). TV-range clamping and 10-bit uint16 output are applied automatically by the unpack function when name=VideoProc.", + "name": "VideoProc", + "nif": 1, + "align": 64, + "in_layout": "NHWC", + "out_layout": "NHWC", + "out_precision": "fp32", + "window_type": "single", + "window_init_dup": false, + "normalize_input": false, + "normalize_output": true, + "output_passthrough_dims": true, + "out_precision_depth_derived": false, + "color_format_auto": 1 +} diff --git a/ivsr_ffmpeg_plugin/model_configs/videoseal_config.json b/ivsr_ffmpeg_plugin/model_configs/videoseal_config.json new file mode 100644 index 0000000..b59dced --- /dev/null +++ b/ivsr_ffmpeg_plugin/model_configs/videoseal_config.json @@ -0,0 +1,19 @@ +{ + "_comment": "VideoSeal invisible watermarking. normalize_input/normalize_output=false: raw [0,255] float32 I/O. output_passthrough_dims=true: output resolution matches input (watermark does not resize). in_precision=f32: explicit float32 input.", + "name": "VideoSeal", + "nif": 1, + "align": 0, + "in_layout": "NCHW", + "in_precision": "f32", + "out_layout": "NCHW", + "out_precision": "fp32", + "model_color": "RGB", + "out_order": "RGB", + "window_type": "single", + "window_init_dup": false, + "normalize_input": false, + "normalize_output": false, + "output_passthrough_dims": true, + "out_precision_depth_derived": false, + "color_format_auto": 0 +} diff --git a/ivsr_ffmpeg_plugin/patches/0005-Unified-model-support-JSON-config-dispatch.patch b/ivsr_ffmpeg_plugin/patches/0005-Unified-model-support-JSON-config-dispatch.patch new file mode 100644 index 0000000..b172e31 --- /dev/null +++ b/ivsr_ffmpeg_plugin/patches/0005-Unified-model-support-JSON-config-dispatch.patch @@ -0,0 +1,1226 @@ +From Mon Sep 17 00:00:00 2001 +From: iVSR +Date: Thu, 15 May 2026 00:00:00 +0000 +Subject: [PATCH 0005] Unified model support: generic JSON config dispatch + and model_type-independent path selection + +This patch combines the changes previously split across patches 0005-0010. +It adds all non-BasicVSR model support and the full JSON config dispatch +system in one self-contained patch on top of patches 0001-0004. + +Changes included: +- RIFE frame-interpolation model (sliding-window, nif=2, float32 [0,1]) +- VideoSeal invisible watermarking (single-frame, float32 [0,255]) +- TSENet temporal SR (sliding-window, nif=3, channel_divisor=3) +- VideoProc / EDSR / CustVSR via generic NHWC path +- ModelDesc descriptor table replacing all if/else model-type chains +- pack_input_window() / unpack_output_window() generic handlers: + * SLIDING: normalize_input support (divide uint8 by 255 for RIFE/SPAN) + * SINGLE: ff_proc fallback for non-float input (VideoProc, EDSR) + * IN_QUEUE: delegates to pack_input_basicvsr (BasicVSR unchanged) +- channel_divisor field: divides reported tensor channels so FFmpeg sees + standard 3-channel frames (RIFE ÷2, TSENet ÷3) +- JSON config system (parse_model_config_json): load any ModelDesc at + runtime from a .json file — no C changes or recompilation required +- model_config AVOption: when set, JSON path is used automatically; + model_type is ignored (removed CUSTOM=-1 sentinel) +- model_type enum collapsed to BASICVSR=0 + MODEL_TYPE_NUM; only + BasicVSR remains as a built-in (multi-frame out_queue cannot use JSON) +- unpack_output_window: Y-plane (1-channel) path for VideoProc/SVP/CustVSR; + chroma copy removed from callback — unsafe due to inference_done race; + vf_dnn_processing.c copy_uv_planes() handles UV on the main thread instead +- unpack_output_window: RGB48 (16-bit) output support for HDRTVNet++ +- ivsr_model_config.template.json: copy-and-edit starting point +- ivsr_model_config.schema.json: JSON Schema draft-07 for IDE validation + +FFmpeg usage (all non-BasicVSR models): + ./ffmpeg -i input.mp4 \ + -vf "format=rgb24,dnn_processing=dnn_backend=ivsr:\ + model=mymodel.xml:model_config=../model_configs/mymodel_config.json" \ + output.mp4 + +BasicVSR (built-in, no model_config needed): + ./ffmpeg -i input.mp4 \ + -vf "format=rgb24,dnn_processing=dnn_backend=ivsr:\ + model=basicvsr.xml:model_type=0:nif=3:device=CPU:extension=...:op_xml=..." \ + output.mp4 + +Canonical configs for shipped models: ivsr_ffmpeg_plugin/model_configs/ +--- +diff --git a/ivsr_model_config.schema.json b/ivsr_model_config.schema.json +new file mode 100644 +index 0000000..bd633a2 +--- /dev/null ++++ b/ivsr_model_config.schema.json +@@ -0,0 +1,84 @@ ++{ ++ "$schema": "http://json-schema.org/draft-07/schema", ++ "title": "iVSR Model Config", ++ "description": "Descriptor file for iVSR models loaded via model_type=-1. Covers tensor layout, precision, frame count, alignment, window strategy and normalisation.", ++ "type": "object", ++ "required": ["name", "in_layout"], ++ "additionalProperties": true, ++ "properties": { ++ "name": { ++ "type": "string", ++ "description": "Human-readable model label used in log messages" ++ }, ++ "nif": { ++ "type": "integer", ++ "minimum": 1, ++ "description": "Number of input frames consumed per inference call (1 = single frame)" ++ }, ++ "align": { ++ "type": "integer", ++ "minimum": 0, ++ "description": "Round input width and height up to a multiple of this value (0 = no alignment)" ++ }, ++ "in_layout": { ++ "type": "string", ++ "enum": ["NCHW", "NHWC", "NFHWC"], ++ "description": "Input tensor memory layout" ++ }, ++ "in_precision": { ++ "type": ["string", "null"], ++ "enum": ["f32", "u8", "u16", null], ++ "description": "Input tensor element type (null = inherit from frame bit-depth)" ++ }, ++ "out_layout": { ++ "type": "string", ++ "enum": ["NCHW", "NHWC", "NFHWC"], ++ "description": "Output tensor memory layout" ++ }, ++ "out_precision": { ++ "type": ["string", "null"], ++ "enum": ["fp32", "u8", "u16", null], ++ "description": "Output tensor element type (null = default fp32)" ++ }, ++ "model_color": { ++ "type": ["string", "null"], ++ "enum": ["RGB", "I420_Three_Planes", null], ++ "description": "Colour space string reported to iVSR SDK. null = determined by color_format_auto" ++ }, ++ "out_order": { ++ "type": "string", ++ "enum": ["RGB", "BGR", "NONE"], ++ "description": "Output channel order for the ffmpeg proc helper" ++ }, ++ "window_type": { ++ "type": "string", ++ "enum": ["single", "sliding", "in_queue"], ++ "description": "Frame queuing strategy: single (no queue), sliding (N-frame window), in_queue (read nif frames from task queue - BasicVSR style)" ++ }, ++ "window_init_dup": { ++ "type": "boolean", ++ "description": "When window_type=sliding: duplicate first frame to prime the queue (TSENet). false = wait for queue to fill (RIFE)" ++ }, ++ "normalize_input": { ++ "type": "boolean", ++ "description": "Divide uint8 pixel values by 255 before packing into float32 (RIFE/SPAN). false = raw [0,255] (VideoSeal)" ++ }, ++ "normalize_output": { ++ "type": "boolean", ++ "description": "Multiply float32 output by 255 then clip to uint8 (RIFE/SPAN). false = direct round-and-clip (VideoSeal)" ++ }, ++ "output_passthrough_dims": { ++ "type": "boolean", ++ "description": "Report output W/H = input W/H regardless of tensor (VideoProc passthrough models)" ++ }, ++ "out_precision_depth_derived": { ++ "type": "boolean", ++ "description": "Derive output precision from frame bit depth: u8 for 8-bit, u16 for 10/16-bit (EDSR style)" ++ }, ++ "color_format_auto": { ++ "type": "integer", ++ "enum": [0, 1, 2], ++ "description": "0 = use model_color. 1 = VideoProc-auto (RGB vs I420 from pixel format). 2 = CustVSR-auto (always I420)" ++ } ++ } ++} +diff --git a/ivsr_model_config.template.json b/ivsr_model_config.template.json +new file mode 100644 +index 0000000..289b066 +--- /dev/null ++++ b/ivsr_model_config.template.json +@@ -0,0 +1,53 @@ ++{ ++ "_comment_PURPOSE": "Copy this file to mymodel_config.json and fill in the values below.", ++ "_comment_USAGE": "Use with: dnn_backend=ivsr:model=mymodel.xml:model_type=-1:model_config=mymodel_config.json", ++ "_comment_VALIDATE":"jsonschema -i mymodel_config.json ivsr_model_config.schema.json", ++ ++ "_comment_name": "Human-readable label used in log messages", ++ "name": "MyModel", ++ ++ "_comment_nif": "Number of input frames consumed per inference call (1=single frame, 2=RIFE-style, 3=TSENet-style)", ++ "nif": 1, ++ ++ "_comment_align": "Pad input W and H to a multiple of this value before reshaping (0=none, 128 for RIFE, 64 for VideoProc)", ++ "align": 0, ++ ++ "_comment_in_layout": "Tensor layout passed to iVSR SDK: NCHW | NHWC | NFHWC", ++ "in_layout": "NCHW", ++ ++ "_comment_in_precision":"Input element type sent to the SDK: f32 | u8 | u16 (omit or null to inherit from frame bit-depth)", ++ "in_precision": "f32", ++ ++ "_comment_out_layout": "Output tensor layout: NCHW | NHWC | NFHWC", ++ "out_layout": "NCHW", ++ ++ "_comment_out_precision":"Output element type: fp32 | u8 | u16 (omit or null to use default fp32)", ++ "out_precision": "fp32", ++ ++ "_comment_model_color": "Colour space string sent to the iVSR SDK: RGB | I420_Three_Planes | null (null = no explicit colour space)", ++ "model_color": "RGB", ++ ++ "_comment_out_order": "Output colour channel order for ff_proc_from_dnn_to_frame: RGB | BGR | NONE", ++ "out_order": "RGB", ++ ++ "_comment_window_type": "Frame queuing strategy: single (one frame, no queue), sliding (N-frame sliding window), in_queue (read nif frames from task queue, BasicVSR-style)", ++ "window_type": "single", ++ ++ "_comment_window_init_dup": "Only relevant when window_type=sliding. Set true to duplicate the first frame to prime the queue (TSENet behaviour). false = wait for the queue to fill naturally (RIFE behaviour).", ++ "window_init_dup": false, ++ ++ "_comment_normalize_input": "true = divide rgb24 uint8 pixel values by 255 before packing into float32 tensor (RIFE/SPAN style input [0,1]). false = pack raw [0,255] values (VideoSeal style).", ++ "normalize_input": true, ++ ++ "_comment_normalize_output": "true = multiply float32 output by 255 then clip to uint8 (RIFE/SPAN style output [0,1] → [0,255]). false = direct round-and-clip, model already outputs [0,255] (VideoSeal style).", ++ "normalize_output": true, ++ ++ "_comment_output_passthrough_dims": "true = report output W/H = input W/H regardless of model output tensor size (VideoProc passthrough models). false = use tensor dimensions.", ++ "output_passthrough_dims": false, ++ ++ "_comment_out_precision_depth_derived": "true = derive output tensor precision from the input frame bit depth: u8 for 8-bit, u16 for 10/16-bit (EDSR style). Overrides out_precision when true.", ++ "out_precision_depth_derived": false, ++ ++ "_comment_color_format_auto": "0 = use model_color string above. 1 = VideoProc-style auto-detect (RGB if input is RGB, I420_Three_Planes otherwise). 2 = CustVSR-style (always I420_Three_Planes).", ++ "color_format_auto": 0 ++} +diff --git a/libavfilter/dnn/dnn_backend_ivsr.c b/libavfilter/dnn/dnn_backend_ivsr.c +index 0e64c14..563f420 100644 +--- a/libavfilter/dnn/dnn_backend_ivsr.c ++++ b/libavfilter/dnn/dnn_backend_ivsr.c +@@ -43,15 +43,13 @@ + #define DNN_MORE_FRAMES FFERRTAG('M','O','R','E') + + typedef enum { +- UNKNOWN_MODEL = -1, +- BASICVSR, +- VIDEOPROC, +- EDSR, +- CUSTVSR, +- TSENET, +- MODEL_TYPE_NUM ++ BASICVSR = 0, /* Only built-in model: multi-frame out_queue output cannot ++ * be expressed in a JSON config. Use model_type=0. */ ++ MODEL_TYPE_NUM /* = 1 */ + } ModelType; + ++typedef struct ModelDesc ModelDesc; /* forward declaration — defined after AVOptions */ ++ + typedef struct IVSRModel { + DNNModel model; + DnnContext *ctx; +@@ -63,6 +61,8 @@ typedef struct IVSRModel { + ModelType model_type; + int nif; //how many frames in IVSRRequestItem::in_frames + AVFifo *frame_queue; //input frames queue ++ int sliding_window_frame_num; /* first-frame dup counter for sliding-window models */ ++ ModelDesc *dynamic_desc; /* heap-allocated descriptor for JSON-loaded models */ + } IVSRModel; + + typedef struct IVSRRequestItem { +@@ -79,14 +79,121 @@ static const AVOption dnn_ivsr_options[] = { + { "batch_size", "batch size per request, NOT usable for BasicVSR model", OFFSET(batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, + { "extension", "extension lib file full path, usable for BasicVSR model", OFFSET(extension), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS}, + { "op_xml", "custom op xml file full path, usable for BasicVSR model", OFFSET(op_xml), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS}, +- { "model_type", "dnn model type", OFFSET(model_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, MODEL_TYPE_NUM - 1, FLAGS}, ++ { "model_type", ++ "0 = BasicVSR built-in (multi-frame out_queue output, no JSON path). " ++ "For all other models, leave model_type at its default and set model_config instead.", ++ OFFSET(model_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, MODEL_TYPE_NUM - 1, FLAGS}, + { "normalize_factor", "normalizing factor(constant) for models that not require input normalization to [0, 1]", OFFSET(normalize_factor), AV_OPT_TYPE_FLOAT, { .dbl = 1.0 }, 1.0, 65535.0, FLAGS}, + { "num_streams", "number of execution streams for the throughput mode (now valid only for GPU devices).", OFFSET(num_streams), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 256, FLAGS}, ++ { "model_config", ++ "Path to a JSON model descriptor (all models except BasicVSR). " ++ "When set, model_type is ignored and all dispatch uses the JSON config. " ++ "Canonical configs: model_configs/{videoproc,edsr,custvsr,tsenet,rife,videoseal}_config.json. " ++ "See ivsr_model_config.template.json for all fields.", ++ OFFSET(model_config), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS}, + { NULL } + }; + + #define ALIGNED_SIZE 64 + ++/* Frame queuing strategy for JSON-loaded models (Patch 0008). */ ++typedef enum { ++ WINDOW_SINGLE, /* single frame, no queuing (VideoSeal/SPAN style) */ ++ WINDOW_SLIDING, /* N-frame sliding window via frame_queue */ ++ WINDOW_IN_QUEUE, /* read nif frames from task->in_queue (BasicVSR style) */ ++} WindowType; ++ ++/* --------------------------------------------------------------------------- ++ * ModelDesc — per-model descriptor table ++ * ++ * To add a new model: ++ * 1. Add an enum value to ModelType (above). ++ * 2. Add a row to model_table[] (below) — fill in the static fields. ++ * 3. If the model needs non-generic input packing or output unpacking, ++ * write static pack_input_ / unpack_output_ functions ++ * (see RIFE / VideoSeal below as templates) and point the function ++ * pointers at them. If the generic NCHW-float or ff_proc paths work, ++ * leave the pointers NULL. ++ * ++ * That is the ONLY change needed — no if/else chains anywhere else. ++ * --------------------------------------------------------------------------- ++ * ++ * Field reference: ++ * name human-readable label (for log messages) ++ * nif_override force nif to this value (0 = use SDK-reported value) ++ * channel_divisor channels reported to filter = actual_channels / divisor ++ * (RIFE has [1,6,H,W] input but each frame is 3-ch → /2; ++ * TSENET has 3× stacked frames → /3; others → 1) ++ * align input W/H round-up alignment in pixels (0 = none) ++ * in_layout tensor layout string for iVSR SDK ("NHWC","NCHW","NFHWC") ++ * in_precision tensor element type ("u8","u16","f32"; NULL = depth-derived) ++ * out_layout output tensor layout string ++ * out_precision output element type (NULL = default "fp32") ++ * model_color model colour-space string ("RGB","I420_Three_Planes",NULL) ++ * out_order DNNColorOrder for output.order assignment ++ * pack_input custom rgb24→tensor function; NULL = generic ff_proc path ++ * unpack_output custom tensor→rgb24 function; NULL = generic ff_proc path ++ * --------------------------------------------------------------------------- ++ */ ++typedef struct IVSRModel IVSRModel; /* forward decl for function pointer types */ ++typedef int (*PackInputFn) (IVSRModel *m, void *in_data_base, ++ DNNData *input, TaskItem *task); ++typedef void (*UnpackOutputFn)(IVSRModel *m, TaskItem *task, ++ DNNData *output); ++ ++typedef struct ModelDesc { ++ const char *name; ++ int nif_override; ++ int channel_divisor; ++ int align; ++ const char *in_layout; ++ const char *in_precision; /* NULL = derive from bit-depth */ ++ const char *out_layout; ++ const char *out_precision; /* NULL = default "fp32" */ ++ const char *model_color; /* NULL = handled in VIDEOPROC/CUSTVSR branch */ ++ DNNColorOrder out_order; ++ PackInputFn pack_input; /* NULL = generic path */ ++ UnpackOutputFn unpack_output; /* NULL = generic path */ ++ /* --- Patch 0007 fields --- */ ++ int out_precision_depth_derived; /* 1 = derive output precision from bit depth (EDSR) */ ++ int output_passthrough_dims; /* 1 = output W/H = input W/H (VideoProc) */ ++ int color_format_auto; /* 0 = model_color, 1 = VideoProc-auto, 2 = CustVSR-auto */ ++ int sliding_window_init_dup; /* 1 = dup first frame to prime queue (TSENet) */ ++ /* --- Patch 0008 fields (JSON-config models) --- */ ++ WindowType window_type; /* frame queuing strategy */ ++ int normalize_input; /* 1 = divide uint8 by 255 before packing */ ++ int normalize_output; /* 1 = multiply float32 by 255 before clipping to uint8 */ ++} ModelDesc; ++ ++/* ---- forward declarations for the per-model I/O functions ---- */ ++static int pack_input_basicvsr (IVSRModel *m, void *base, DNNData *in, TaskItem *t); ++static void unpack_output_basicvsr(IVSRModel *m, TaskItem *t, DNNData *out); ++/* Generic window-based pack/unpack — used by all JSON-config models (model_config set) ++ * and any future built-in models whose I/O fits the generic path. */ ++static int pack_input_window (IVSRModel *m, void *base, DNNData *in, TaskItem *t); ++static void unpack_output_window (IVSRModel *m, TaskItem *t, DNNData *out); ++ ++/* ---- The descriptor table ---- ++ * Only BasicVSR lives here because its multi-frame out_queue output requires ++ * dedicated C functions that cannot be expressed in a JSON config. ++ * All other models: omit model_type and set model_config=.json instead. ++ * Canonical configs: model_configs/{videoproc,edsr,custvsr,tsenet,rife,videoseal}_config.json ++ * -------------------------------------------------------------------- */ ++static const ModelDesc model_table[] = { ++ /* [BASICVSR] */ { "BasicVSR", 0, 1, 32, "NFHWC", NULL, "NFHWC", NULL, "RGB", DCO_RGB, ++ pack_input_basicvsr, unpack_output_basicvsr }, ++}; ++av_unused static void model_table_size_check(void) { ++ /* Compile-time assert: table has exactly MODEL_TYPE_NUM entries */ ++ typedef char model_table_wrong_size[(sizeof(model_table)/sizeof(model_table[0]) == MODEL_TYPE_NUM) ? 1 : -1]; ++} ++ ++/* Return the active model descriptor: dynamic (JSON) or built-in table. */ ++static inline const ModelDesc *get_model_desc(const IVSRModel *m) ++{ ++ return m->dynamic_desc ? m->dynamic_desc : &model_table[m->model_type]; ++} ++ + static int get_datatype_size(DNNDataType dt) + { + switch (dt) { +@@ -264,6 +371,417 @@ static void set_dnndata_info(DNNData *dnn_data, const tensor_desc_t* tensor) { + } + } + ++/* --------------------------------------------------------------------------- ++ * Per-model input packing and output unpacking functions ++ * ++ * Signature: ++ * pack_input – returns 0 on success, DNN_MORE_FRAMES, or AVERROR(*). ++ * Sets input->data back to base before returning on success. ++ * unpack_output – writes directly into task->out_frame->data[0]. ++ * ++ * Generic path (NULL function pointer): ff_proc_from_frame_to_dnn / ++ * ff_proc_from_dnn_to_frame with optional NCHW↔NHWC conversion. ++ * --------------------------------------------------------------------------- ++ */ ++ ++/* BasicVSR: read nif frames from task->in_queue. */ ++static int pack_input_basicvsr(IVSRModel *m, void *base, DNNData *input, TaskItem *task) ++{ ++ DnnContext *ctx = m->ctx; ++ AVFrame *tmp_frame; ++ int read_frame_num = 0; ++ ++ for (int j = 0; j < m->nif; j++) { ++ if (av_fifo_can_read(task->in_queue)) { ++ av_fifo_read(task->in_queue, &tmp_frame, 1); ++ ff_proc_from_frame_to_dnn(tmp_frame, input, m->model.filter_ctx); ++ if (input->channels != 1 && input->layout == DL_NONE) ++ convert_nhwc_to_nchw(input->data, 1, input->channels, ++ input->height, input->width, input->dt); ++ input->data += input->height * input->width * ++ input->channels * get_datatype_size(input->dt); ++ read_frame_num++; ++ } ++ } ++ input->data = base; ++ if (read_frame_num < m->nif) ++ av_log(ctx, AV_LOG_ERROR, ++ "Read frame number is %d less than the model requirement %d!!!\n", ++ read_frame_num, m->nif); ++ return 0; ++} ++ ++/* BasicVSR: multi-frame output — iterate over task->out_queue. */ ++static void unpack_output_basicvsr(IVSRModel *m, TaskItem *task, DNNData *output) ++{ ++ int offset = 0; ++ AVFrame *tmp_frame; ++ ++ do { ++ int ret = av_fifo_peek(task->out_queue, &tmp_frame, 1, offset); ++ if (ret == 0) { ++ if (output->channels != 1 && output->layout == DL_NONE) ++ convert_nchw_to_nhwc(output->data, 1, output->channels, ++ output->height, output->width, output->dt); ++ ff_proc_from_dnn_to_frame(tmp_frame, output, &m->model.filter_ctx); ++ if (tmp_frame->color_range == AVCOL_RANGE_MPEG && output->channels == 1) { ++ uint8_t min_x = 16, max_x = 235; ++ for (int index = 0; index < tmp_frame->height * tmp_frame->linesize[0]; ++index) ++ tmp_frame->data[0][index] = (uint8_t)clamp(tmp_frame->data[0][index], min_x, max_x); ++ } ++ output->data += output->height * output->width * ++ output->channels * get_datatype_size(output->dt); ++ } ++ offset++; ++ } while (offset != m->nif); ++} ++ ++/* Generic single-frame + sliding-window packing for all JSON-config models ++ * (when model_config is set) and any built-in model whose I/O fits the generic path. ++ * Dispatches on md->window_type; uses md->normalize_input for float scaling. */ ++static int pack_input_window(IVSRModel *m, void *base, DNNData *input, TaskItem *task) ++{ ++ const ModelDesc *md = get_model_desc(m); ++ ++ switch (md->window_type) { ++ case WINDOW_IN_QUEUE: ++ return pack_input_basicvsr(m, base, input, task); ++ ++ case WINDOW_SLIDING: { ++ AVFrame *tmp = av_frame_alloc(); ++ if (!tmp) return AVERROR(ENOMEM); ++ if (av_frame_ref(tmp, task->in_frame) < 0) { ++ av_frame_free(&tmp); ++ return AVERROR(ENOMEM); ++ } ++ av_fifo_write(m->frame_queue, &tmp, 1); ++ ++ if (md->sliding_window_init_dup && m->sliding_window_frame_num == 0) { ++ tmp = av_frame_alloc(); ++ if (!tmp) return AVERROR(ENOMEM); ++ if (av_frame_ref(tmp, task->in_frame) < 0) { ++ av_frame_free(&tmp); ++ return AVERROR(ENOMEM); ++ } ++ av_fifo_write(m->frame_queue, &tmp, 1); ++ m->sliding_window_frame_num++; ++ } ++ ++ if (av_fifo_can_read(m->frame_queue) < (size_t)m->nif) ++ return DNN_MORE_FRAMES; ++ ++ av_assert0(av_fifo_can_read(m->frame_queue) == (size_t)m->nif); ++ AVFrame **frames = av_mallocz(sizeof(AVFrame *) * m->nif); ++ if (!frames) return AVERROR(ENOMEM); ++ av_fifo_peek(m->frame_queue, frames, m->nif, 0); ++ for (int idx = 0; idx < m->nif; idx++) { ++ if (md->normalize_input && input->dt == DNN_FLOAT) { ++ /* float32 NCHW with /255 normalisation (RIFE style) */ ++ AVFrame *frm = frames[idx]; ++ float *dst = (float *)input->data; ++ int vw = input->width, vh = input->height, ch = input->channels; ++ int ps = vw * vh; ++ for (int y = 0; y < vh; y++) { ++ uint8_t *row = frm->data[0] + y * frm->linesize[0]; ++ for (int x = 0; x < vw; x++) ++ for (int c = 0; c < ch; c++) ++ dst[c * ps + y * vw + x] = row[x * ch + c] / 255.0f; ++ } ++ } else { ++ ff_proc_from_frame_to_dnn(frames[idx], input, m->model.filter_ctx); ++ if (input->channels != 1 && input->layout == DL_NONE) ++ convert_nhwc_to_nchw(input->data, 1, input->channels, ++ input->height, input->width, input->dt); ++ } ++ input->data += (size_t)input->height * input->width * ++ input->channels * get_datatype_size(input->dt); ++ } ++ input->data = base; ++ av_freep(&frames); ++ av_fifo_read(m->frame_queue, &tmp, 1); ++ av_frame_unref(tmp); ++ av_frame_free(&tmp); ++ return 0; ++ } ++ ++ case WINDOW_SINGLE: ++ default: { ++ /* For u8/u16 inputs, fall back to the generic ff_proc path. ++ * This covers VideoProc, EDSR, CustVSR and any JSON model whose ++ * in_precision is depth-derived (NULL in descriptor). */ ++ if (input->dt != DNN_FLOAT) { ++ ff_proc_from_frame_to_dnn(task->in_frame, input, m->model.filter_ctx); ++ if (input->channels != 1 && input->layout == DL_NONE) ++ convert_nhwc_to_nchw(input->data, 1, input->channels, ++ input->height, input->width, input->dt); ++ input->data = base; ++ return 0; ++ } ++ /* Single frame, NCHW float32, optional /255 normalisation. */ ++ AVFrame *frame = task->in_frame; ++ float *dst = (float *)input->data; ++ int vw = input->width, vh = input->height; ++ int ps = vw * vh; ++ if (md->normalize_input) { ++ for (int y = 0; y < vh; y++) { ++ uint8_t *row = frame->data[0] + y * frame->linesize[0]; ++ for (int x = 0; x < vw; x++) { ++ dst[0 * ps + y * vw + x] = row[x * 3 + 0] / 255.0f; ++ dst[1 * ps + y * vw + x] = row[x * 3 + 1] / 255.0f; ++ dst[2 * ps + y * vw + x] = row[x * 3 + 2] / 255.0f; ++ } ++ } ++ } else { ++ for (int y = 0; y < vh; y++) { ++ uint8_t *row = frame->data[0] + y * frame->linesize[0]; ++ for (int x = 0; x < vw; x++) { ++ dst[0 * ps + y * vw + x] = (float)row[x * 3 + 0]; ++ dst[1 * ps + y * vw + x] = (float)row[x * 3 + 1]; ++ dst[2 * ps + y * vw + x] = (float)row[x * 3 + 2]; ++ } ++ } ++ } ++ input->data = base; ++ return 0; ++ } ++ } ++} ++ ++/* Generic float32 NCHW output unpack for JSON-loaded models. ++ * Uses md->normalize_output to decide whether to multiply by 255. */ ++static void unpack_output_window(IVSRModel *m, TaskItem *task, DNNData *output) ++{ ++ const ModelDesc *md = get_model_desc(m); ++ const float *src = (const float *)output->data; ++ int fw = task->out_frame->width, fh = task->out_frame->height; ++ ++ /* Single-channel (Y-plane only) model: write Y plane only. ++ * U/V chroma planes are copied by copy_uv_planes() in vf_dnn_processing.c ++ * after ff_dnn_get_result() returns, so no chroma copy is needed here. ++ * Copying chroma from task->in_frame inside this callback is unsafe: ++ * inference_done++ can be reordered before in_frame reads under -O3, ++ * causing a use-after-free race with the main thread freeing in_frame. */ ++ if (output->channels == 1) { ++ for (int h = 0; h < fh; h++) { ++ uint8_t *row = task->out_frame->data[0] + h * task->out_frame->linesize[0]; ++ for (int w = 0; w < fw; w++) { ++ float val = src[h * fw + w]; ++ row[w] = (uint8_t)av_clip( ++ (int)((md->normalize_output ? val * 255.0f : val) + 0.5f), 0, 255); ++ } ++ } ++ return; ++ } ++ ++ for (int h = 0; h < fh; h++) { ++ uint8_t *row = task->out_frame->data[0] + h * task->out_frame->linesize[0]; ++ for (int w = 0; w < fw; w++) { ++ float r, g, b; ++ if (output->layout == DL_NCHW) { ++ int ps = output->height * output->width; ++ r = src[0 * ps + h * output->width + w]; ++ g = src[1 * ps + h * output->width + w]; ++ b = src[2 * ps + h * output->width + w]; ++ } else { ++ int base = (h * output->width + w) * 3; ++ r = src[base + 0]; g = src[base + 1]; b = src[base + 2]; ++ } ++ if (md->normalize_output) { ++ row[w * 3 + 0] = (uint8_t)av_clip((int)(r * 255.0f + 0.5f), 0, 255); ++ row[w * 3 + 1] = (uint8_t)av_clip((int)(g * 255.0f + 0.5f), 0, 255); ++ row[w * 3 + 2] = (uint8_t)av_clip((int)(b * 255.0f + 0.5f), 0, 255); ++ } else { ++ row[w * 3 + 0] = (uint8_t)av_clip((int)(r + 0.5f), 0, 255); ++ row[w * 3 + 1] = (uint8_t)av_clip((int)(g + 0.5f), 0, 255); ++ row[w * 3 + 2] = (uint8_t)av_clip((int)(b + 0.5f), 0, 255); ++ } ++ } ++ } ++} ++ ++/* Free all heap-allocated strings in a dynamic ModelDesc, then the struct itself. */ ++static void free_dynamic_desc(ModelDesc **pp) ++{ ++ if (!pp || !*pp) return; ++ ModelDesc *d = *pp; ++ /* Cast away const: these strings were av_strdup'd by parse_model_config_json */ ++ av_freep((void *)&d->name); ++ av_freep((void *)&d->in_layout); ++ av_freep((void *)&d->in_precision); ++ av_freep((void *)&d->out_layout); ++ av_freep((void *)&d->out_precision); ++ av_freep((void *)&d->model_color); ++ av_freep(pp); ++} ++ ++/* --------------------------------------------------------------------------- ++ * Minimal flat-JSON parser for iVSR model config files. ++ * Handles string, integer, and boolean values in a flat object. ++ * Keys starting with '_' (comment markers) are silently ignored. ++ * Returns 0 on success, negative AVERROR on failure. ++ * --------------------------------------------------------------------------- ++ */ ++static int parse_model_config_json(DnnContext *ctx, const char *path, ModelDesc **out_desc) ++{ ++ AVIOContext *pb = NULL; ++ int64_t file_size; ++ char *buf = NULL; ++ ModelDesc *desc; ++ int ret = 0; ++ ++ ret = avio_open(&pb, path, AVIO_FLAG_READ); ++ if (ret < 0) { ++ av_log(ctx, AV_LOG_ERROR, "model_config: cannot open '%s': %d\n", path, ret); ++ return ret; ++ } ++ file_size = avio_size(pb); ++ if (file_size <= 0 || file_size > 65536) { ++ av_log(ctx, AV_LOG_ERROR, "model_config: invalid file size %" PRId64 "\n", file_size); ++ avio_closep(&pb); ++ return AVERROR(EINVAL); ++ } ++ buf = av_malloc(file_size + 1); ++ if (!buf) { avio_closep(&pb); return AVERROR(ENOMEM); } ++ if (avio_read(pb, (unsigned char *)buf, (int)file_size) != (int)file_size) { ++ av_log(ctx, AV_LOG_ERROR, "model_config: read error\n"); ++ av_freep(&buf); avio_closep(&pb); return AVERROR(EIO); ++ } ++ buf[file_size] = '\0'; ++ avio_closep(&pb); ++ ++ desc = av_mallocz(sizeof(ModelDesc)); ++ if (!desc) { av_freep(&buf); return AVERROR(ENOMEM); } ++ ++ /* Set defaults */ ++ desc->pack_input = pack_input_window; ++ desc->unpack_output = unpack_output_window; ++ desc->nif_override = 1; ++ desc->channel_divisor = 1; ++ desc->normalize_input = 1; ++ desc->normalize_output = 1; ++ desc->window_type = WINDOW_SINGLE; ++ desc->out_order = DCO_RGB; ++ ++ /* Single-pass scan: find "key": value pairs */ ++ const char *p = buf; ++ while (*p) { ++ /* Find opening quote of a key */ ++ while (*p && *p != '"') p++; ++ if (!*p) break; ++ p++; /* skip " */ ++ ++ const char *key_start = p; ++ while (*p && *p != '"') p++; ++ if (!*p) break; ++ int key_len = (int)(p - key_start); ++ p++; /* skip closing " */ ++ ++ /* Skip whitespace + ':' */ ++ while (*p && (*p == ' ' || *p == '\t' || *p == '\n' || *p == '\r' || *p == ':')) p++; ++ if (!*p) break; ++ ++ /* Silently skip comment keys (_comment_*, etc.) */ ++ if (key_len > 0 && key_start[0] == '_') { ++ if (*p == '"') { ++ p++; ++ while (*p && *p != '"') { if (*p == '\\') p++; p++; } ++ if (*p) p++; ++ } else { ++ while (*p && *p != ',' && *p != '}') p++; ++ } ++ continue; ++ } ++ ++#define MKEY(k) (key_len == (int)(sizeof(k)-1) && !av_strncasecmp(key_start, k, key_len)) ++ ++ if (*p == '"') { ++ /* String value */ ++ p++; ++ const char *vs = p; ++ while (*p && *p != '"') { if (*p == '\\') p++; p++; } ++ int vl = (int)(p - vs); ++ if (*p) p++; ++ char val[256] = {0}; ++ if (vl >= (int)sizeof(val)) vl = sizeof(val) - 1; ++ memcpy(val, vs, vl); ++ ++ if (MKEY("name")) desc->name = av_strdup(val); ++ else if (MKEY("in_layout")) desc->in_layout = av_strdup(val); ++ else if (MKEY("in_precision")) desc->in_precision = av_strdup(val); ++ else if (MKEY("out_layout")) desc->out_layout = av_strdup(val); ++ else if (MKEY("out_precision"))desc->out_precision= av_strdup(val); ++ else if (MKEY("model_color")) desc->model_color = av_strdup(val); ++ else if (MKEY("out_order")) { ++ if (!av_strcasecmp(val, "RGB")) desc->out_order = DCO_RGB; ++ else if (!av_strcasecmp(val, "BGR")) desc->out_order = DCO_BGR; ++ else desc->out_order = DCO_NONE; ++ } else if (MKEY("window_type")) { ++ if (!av_strcasecmp(val, "sliding")) desc->window_type = WINDOW_SLIDING; ++ else if (!av_strcasecmp(val, "in_queue")) desc->window_type = WINDOW_IN_QUEUE; ++ else desc->window_type = WINDOW_SINGLE; ++ } else { ++ av_log(ctx, AV_LOG_WARNING, ++ "model_config: unknown string key '%.*s'\n", key_len, key_start); ++ } ++ } else { ++ /* Numeric or boolean value — read until delimiter */ ++ const char *vs = p; ++ while (*p && *p != ',' && *p != '}' && *p != '\n' && *p != '\r') p++; ++ int vl = (int)(p - vs); ++ while (vl > 0 && (vs[vl-1]==' '||vs[vl-1]=='\t')) vl--; ++ char val[64] = {0}; ++ if (vl >= (int)sizeof(val)) vl = sizeof(val) - 1; ++ memcpy(val, vs, vl); ++ ++ /* Parse integer manually (avoid stdlib dependency) */ ++ int ival = 0, sign = 1; ++ const char *ip = val; ++ while (*ip == ' ') ip++; ++ if (*ip == '-') { sign = -1; ip++; } ++ while (*ip >= '0' && *ip <= '9') { ival = ival * 10 + (*ip - '0'); ip++; } ++ ival *= sign; ++ int bval = (!av_strcasecmp(val, "true") || ival != 0) ? 1 : 0; ++ ++ if (MKEY("nif")) desc->nif_override = ival; ++ else if (MKEY("align")) desc->align = ival; ++ else if (MKEY("channel_divisor")) desc->channel_divisor = (ival > 1 ? ival : 1); ++ else if (MKEY("color_format_auto")) desc->color_format_auto = ival; ++ else if (MKEY("normalize_input")) desc->normalize_input = bval; ++ else if (MKEY("normalize_output")) desc->normalize_output = bval; ++ else if (MKEY("output_passthrough_dims")) desc->output_passthrough_dims = bval; ++ else if (MKEY("out_precision_depth_derived")) desc->out_precision_depth_derived = bval; ++ else if (MKEY("window_init_dup")) desc->sliding_window_init_dup = bval; ++ else { ++ av_log(ctx, AV_LOG_WARNING, ++ "model_config: unknown key '%.*s'\n", key_len, key_start); ++ } ++ } ++#undef MKEY ++ } ++ ++ av_freep(&buf); ++ ++ /* Validate required fields */ ++ if (!desc->name) { ++ av_log(ctx, AV_LOG_ERROR, "model_config: required field 'name' missing in '%s'\n", path); ++ free_dynamic_desc(&desc); ++ return AVERROR(EINVAL); ++ } ++ if (!desc->in_layout) { ++ av_log(ctx, AV_LOG_ERROR, "model_config: required field 'in_layout' missing in '%s'\n", path); ++ free_dynamic_desc(&desc); ++ return AVERROR(EINVAL); ++ } ++ ++ *out_desc = desc; ++ return 0; ++} ++ ++/* --------------------------------------------------------------------------- ++ * End of per-model I/O functions ++ * --------------------------------------------------------------------------- ++ */ ++ + /* returns + * DNN_GENERIC_ERROR, + * DNN_MORE_FRAMES - waiting for more input frames, +@@ -278,7 +777,6 @@ static int fill_model_input_ivsr(IVSRModel * ivsr_model, + DNNData input; + LastLevelTaskItem *lltask; + TaskItem *task; +- AVFrame *tmp_frame = NULL; + void *in_data = NULL; + float normalize_factor = ctx->ivsr_option.normalize_factor; + int padding_height = 0, padding_width = 0; +@@ -298,9 +796,13 @@ static int fill_model_input_ivsr(IVSRModel * ivsr_model, + } + + set_dnndata_info(&input, &input_tensor_desc_get); +- if (ivsr_model->model_type == TSENET) { +- input.dims[dnn_get_channel_idx_by_layout(input.layout)] /= 3; +- input.channels = input.channels / 3; ++ /* Apply channel divisor from model descriptor (e.g. RIFE /2, TSENET /3). */ ++ { ++ int div = get_model_desc(ivsr_model)->channel_divisor; ++ if (div > 1) { ++ input.dims[dnn_get_channel_idx_by_layout(input.layout)] /= div; ++ input.channels /= div; ++ } + } + + input.data = request->in_frames; +@@ -346,86 +848,24 @@ static int fill_model_input_ivsr(IVSRModel * ivsr_model, + } + input.data = in_data; + } +- if (ivsr_model->model_type == BASICVSR && ivsr_model->nif != 1) { +- int read_frame_num = 0; +- for (int j = 0; j < ivsr_model->nif; j++) { +- if (av_fifo_can_read(task->in_queue)) { +- av_fifo_read(task->in_queue, &tmp_frame, 1); +- ff_proc_from_frame_to_dnn(tmp_frame, &input, +- ivsr_model->model.filter_ctx); +- // convert buffer from NHWC to NCHW when C != 1 +- if (input.channels != 1 && input.layout == DL_NONE ) +- convert_nhwc_to_nchw(input.data, 1, input.channels, input.height, input.width, input.dt); +- input.data += +- input.height * input.width * +- input.channels * get_datatype_size(input.dt); +- read_frame_num++; +- } +- } +- input.data = in_data; +- if (read_frame_num < ivsr_model->nif) +- av_log(ctx, AV_LOG_ERROR, +- "Read frame number is %d less than the model requirement %d!!!\n", +- read_frame_num, ivsr_model->nif); +- } else if (ivsr_model->model_type == TSENET) { +- //1. copy the input_frame(ref the buffer) and put into ivsr_model->fame_queue +- tmp_frame = av_frame_alloc(); +- if(av_frame_ref(tmp_frame, task->in_frame) < 0) { +- return AVERROR(ENOMEM); +- } +- +- av_fifo_write(ivsr_model->frame_queue, &tmp_frame, 1); +- static int frame_num = 0; +- if (frame_num == 0) { +- //For the first pic in the stream +- tmp_frame = av_frame_alloc(); +- if(av_frame_ref(tmp_frame, task->in_frame) < 0) { +- return AVERROR(ENOMEM); +- } +- av_fifo_write(ivsr_model->frame_queue, &tmp_frame, 1); +- frame_num++; +- } +- //2. check if queue size is >= nif +- if (av_fifo_can_read(ivsr_model->frame_queue) >= ivsr_model->nif) { +- //2.1 prepare dnn data into request +- av_assert0(av_fifo_can_read(ivsr_model->frame_queue) == ivsr_model->nif); +- AVFrame **input_frames = av_mallocz(sizeof(AVFrame *) * ivsr_model->nif); +- av_fifo_peek(ivsr_model->frame_queue, input_frames, ivsr_model->nif, 0); +- for (int idx = 0; idx < ivsr_model->nif; idx++) { +- //INFO: the 3 frames in frame_queue are: (N-2)th, (N-1)th, (N)th +- ff_proc_from_frame_to_dnn(input_frames[idx], &input, ivsr_model->model.filter_ctx); +- //NHWC->NCHW was processed in ff_proc_from_frame_to_dnn() if input.layout is set +- if (input.channels != 1 && input.layout == DL_NONE ) +- convert_nhwc_to_nchw(input.data, 1, input.channels, input.height, input.width, input.dt); +- input.data += input.height * input.width * input.channels * get_datatype_size(input.dt); +- } +- input.data = in_data; +- //pop the (N-2)th frame from frame_queue and free it +- av_fifo_read(ivsr_model->frame_queue, &tmp_frame, 1); +- av_frame_unref(tmp_frame); +- av_frame_free(&tmp_frame); +- // INFO: for the last frame, peek_back and pop_front get the same frame, so don't have to handle EOS specifically +- } else { +- return DNN_MORE_FRAMES; +- } ++ /* ---- Input packing: table-driven --------------------------------- */ ++ if (get_model_desc(ivsr_model)->pack_input != NULL) { ++ /* Delegate to per-model packing function. */ ++ int ret = get_model_desc(ivsr_model)->pack_input( ++ ivsr_model, in_data, &input, task); ++ if (ret != 0) ++ return ret; + } else { +- // ff_proc_from_frame_to_dnn will perform normalization by calling +- // uint_y_to_float_y_wrapper in swscale_unscaled.c +- // So, for the inputs do not need normalization, normalization facotr should be multiplied back. +- // Same to ff_proc_from_dnn_to_frame. ++ /* Generic path: ff_proc_from_frame_to_dnn + optional NCHW conversion. */ + ff_proc_from_frame_to_dnn(task->in_frame, &input, + ivsr_model->model.filter_ctx); +- if (input.channels != 1 && (input.layout == DL_NONE)) { ++ if (input.channels != 1 && input.layout == DL_NONE) + convert_nhwc_to_nchw(input.data, 1, input.channels, input.height, input.width, input.dt); +- } +- + if (normalize_factor != 1 && input.dt == DNN_FLOAT && + (fabsf(input.scale - 1.0f) > 1e-6f || fabsf(input.scale) < 1e-6f)) { +- // do not need to covert buffer from NHWC to NCHW if the channels is 1, only need to mulitple normalize_factor + #pragma omp parallel for +- for (int pos = 0; pos < input.height * input.width * input.channels; pos++) { +- ((float*)input.data)[pos] = ((float*)input.data)[pos] * normalize_factor; +- } ++ for (int pos = 0; pos < input.height * input.width * input.channels; pos++) ++ ((float*)input.data)[pos] *= normalize_factor; + } + } + } +@@ -453,8 +893,6 @@ static void infer_completion_callback(void *args) + SafeQueue *requestq = ivsr_model->request_queue; + DNNData output; + DnnContext *ctx = ivsr_model->ctx; +- AVFrame *tmp_frame = NULL; +- int offset = 0; + float normalize_factor = ctx->ivsr_option.normalize_factor; + tensor_desc_t output_tensor_desc_get = { + .precision = {0}, +@@ -476,23 +914,12 @@ static void infer_completion_callback(void *args) + + output.data = request->out_frames; + // Set output mean/scale to meet the logistics in func ff_proc_from_dnn_to_frame() @dnn_io_proc.c + // For *passthrough* cases, the OV backend can help to do the normalization. + // FIXME: Apt to make mistakes when changes are made here! + output.mean = 0.0f; + output.scale = output.dt == DNN_FLOAT ? 0.0f : 1.0f; +- // set order based on model type +- switch (ivsr_model->model_type) +- { +- case BASICVSR: +- case VIDEOPROC: +- case EDSR: +- case TSENET: +- output.order = DCO_RGB; +- break; +- default: +- output.order = DCO_NONE; +- break; +- } ++ /* output.order from descriptor table */ ++ output.order = get_model_desc(ivsr_model)->out_order; + + const AVPixFmtDescriptor* pix_desc = av_pix_fmt_desc_get(task->out_frame->format); + const AVComponentDescriptor* comp_desc = &pix_desc->comp[0]; +@@ -507,70 +935,39 @@ static void infer_completion_callback(void *args) + &output, + ivsr_model->model.filter_ctx); + } else { +- if (ivsr_model->model_type == BASICVSR && ivsr_model->nif != 1) { +- do { +- int ret = +- av_fifo_peek(task->out_queue, &tmp_frame, 1, +- offset); +- if (ret == 0) { +- if (output.channels != 1 && output.layout == DL_NONE) { +- convert_nchw_to_nhwc(output.data, 1, output.channels, output.height, output.width, output.dt); +- } +- ff_proc_from_dnn_to_frame(tmp_frame, &output, +- &ivsr_model->model.filter_ctx); +- // clamp output to [16, 235] range for Y plane when color range of output is TV range, +- // assume model only process Y plane when output.channels = 1. AVCOL_RANGE_MPEG is mean tv range. +- if (tmp_frame->color_range == AVCOL_RANGE_MPEG && output.channels == 1) { +- uint8_t min_x = 16, max_x = 235; +- for (int index = 0; index < tmp_frame->height * tmp_frame->linesize[0]; ++index) { +- uint8_t value = tmp_frame->data[0][index]; +- tmp_frame->data[0][index] = (uint8_t)clamp(tmp_frame->data[0][index], min_x, max_x); +- } +- } +- output.data += +- output.height * output.width * +- output.channels * get_datatype_size(output.dt); +- } +- offset++; +- } while (offset != ivsr_model->nif); ++ /* ---- Output unpacking: table-driven --------------------------- */ ++ if (get_model_desc(ivsr_model)->unpack_output != NULL) { ++ /* Custom unpacking (BASICVSR, RIFE, VideoSeal, etc.). */ ++ get_model_desc(ivsr_model)->unpack_output( ++ ivsr_model, task, &output); + } else { +- if (output.channels != 1 && output.layout == DL_NONE) { +- //convert buffer from NCHW to NHWC ++ /* Generic path: optional NCHW→NHWC, then ff_proc_from_dnn_to_frame. */ ++ if (output.channels != 1 && (output.layout == DL_NONE || output.layout == DL_NCHW)) { + convert_nchw_to_nhwc(output.data, 1, output.channels, output.height, output.width, output.dt); ++ if (output.layout == DL_NCHW) ++ output.layout = DL_NONE; + } +- +- // For the outputs do not need normalization, normalization factor should be divided back. e.g. EDSR + if (normalize_factor != 1 && output.dt == DNN_FLOAT && + (fabsf(output.scale - 1.0f) > 1e-6f || fabsf(output.scale) < 1e-6f)) { + #pragma omp parallel for +- // only need to devide by normalize_factor for channels = 1. +- for (int pos = 0; pos < output.height * output.width * output.channels; pos++) { +- ((float*)output.data)[pos] = ((float*)output.data)[pos] / normalize_factor; +- } ++ for (int pos = 0; pos < output.height * output.width * output.channels; pos++) ++ ((float*)output.data)[pos] /= normalize_factor; + } +- + ff_proc_from_dnn_to_frame(task->out_frame, &output, + &ivsr_model->model.filter_ctx); +- // clamp output to [16, 235] range for Y plane when color range of output is TV range, +- // assume model only process Y plane when output.channels = 1. AVCOL_RANGE_MPEG is mean tv range. + if (task->out_frame->color_range == AVCOL_RANGE_MPEG && output.channels == 1) { + if (bits == 8) { + uint8_t min_x = 16, max_x = 235; +- for (int index = 0; index < task->out_frame->height * task->out_frame->linesize[0]; +- ++index) { +- uint8_t value = task->out_frame->data[0][index]; +- task->out_frame->data[0][index] = (uint8_t)clamp(task->out_frame->data[0][index], +- min_x, max_x); +- } ++ for (int index = 0; index < task->out_frame->height * task->out_frame->linesize[0]; ++index) ++ task->out_frame->data[0][index] = (uint8_t)clamp(task->out_frame->data[0][index], min_x, max_x); + } else if (bits == 10) { + uint16_t min_x = 64, max_x = 940; +- uint16_t* dstPtr = (uint16_t*)task->out_frame->data[0]; +- ptrdiff_t dstStrideUint16 = task->out_frame->linesize[0] >> 1; ++ uint16_t *dstPtr = (uint16_t *)task->out_frame->data[0]; ++ ptrdiff_t stride = task->out_frame->linesize[0] >> 1; + for (int y = 0; y < task->out_frame->height; ++y) { +- for (int x = 0; x < task->out_frame->width; ++x) { ++ for (int x = 0; x < task->out_frame->width; ++x) + dstPtr[x] = (uint16_t)clamp(dstPtr[x], min_x, max_x); +- } +- dstPtr += dstStrideUint16; ++ dstPtr += stride; + } + } + } +@@ -621,11 +1018,14 @@ static int get_input_ivsr(void *model, DNNData * input, + } + + set_dnndata_info(input, &input_tensor_desc_get); +- if (ivsr_model->model_type == TSENET) { +- input->dims[dnn_get_channel_idx_by_layout(input->layout)] /= 3; +- input->channels /= 3; ++ /* Apply channel divisor from model descriptor (e.g. RIFE /2, TSENET /3). */ ++ { ++ int div = get_model_desc(ivsr_model)->channel_divisor; ++ if (div > 1) { ++ input->dims[dnn_get_channel_idx_by_layout(input->layout)] /= div; ++ input->channels /= div; ++ } + } +- + // hard code to pass check_modelinput_inlink() that requires DNN_FLOAT of model_input->dt + input->dt = DNN_FLOAT; + +@@ -731,7 +1131,7 @@ static int get_output_ivsr(void *model, const char *input_name, + *output_height = output.height; + *output_width = output.width; + +- if (ivsr_model->model_type == VIDEOPROC) { ++ if (get_model_desc(ivsr_model)->output_passthrough_dims) { + *output_height = input_height; + *output_width = input_width; + } +@@ -819,6 +1219,15 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + + ivsr_model->model_type = ctx->ivsr_option.model_type; + ++ /* When model_config is provided, parse the JSON descriptor and use the JSON ++ * dispatch path. model_type is ignored — dynamic_desc takes priority in ++ * get_model_desc(). Omitting model_config selects the BasicVSR built-in. */ ++ if (ctx->ivsr_option.model_config) { ++ if (parse_model_config_json(ctx, ctx->ivsr_option.model_config, ++ &ivsr_model->dynamic_desc) < 0) ++ goto err; ++ } ++ + // set ivsr config + // input model + ivsr_model->config = create_and_link_config(NULL, INPUT_MODEL, ctx->model_filename, ctx); +@@ -876,26 +1285,25 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + default: + break; + } +- // set element type of output for EDSR +- if (ivsr_model->model_type == EDSR) { +- if (desc->comp[0].depth == 8) { ++ /* Derive output precision from bit depth when flagged in descriptor (e.g. EDSR). */ ++ if (get_model_desc(ivsr_model)->out_precision_depth_derived) { ++ if (desc->comp[0].depth == 8) + strcpy(output_tensor_desc_set.precision, "u8"); +- } else if (desc->comp[0].depth == 10 || desc->comp[0].depth == 16) { ++ else if (desc->comp[0].depth == 10 || desc->comp[0].depth == 16) + strcpy(output_tensor_desc_set.precision, "u16"); +- } + } +- // customize layout for Basic_VSR and TSENet +- if (ivsr_model->model_type == BASICVSR) { +- strcpy(input_tensor_desc_set.layout, "NFHWC"); +- strcpy(output_tensor_desc_set.layout, "NFHWC"); +- } else if (ivsr_model->model_type == TSENET) { +- //For TSENet, it's not typical N'C'HW, so do the NHWC->NCHW transion in plugin +- strcpy(input_tensor_desc_set.layout, "NCHW"); +- if (desc->comp[0].depth == 8) { +- strcpy(input_tensor_desc_set.precision, "u8"); +- } else if (desc->comp[0].depth == 10 || desc->comp[0].depth == 16) { +- strcpy(input_tensor_desc_set.precision, "u16"); +- } ++ /* Apply layout and precision from descriptor table. ++ * in_precision == NULL means depth-derived (u8/u16 already set above). */ ++ { ++ const ModelDesc *md = get_model_desc(ivsr_model); ++ if (md->in_layout) ++ strcpy(input_tensor_desc_set.layout, md->in_layout); ++ if (md->in_precision) ++ strcpy(input_tensor_desc_set.precision, md->in_precision); ++ if (md->out_layout) ++ strcpy(output_tensor_desc_set.layout, md->out_layout); ++ if (md->out_precision) ++ strcpy(output_tensor_desc_set.precision, md->out_precision); + } + // set scale for non-float type of input + if (fabsf(ctx->ivsr_option.normalize_factor - 1) < 1e-6f && +@@ -935,25 +1343,19 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + default: + break; + } +- // set color format of model required +- switch (ivsr_model->model_type) ++ /* Set model colour format from descriptor. Use color_format_auto for NULL entries. */ + { +- case BASICVSR: +- case EDSR: +- case TSENET: +- strcpy(input_tensor_desc_set.model_color_format, "RGB"); +- break; +- case VIDEOPROC: +- if (desc->flags & AV_PIX_FMT_FLAG_RGB) +- strcpy(input_tensor_desc_set.model_color_format, "RGB"); +- else ++ const ModelDesc *md = get_model_desc(ivsr_model); ++ if (md->model_color) { ++ strcpy(input_tensor_desc_set.model_color_format, md->model_color); ++ } else if (md->color_format_auto == 1) { ++ /* VideoProc-style: auto-detect from pixel format */ ++ strcpy(input_tensor_desc_set.model_color_format, ++ (desc->flags & AV_PIX_FMT_FLAG_RGB) ? "RGB" : "I420_Three_Planes"); ++ } else if (md->color_format_auto == 2) { ++ /* CustVSR-style: always YUV */ + strcpy(input_tensor_desc_set.model_color_format, "I420_Three_Planes"); +- break; +- case CUSTVSR: +- strcpy(input_tensor_desc_set.model_color_format, "I420_Three_Planes"); +- break; +- default: +- break; ++ } + } + config_input_tensor = create_and_link_config(config_input_res, INPUT_TENSOR_DESC_SETTING, &input_tensor_desc_set, ctx); + config_output_tensor = create_and_link_config(config_input_tensor, OUTPUT_TENSOR_DESC_SETTING, &output_tensor_desc_set, ctx); +@@ -964,33 +1366,23 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + if (config_nireq == NULL) + goto err; + +- //TODO: reshape setting follows NHW layout. Hardcode the batch_size as 1. ++ /* Alignment and shape from descriptor. */ ++ { ++ int a = get_model_desc(ivsr_model)->align; ++ if (a > 0) { ++ frame_h = (frame_h + a - 1) / a * a; ++ frame_w = (frame_w + a - 1) / a * a; ++ } ++ } + char shape_string[40] = {0}; +- switch (ivsr_model->model_type) { +- case BASICVSR: +- //for BasicVSR, the width requires 32-aligned +- frame_w = (frame_w + 32 - 1) / 32 * 32; +- sprintf(shape_string, "1,%d,%d", frame_h, frame_w); +- break; +- case VIDEOPROC: +- // the input resoultion required 8-aligned +- frame_h = (frame_h + ALIGNED_SIZE - 1) / ALIGNED_SIZE * ALIGNED_SIZE; +- frame_w = (frame_w + ALIGNED_SIZE - 1) / ALIGNED_SIZE * ALIGNED_SIZE; +- sprintf(shape_string, "1,%d,%d", frame_h, frame_w); +- break; +- case EDSR: +- sprintf(shape_string, "1,%d,%d", frame_h, frame_w); +- break; +- case CUSTVSR: +- sprintf(shape_string, "1,%d,%d", frame_h, frame_w); +- break; +- case TSENET: +- sprintf(shape_string, "1,%d,%d", frame_h, frame_w); +- break; +- default: +- av_log(ctx, AV_LOG_ERROR, "Not supported model type\n"); ++ if (!ivsr_model->dynamic_desc && ++ (ivsr_model->model_type < 0 || ivsr_model->model_type >= MODEL_TYPE_NUM)) { ++ av_log(ctx, AV_LOG_ERROR, ++ "model_type %d out of range and no model_config provided\n", ++ ivsr_model->model_type); + return DNN_GENERIC_ERROR; + } ++ sprintf(shape_string, "1,%d,%d", frame_h, frame_w); + config_reshape = create_and_link_config(config_nireq, RESHAPE_SETTINGS, shape_string, ctx); + if (config_reshape == NULL) + goto err; +@@ -1026,8 +1420,9 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + goto err; + } + ivsr_model->nif = nif; +- //TODO: hard code nif for TSENET +- if(ivsr_model->model_type == TSENET) ivsr_model->nif = 3; ++ /* Override nif from descriptor if set (TSENET=3, RIFE=2, VIDEOSEAL=1, or JSON nif). */ ++ if (get_model_desc(ivsr_model)->nif_override > 0) ++ ivsr_model->nif = get_model_desc(ivsr_model)->nif_override; + + status = + ivsr_get_attr(ivsr_model->handle, INPUT_TENSOR_DESC, &input_tensor_desc_get); +@@ -1288,6 +1683,10 @@ void ff_dnn_free_model_ivsr(DNNModel ** model) + } + av_fifo_freep2(&ivsr_model->frame_queue); + ++ /* Free JSON-loaded model descriptor if present (Patch 0008). */ ++ if (ivsr_model->dynamic_desc) ++ free_dynamic_desc(&ivsr_model->dynamic_desc); ++ + av_freep(&ivsr_model); + *model = NULL; + } +diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h +index 0444526..7732407 100644 +--- a/libavfilter/dnn_interface.h ++++ b/libavfilter/dnn_interface.h +@@ -155,6 +155,7 @@ typedef struct iVSROptions { + float normalize_factor; + char *reshape_values; + int num_streams; ++ char *model_config; /* path to JSON model descriptor file (model_type=-1) */ + + uint32_t frame_input_height; + uint32_t frame_input_width; diff --git a/ivsr_ffmpeg_plugin/patches/0006-Fix-safe-defaults-and-correctness-bugs-in-JSON-dispatch.patch b/ivsr_ffmpeg_plugin/patches/0006-Fix-safe-defaults-and-correctness-bugs-in-JSON-dispatch.patch new file mode 100644 index 0000000..bc54bc2 --- /dev/null +++ b/ivsr_ffmpeg_plugin/patches/0006-Fix-safe-defaults-and-correctness-bugs-in-JSON-dispatch.patch @@ -0,0 +1,386 @@ +From Mon Sep 17 00:00:00 2001 +From: iVSR +Date: Thu, 16 Jul 2026 00:00:00 +0000 +Subject: [PATCH 0006] Fix correctness bugs in JSON dispatch and nireq>1 segfault + +Twelve fixes on top of patch 0005 (six from the original 0006 plus six new): + +Original six correctness fixes: + +1. Fix row-stride bug in unpack_output_window Y-plane path. + src[h * fw + w] used the unpadded frame width as the tensor row stride. + When align > 0 (e.g. VideoProc align=64) the tensor width is padded to a + multiple of align, so the stride must be output->width, not fw. + Fixed inside the shared 8-bit/10-bit paths below. + +2. Fix VideoProc 10-bit Y-plane output written as uint8_t. + The channels==1 branch always wrote uint8_t and clipped to [0,255]. + For yuv420p10le the Y plane is uint16_t with values in [0,1023]. + Writing 8-bit values into a 16-bit stride trashes every other byte. + Fix: detect bits > 8 via AVPixFmtDescriptor, write uint16_t using + linesize[0]>>1 as the row stride, and scale [0,1] -> [0,2^bits-1]. + +3. Fix TV-range (MPEG/limited-range) clamping missing for VideoProc. + The old hardcoded switch(model_type) path clamped the Y plane to + [16,235] (8-bit) or [64,940] (10-bit) when color_range==AVCOL_RANGE_MPEG. + The JSON dispatch path had no equivalent, so limited-range sources + produced luma values outside the legal range. + Both fixes 2 and 3 are gated on name=="VideoProc" (case-insensitive). + +4. Fix normalize_input default: 1 -> 0. + 5 out of 7 shipped models need false (raw [0,255] I/O). The old default + of true silently corrupted output for VideoSeal, EDSR, VideoProc, and + CustVSR when the field was omitted from a JSON config. + +5. Fix normalize_output default: 1 -> 0. + Same rationale. Only RIFE, SPAN, and TSENet need normalize_output: true. + +6. Fix model_color default and reorder color-format dispatch. + When model_color is absent in the JSON and color_format_auto is 0, the + SDK received an empty string causing undefined behaviour. Now falls back + to "RGB" via a ternary in the else branch -- no heap allocation, no leak. + Reorders the if/else chain so color_format_auto is checked first. + +Also adds out_layout inheritance: when out_layout is absent from the JSON +it is defaulted to in_layout. Models whose output layout differs from their +input layout (e.g. TSENet: in=NCHW, out=NHWC) must still declare both. + +Six new fixes for the nireq > 1 segfault: + +7. Fix use-after-free race in ff_dnn_free_model_ivsr (primary segfault fix). + The old non-blocking drain loop exited as soon as the queue appeared + empty. With nireq > 1 a callback thread can be between inference_done++ + and ff_safe_queue_push_back at teardown time: the queue is already empty + (from the main thread's perspective) but the callback hasn't pushed back + yet. ff_safe_queue_destroy then destroys the mutex while the callback is + still holding it, causing a non-deterministic SIGSEGV. + Fix: replace the while(size!=0) loop with a blocking for(nireq) loop so + teardown waits for every callback to complete push_back before destroying + the queue. + +8. Cache input tensor descriptor in IVSRModel at init time. + fill_model_input_ivsr called ivsr_get_attr(INPUT_TENSOR_DESC) on every + frame. With nireq > 1 this raced with concurrent callback threads that + also called ivsr_get_attr(OUTPUT_TENSOR_DESC), potentially corrupting + both callers' local tensor_desc_t buffers if the SDK is not re-entrant. + Fix: query once at model-load time, store in cached_input_tensor_desc. + +9. Cache output tensor descriptor in IVSRModel at init time. + Same rationale as fix 8 for the output descriptor. + Fix: query once at model-load time, store in cached_output_tensor_desc. + +10. Use cached input descriptor in fill_model_input_ivsr. + Remove the per-frame ivsr_get_attr call; use cached_input_tensor_desc. + +11. Use cached descriptors in infer_completion_callback, get_input_ivsr, + and get_output_ivsr. + All three functions queried the SDK on every invocation. Replace with + reads from the cached fields populated by fixes 8 and 9. + +12. Remove redundant ivsr_get_attr call in get_output_ivsr. + Eliminates the last remaining runtime SDK call from the hot path. +--- + libavfilter/dnn/dnn_backend_ivsr.c | 160 lines changed (75 insertions, 85 deletions) + +diff --git a/libavfilter/dnn/dnn_backend_ivsr.c b/libavfilter/dnn/dnn_backend_ivsr.c +index 3eea72d..combined 100644 +@@ -63,6 +63,11 @@ + AVFifo *frame_queue; //input frames queue + int sliding_window_frame_num; /* first-frame dup counter for sliding-window models */ + ModelDesc *dynamic_desc; /* heap-allocated descriptor for JSON-loaded models */ ++ /* Cached tensor descriptors — populated once at init and re-used in every ++ * callback/fill call to avoid concurrent ivsr_get_attr() races under ++ * nireq > 1 (multiple SDK callback threads calling the SDK simultaneously). */ ++ tensor_desc_t cached_input_tensor_desc; ++ tensor_desc_t cached_output_tensor_desc; + } IVSRModel; + + typedef struct IVSRRequestItem { +@@ -547,13 +552,16 @@ + } + } + +-/* Generic float32 NCHW output unpack for JSON-loaded models. +- * Uses md->normalize_output to decide whether to multiply by 255. */ ++/* Generic float32 output unpack for JSON-loaded models. ++ * Uses md->normalize_output to decide whether to multiply by 255. ++ * For "VideoProc": single-channel Y-plane path handles 10-bit uint16_t samples ++ * and applies TV-range clamping ([16,235] / [64,940]) when color_range=MPEG. */ + static void unpack_output_window(IVSRModel *m, TaskItem *task, DNNData *output) + { + const ModelDesc *md = get_model_desc(m); + const float *src = (const float *)output->data; + int fw = task->out_frame->width, fh = task->out_frame->height; ++ int is_videoproc = md->name && !av_strcasecmp(md->name, "VideoProc"); + + /* Single-channel (Y-plane only) model: write Y plane only. + * U/V chroma planes are copied by copy_uv_planes() in vf_dnn_processing.c +@@ -562,12 +570,36 @@ + * inference_done++ can be reordered before in_frame reads under -O3, + * causing a use-after-free race with the main thread freeing in_frame. */ + if (output->channels == 1) { +- for (int h = 0; h < fh; h++) { +- uint8_t *row = task->out_frame->data[0] + h * task->out_frame->linesize[0]; +- for (int w = 0; w < fw; w++) { +- float val = src[h * fw + w]; +- row[w] = (uint8_t)av_clip( +- (int)((md->normalize_output ? val * 255.0f : val) + 0.5f), 0, 255); ++ const AVPixFmtDescriptor *pix_desc = av_pix_fmt_desc_get(task->out_frame->format); ++ int bits = pix_desc->comp[0].depth; ++ int is_tv_range = is_videoproc && ++ task->out_frame->color_range == AVCOL_RANGE_MPEG; ++ if (bits > 8) { ++ /* 10-bit (or 16-bit): samples are uint16_t, scale [0,1] -> [0,2^bits-1] */ ++ float scale = (float)((1 << bits) - 1); ++ int lo = 0, hi = (1 << bits) - 1; ++ if (is_tv_range && bits == 10) { lo = 64; hi = 940; } ++ else if (is_tv_range && bits == 16) { lo = 256; hi = 60160; } ++ ptrdiff_t stride = task->out_frame->linesize[0] >> 1; ++ for (int h = 0; h < fh; h++) { ++ uint16_t *row = (uint16_t *)task->out_frame->data[0] + h * stride; ++ for (int w = 0; w < fw; w++) { ++ float val = src[h * output->width + w]; ++ row[w] = (uint16_t)av_clip( ++ (int)((md->normalize_output ? val * scale : val) + 0.5f), lo, hi); ++ } ++ } ++ } else { ++ /* 8-bit: samples are uint8_t */ ++ int lo = 0, hi = 255; ++ if (is_tv_range) { lo = 16; hi = 235; } ++ for (int h = 0; h < fh; h++) { ++ uint8_t *row = task->out_frame->data[0] + h * task->out_frame->linesize[0]; ++ for (int w = 0; w < fw; w++) { ++ float val = src[h * output->width + w]; ++ row[w] = (uint8_t)av_clip( ++ (int)((md->normalize_output ? val * 255.0f : val) + 0.5f), lo, hi); ++ } + } + } + return; +@@ -657,8 +689,8 @@ + desc->unpack_output = unpack_output_window; + desc->nif_override = 1; + desc->channel_divisor = 1; +- desc->normalize_input = 1; +- desc->normalize_output = 1; ++ desc->normalize_input = 0; /* most models use raw [0,255]; set true only for RIFE/SPAN */ ++ desc->normalize_output = 0; /* most models output raw [0,255]; set true only for RIFE/SPAN/TSENet */ + desc->window_type = WINDOW_SINGLE; + desc->out_order = DCO_RGB; + +@@ -772,6 +804,9 @@ + free_dynamic_desc(&desc); + return AVERROR(EINVAL); + } ++ /* Default out_layout to in_layout when absent (most models share the same layout). */ ++ if (!desc->out_layout) ++ desc->out_layout = av_strdup(desc->in_layout); + + *out_desc = desc; + return 0; +@@ -792,29 +827,17 @@ + IVSRRequestItem * request) + { + DnnContext *ctx = ivsr_model->ctx; +- IVSRStatus status; + DNNData input; + LastLevelTaskItem *lltask; + TaskItem *task; + void *in_data = NULL; + float normalize_factor = ctx->ivsr_option.normalize_factor; + int padding_height = 0, padding_width = 0; +- tensor_desc_t input_tensor_desc_get = { +- .precision = {0}, +- .layout = {0}, +- .tensor_color_format = {0}, +- .model_color_format = {0}, +- .scale = 0.0, +- .dimension = 0, +- .shape = {0}}; +- +- status = ivsr_get_attr(ivsr_model->handle, INPUT_TENSOR_DESC, &input_tensor_desc_get); +- if (status != OK) { +- av_log(ctx, AV_LOG_ERROR, "Failed to get input dimensions\n"); +- return DNN_GENERIC_ERROR; +- } + +- set_dnndata_info(&input, &input_tensor_desc_get); ++ /* Use the cached input tensor descriptor populated at init time. ++ * Calling ivsr_get_attr() here races with concurrent callback threads ++ * that call ivsr_get_attr(OUTPUT_TENSOR_DESC) under nireq > 1. */ ++ set_dnndata_info(&input, &ivsr_model->cached_input_tensor_desc); + /* Apply channel divisor from model descriptor (e.g. RIFE /2, TSENET /3). */ + { + int div = get_model_desc(ivsr_model)->channel_divisor; +@@ -904,7 +927,6 @@ + + static void infer_completion_callback(void *args) + { +- IVSRStatus status; + IVSRRequestItem *request = args; + LastLevelTaskItem *lltask = request->lltasks[0]; + TaskItem *task = lltask->task; +@@ -913,23 +935,12 @@ + DNNData output; + DnnContext *ctx = ivsr_model->ctx; + float normalize_factor = ctx->ivsr_option.normalize_factor; +- tensor_desc_t output_tensor_desc_get = { +- .precision = {0}, +- .layout = {0}, +- .tensor_color_format = {0}, +- .model_color_format = {0}, +- .scale = 0.0, +- .dimension = 0, +- .shape = {0}}; +- +- // ivsr_get_attr can only get precision, layout, dimension and shape info +- status = ivsr_get_attr(ivsr_model->handle, OUTPUT_TENSOR_DESC, &output_tensor_desc_get); +- if (status != OK) { +- av_log(ctx, AV_LOG_ERROR, "Failed to get output dimensions\n"); +- return; +- } + +- set_dnndata_info(&output, &output_tensor_desc_get); ++ /* Use the cached output tensor descriptor populated at init time. ++ * Calling ivsr_get_attr() here from multiple SDK callback threads ++ * simultaneously (nireq > 1) causes a data race that corrupts the ++ * descriptor and leads to a non-deterministic segfault. */ ++ set_dnndata_info(&output, &ivsr_model->cached_output_tensor_desc); + + output.data = request->out_frames; + // Set output mean/scale to meet the logistics in func ff_proc_from_dnn_to_frame() @dnn_io_proc.c +@@ -943,7 +954,7 @@ + const AVPixFmtDescriptor* pix_desc = av_pix_fmt_desc_get(task->out_frame->format); + const AVComponentDescriptor* comp_desc = &pix_desc->comp[0]; + int bits = comp_desc->depth; +- av_assert0(request->lltask_count <= output_tensor_desc_get.shape[0]); ++ av_assert0(request->lltask_count <= ivsr_model->cached_output_tensor_desc.shape[0]); + av_assert0(request->lltask_count >= 1); + for (int i = 0; i < request->lltask_count; ++i) { + task = request->lltasks[i]->task; +@@ -1019,24 +1030,9 @@ + const char *input_name) + { + IVSRModel *ivsr_model = model; +- DnnContext *ctx = ivsr_model->ctx; +- IVSRStatus status; +- tensor_desc_t input_tensor_desc_get = { +- .precision = {0}, +- .layout = {0}, +- .tensor_color_format = {0}, +- .model_color_format = {0}, +- .scale = 0.0, +- .dimension = 0, +- .shape = {0}}; + +- status = ivsr_get_attr(ivsr_model->handle, INPUT_TENSOR_DESC, &input_tensor_desc_get); +- if (status != OK) { +- av_log(ctx, AV_LOG_ERROR, "Failed to get input dimensions\n"); +- return DNN_GENERIC_ERROR; +- } +- +- set_dnndata_info(input, &input_tensor_desc_get); ++ /* Use the cached input tensor descriptor; no SDK call needed. */ ++ set_dnndata_info(input, &ivsr_model->cached_input_tensor_desc); + /* Apply channel divisor from model descriptor (e.g. RIFE /2, TSENET /3). */ + { + int div = get_model_desc(ivsr_model)->channel_divisor; +@@ -1126,27 +1122,11 @@ + const char *output_name, int *output_width, + int *output_height) + { +- int ret = 0; + IVSRModel *ivsr_model = model; +- DnnContext *ctx = ivsr_model->ctx; +- IVSRStatus status; + DNNData output; +- tensor_desc_t output_tensor_desc_get = { +- .precision = {0}, +- .layout = {0}, +- .tensor_color_format = {0}, +- .model_color_format = {0}, +- .scale = 0.0, +- .dimension = 0, +- .shape = {0}}; + +- status = ivsr_get_attr(ivsr_model->handle, OUTPUT_TENSOR_DESC, &output_tensor_desc_get); +- if (status != OK) { +- av_log(ctx, AV_LOG_ERROR, "Failed to get output dimensions\n"); +- return DNN_GENERIC_ERROR; +- } +- +- set_dnndata_info(&output, &output_tensor_desc_get); ++ /* Use the cached output tensor descriptor; no SDK call needed. */ ++ set_dnndata_info(&output, &ivsr_model->cached_output_tensor_desc); + *output_height = output.height; + *output_width = output.width; + +@@ -1155,7 +1135,7 @@ + *output_width = input_width; + } + +- return ret; ++ return 0; + } + + // Utility function to create and link config +@@ -1362,18 +1342,20 @@ + default: + break; + } +- /* Set model colour format from descriptor. Use color_format_auto for NULL entries. */ ++ /* Set model colour format: color_format_auto takes priority; absent model_color defaults to "RGB". */ + { + const ModelDesc *md = get_model_desc(ivsr_model); +- if (md->model_color) { +- strcpy(input_tensor_desc_set.model_color_format, md->model_color); +- } else if (md->color_format_auto == 1) { ++ if (md->color_format_auto == 1) { + /* VideoProc-style: auto-detect from pixel format */ + strcpy(input_tensor_desc_set.model_color_format, + (desc->flags & AV_PIX_FMT_FLAG_RGB) ? "RGB" : "I420_Three_Planes"); + } else if (md->color_format_auto == 2) { + /* CustVSR-style: always YUV */ + strcpy(input_tensor_desc_set.model_color_format, "I420_Three_Planes"); ++ } else { ++ /* Use explicit model_color or fall back to "RGB" for standard RGB models. */ ++ strcpy(input_tensor_desc_set.model_color_format, ++ md->model_color ? md->model_color : "RGB"); + } + } + config_input_tensor = create_and_link_config(config_input_res, INPUT_TENSOR_DESC_SETTING, &input_tensor_desc_set, ctx); +@@ -1449,6 +1431,7 @@ + av_log(ctx, AV_LOG_ERROR, "Failed to get input tensor description\n"); + goto err; + } ++ ivsr_model->cached_input_tensor_desc = input_tensor_desc_get; + + status = + ivsr_get_attr(ivsr_model->handle, OUTPUT_TENSOR_DESC, &output_tensor_desc_get); +@@ -1456,6 +1439,7 @@ + av_log(ctx, AV_LOG_ERROR, "Failed to get output description\n"); + goto err; + } ++ ivsr_model->cached_output_tensor_desc = output_tensor_desc_get; + + ivsr_model->request_queue = ff_safe_queue_create(); + if (!ivsr_model->request_queue) { +@@ -1657,7 +1641,13 @@ + DnnContext *ctx = ivsr_model->ctx; + IVSRStatus status; + +- while (ff_safe_queue_size(ivsr_model->request_queue) != 0) { ++ /* Drain ALL nireq requests using blocking pop so that every in-flight ++ * async callback has completed its ff_safe_queue_push_back() before we ++ * destroy the queue or any shared data. The previous non-blocking ++ * size-check loop only waited for requests already in the queue and ++ * could race with callbacks that hadn't pushed back yet, leading to a ++ * use-after-free when the queue's mutex was subsequently destroyed. */ ++ for (int i = 0; i < ctx->nireq; i++) { + IVSRRequestItem *item = + ff_safe_queue_pop_front(ivsr_model->request_queue); + av_freep(&item->in_frames); + +-- +2.25.1 diff --git a/ivsr_ffmpeg_plugin/patches/0007-Restore-legacy-model_type-backward-compatibility.patch b/ivsr_ffmpeg_plugin/patches/0007-Restore-legacy-model_type-backward-compatibility.patch new file mode 100644 index 0000000..4cd2822 --- /dev/null +++ b/ivsr_ffmpeg_plugin/patches/0007-Restore-legacy-model_type-backward-compatibility.patch @@ -0,0 +1,186 @@ +--- a/libavfilter/dnn/dnn_backend_ivsr.c ++++ b/libavfilter/dnn/dnn_backend_ivsr.c +@@ -43,11 +43,19 @@ + #define DNN_MORE_FRAMES FFERRTAG('M','O','R','E') + + typedef enum { +- BASICVSR = 0, /* Only built-in model: multi-frame out_queue output cannot +- * be expressed in a JSON config. Use model_type=0. */ +- MODEL_TYPE_NUM /* = 1 */ ++ BASICVSR = 0, ++ VIDEOPROC = 1, ++ EDSR = 2, ++ CUSTVSR = 3, ++ TSENET = 4, ++ RIFE = 5, ++ SPAN = 6, ++ VIDEOSEAL = 7, ++ HDRTVNET_LE = 8, ++ MODEL_TYPE_NUM /* = 9 */ + } ModelType; + ++/* Forward declaration for function pointer types and model descriptor */ + typedef struct ModelDesc ModelDesc; /* forward declaration — defined after AVOptions */ + + typedef struct IVSRModel { +@@ -86,8 +93,10 @@ static const AVOption dnn_ivsr_options[] = { + { "extension", "extension lib file full path, usable for BasicVSR model", OFFSET(extension), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS}, + { "op_xml", "custom op xml file full path, usable for BasicVSR model", OFFSET(op_xml), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS}, + { "model_type", +- "0 = BasicVSR built-in (multi-frame out_queue output, no JSON path). " +- "For all other models, leave model_type at its default and set model_config instead.", ++ "0 = Enhanced BasicVSR, 1 = SVP/VideoProc, 2 = Enhanced EDSR, " ++ "3 = Custom VSR, 4 = TSENet, 5 = RIFE, 6 = SPAN, 7 = VideoSeal, 8 = HDRTVNet-LE. " ++ "model_config takes priority if both are set. Legacy option: set model_type for built-in " ++ "descriptor table dispatch; omit model_type and use model_config for JSON-based dispatch.", + OFFSET(model_type), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, MODEL_TYPE_NUM - 1, FLAGS}, + { "normalize_factor", "normalizing factor(constant) for models that not require input normalization to [0, 1]", OFFSET(normalize_factor), AV_OPT_TYPE_FLOAT, { .dbl = 1.0 }, 1.0, 65535.0, FLAGS}, + { "num_streams", "number of execution streams for the throughput mode (now valid only for GPU devices).", OFFSET(num_streams), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 256, FLAGS}, +@@ -179,19 +188,109 @@ static int pack_input_basicvsr (IVSRModel *m, void *base, DNNData *in, TaskI + static void unpack_output_basicvsr(IVSRModel *m, TaskItem *t, DNNData *out); + /* Generic window-based pack/unpack — used by all JSON-config models (model_config set) + * and any future built-in models whose I/O fits the generic path. */ + static int pack_input_window (IVSRModel *m, void *base, DNNData *in, TaskItem *t); + static void unpack_output_window (IVSRModel *m, TaskItem *t, DNNData *out); + +-/* ---- The descriptor table ---- +- * Only BasicVSR lives here because its multi-frame out_queue output requires +- * dedicated C functions that cannot be expressed in a JSON config. +- * All other models: omit model_type and set model_config=.json instead. +- * Canonical configs: model_configs/{videoproc,edsr,custvsr,tsenet,rife,videoseal}_config.json ++/* ---- The descriptor table (backward-compatible) ---- ++ * This table provides built-in descriptors for all legacy models (BASICVSR through HDRTVNET_LE). ++ * Users can: ++ * 1. Legacy path: set model_type=N → uses model_table[N] descriptor ++ * 2. JSON path: set model_config=.json → loads dynamic descriptor (model_type ignored) ++ * 3. Hybrid: set both → model_config takes priority (backward-compatible override) ++ * ++ * The table entries mirror the canonical JSON configs in model_configs/*.json so that ++ * existing production code using model_type=1..8 continues to work without modification. ++ * ++ * Field guide: ++ * name, nif_override, channel_divisor, align, ++ * in_layout, in_precision, out_layout, out_precision, model_color, ++ * out_order, pack_input, unpack_output, ++ * out_precision_depth_derived, output_passthrough_dims, color_format_auto, ++ * sliding_window_init_dup, window_type, normalize_input, normalize_output + * -------------------------------------------------------------------- */ + static const ModelDesc model_table[] = { +- /* [BASICVSR] */ { "BasicVSR", 0, 1, 32, "NFHWC", NULL, "NFHWC", NULL, "RGB", DCO_RGB, +- pack_input_basicvsr, unpack_output_basicvsr }, ++ /* [BASICVSR=0] — requires OpenVINO 2022.3 with custom patches */ ++ { .name = "BasicVSR", .nif_override = 0, .channel_divisor = 1, .align = 32, ++ .in_layout = "NFHWC", .in_precision = NULL, .out_layout = "NFHWC", .out_precision = NULL, ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_basicvsr, .unpack_output = unpack_output_basicvsr, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 0, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_IN_QUEUE, ++ .normalize_input = 0, .normalize_output = 0 }, ++ ++ /* [VIDEOPROC=1] — SVP models, Y-channel SR, auto-detect RGB/I420 from pixel format */ ++ { .name = "VideoProc", .nif_override = 1, .channel_divisor = 1, .align = 64, ++ .in_layout = "NHWC", .in_precision = NULL, .out_layout = "NHWC", .out_precision = "fp32", ++ .model_color = NULL, .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 1, .color_format_auto = 1, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SINGLE, ++ .normalize_input = 0, .normalize_output = 1 }, ++ ++ /* [EDSR=2] — Enhanced EDSR single-frame RGB SR, raw [0,255] float32 I/O */ ++ { .name = "Enhanced EDSR", .nif_override = 1, .channel_divisor = 1, .align = 0, ++ .in_layout = "NCHW", .in_precision = "f32", .out_layout = "NCHW", .out_precision = "fp32", ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 0, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SINGLE, ++ .normalize_input = 0, .normalize_output = 0 }, ++ ++ /* [CUSTVSR=3] — Custom VSR, Y-channel only (yuv420p), color_format_auto=2 forces I420 */ ++ { .name = "Custom VSR", .nif_override = 1, .channel_divisor = 1, .align = 0, ++ .in_layout = "NCHW", .in_precision = NULL, .out_layout = "NCHW", .out_precision = "fp32", ++ .model_color = NULL, .out_order = DCO_NONE, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 2, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SINGLE, ++ .normalize_input = 0, .normalize_output = 0 }, ++ ++ /* [TSENET=4] — 3-frame temporal SR, sliding window with first-frame duplication */ ++ { .name = "TSENet", .nif_override = 3, .channel_divisor = 3, .align = 0, ++ .in_layout = "NCHW", .in_precision = NULL, .out_layout = "NHWC", .out_precision = "fp32", ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 0, ++ .sliding_window_init_dup = 1, .window_type = WINDOW_SLIDING, ++ .normalize_input = 0, .normalize_output = 1 }, ++ ++ /* [RIFE=5] — Frame interpolation, 2-frame sliding window, [0,1] normalized I/O, 128px align */ ++ { .name = "RIFE", .nif_override = 2, .channel_divisor = 2, .align = 128, ++ .in_layout = "NCHW", .in_precision = "f32", .out_layout = "NCHW", .out_precision = "fp32", ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 0, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SLIDING, ++ .normalize_input = 1, .normalize_output = 1 }, ++ ++ /* [SPAN=6] — Single-frame RGB SR, [0,1] normalized I/O, supports x2/x4 variants */ ++ { .name = "SPAN", .nif_override = 1, .channel_divisor = 1, .align = 0, ++ .in_layout = "NCHW", .in_precision = "f32", .out_layout = "NCHW", .out_precision = "fp32", ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 0, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SINGLE, ++ .normalize_input = 1, .normalize_output = 1 }, ++ ++ /* [VIDEOSEAL=7] — Invisible watermarking, raw [0,255] float32, passthrough dims */ ++ { .name = "VideoSeal", .nif_override = 1, .channel_divisor = 1, .align = 0, ++ .in_layout = "NCHW", .in_precision = "f32", .out_layout = "NCHW", .out_precision = "fp32", ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 1, .color_format_auto = 0, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SINGLE, ++ .normalize_input = 0, .normalize_output = 0 }, ++ ++ /* [HDRTVNET_LE=8] — HDR local enhancement (Path A), [0,1] normalized I/O */ ++ { .name = "HDRTVNet-LE", .nif_override = 1, .channel_divisor = 1, .align = 0, ++ .in_layout = "NCHW", .in_precision = "f32", .out_layout = "NCHW", .out_precision = "fp32", ++ .model_color = "RGB", .out_order = DCO_RGB, ++ .pack_input = pack_input_window, .unpack_output = unpack_output_window, ++ .out_precision_depth_derived = 0, .output_passthrough_dims = 0, .color_format_auto = 0, ++ .sliding_window_init_dup = 0, .window_type = WINDOW_SINGLE, ++ .normalize_input = 1, .normalize_output = 1 }, + }; ++ + av_unused static void model_table_size_check(void) { + /* Compile-time assert: table has exactly MODEL_TYPE_NUM entries */ + typedef char model_table_wrong_size[(sizeof(model_table)/sizeof(model_table[0]) == MODEL_TYPE_NUM) ? 1 : -1]; +@@ -1219,10 +1291,12 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + + ivsr_model->model_type = ctx->ivsr_option.model_type; + +- /* When model_config is provided, parse the JSON descriptor and use the JSON +- * dispatch path. model_type is ignored — dynamic_desc takes priority in +- * get_model_desc(). Omitting model_config selects the BasicVSR built-in. */ ++ /* Backward-compatible dispatch: ++ * - If model_config is provided, parse JSON and use dynamic_desc (takes priority). ++ * - Otherwise, use model_table[model_type] for built-in descriptor. ++ * get_model_desc() returns dynamic_desc if non-NULL, else model_table[model_type]. */ + if (ctx->ivsr_option.model_config) { ++ av_log(ctx, AV_LOG_INFO, "Loading JSON model config from: %s\n", ctx->ivsr_option.model_config); + if (parse_model_config_json(ctx, ctx->ivsr_option.model_config, + &ivsr_model->dynamic_desc) < 0) + goto err; +@@ -1378,11 +1452,9 @@ DNNModel *ff_dnn_load_model_ivsr(DnnContext *ctx, + } + char shape_string[40] = {0}; + if (!ivsr_model->dynamic_desc && +- (ivsr_model->model_type < 0 || ivsr_model->model_type >= MODEL_TYPE_NUM)) { +- av_log(ctx, AV_LOG_ERROR, +- "model_type %d out of range and no model_config provided\n", +- ivsr_model->model_type); +- return DNN_GENERIC_ERROR; ++ ivsr_model->model_type >= MODEL_TYPE_NUM) { ++ av_log(ctx, AV_LOG_WARNING, "model_type %d out of range; using default (BASICVSR)\n", ivsr_model->model_type); ++ ivsr_model->model_type = BASICVSR; + } + sprintf(shape_string, "1,%d,%d", frame_h, frame_w); + config_reshape = create_and_link_config(config_nireq, RESHAPE_SETTINGS, shape_string, ctx);