Skip to content

Example CV+VLM pipeline for inclusion in VIPPET. - #978

Draft
tjanczak wants to merge 2 commits into
mainfrom
vippet_vlm_pipeline
Draft

Example CV+VLM pipeline for inclusion in VIPPET.#978
tjanczak wants to merge 2 commits into
mainfrom
vippet_vlm_pipeline

Conversation

@tjanczak

Copy link
Copy Markdown
Contributor

Description

Please include a summary of the changes and the related issue. List any dependencies that are required for this change.

Fixes # (issue)

Any Newly Introduced Dependencies

Please describe any newly introduced 3rd party dependencies in this change. List their name, license information and how they are used in the project.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Checklist:

  • I agree to use the MIT license for my code changes.
  • I have not introduced any 3rd party components incompatible with MIT.
  • I have not included any company confidential information, trade secret, password or security token.
  • I have performed a self-review of my code.

@brmarkus

Copy link
Copy Markdown

Do you want to spell-out VIPPET at least once?

Do you mean visual-pipeline-and-platform-evaluation-tool, referring to

https://github.com/open-edge-platform/edge-ai-libraries/tree/main/tools/visual-pipeline-and-platform-evaluation-tool
?

@brmarkus

Copy link
Copy Markdown

Following the README steps on an Ubuntu 24.04.4 LTS,
with Python v3.12.3,
created a Python virtual environment,
have installed the requirements,
on an Intel NUC, no dGPU, no NVIDIA-GPU/drivers,
using kernel 6.17.0-35-generic an exception occurs during model export:

$ python3 export_models.py‎ 
[YOLO] Downloading and exporting yolo11s to OpenVINO IR (INT8)...
Downloading https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11s.pt to '/localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s.pt': 100% ━━━━━━━━━━━━ 18.4MB 13.9MB/s 1.3s
Ultralytics 8.4.57 🚀 Python-3.12.3 torch-2.13.0+cpu CPU (Intel Core Ultra 9 185H)
WARNING ⚠️ INT8 export requires a missing 'data' arg for calibration. Using default 'data=coco8.yaml'.
YOLO11s summary (fused): 100 layers, 9,443,760 parameters, 0 gradients, 21.5 GFLOPs

PyTorch: starting from '/localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (18.4 MB)
OpenVINO: collecting INT8 calibration images from 'data=coco8.yaml'

WARNING ⚠️ Dataset 'coco8.yaml' images not found, missing path '/localdisk/datasets/coco8/images/val'
Downloading https://ultralytics.com/assets/coco8.zip to '/localdisk/datasets/coco8.zip': 100% ━━━━━━━━━━━━ 432.8KB 6.1MB/s 0.1s
Unzipping /localdisk/datasets/coco8.zip to /localdisk/datasets/coco8...: 100% ━━━━━━━━━━━━ 25/25 6.7Kfiles/s 0.0s
Dataset download success ✅ (0.4s), saved to /localdisk/datasets

val: Fast image access ✅ (ping: 0.0±0.0 ms, read: 7384.4±4229.4 MB/s, size: 54.0 KB)
val: Scanning /localdisk/datasets/coco8/labels/val... 4 images, 0 backgrounds, 0 corrupt: 100% ━━━━━━━━━━━━ 4/4 932.3it/s 0.0s
val: New cache created: /localdisk/datasets/coco8/labels/val.cache
WARNING ⚠️ OpenVINO: >300 images recommended for INT8 calibration, found 4 images.

OpenVINO: starting export with openvino 2026.2.0-21903-52ddc073857-releases/2026/2...
INFO:nncf:17 ignored nodes were found by patterns in the NNCFGraph
INFO:nncf:1 ignored nodes were found by types in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 199 __module.model.23/aten::add/Add_6
INFO:nncf:Not adding activation input quantizer for operation: 200 __module.model.23/aten::sub/Subtract
INFO:nncf:Not adding activation input quantizer for operation: 214 __module.model.23/aten::add/Add_7
230 __module.model.23/aten::div/Divide

INFO:nncf:Not adding activation input quantizer for operation: 215 __module.model.23/aten::sub/Subtract_1
INFO:nncf:Not adding activation input quantizer for operation: 156 __module.model.23/aten::mul/Multiply_3
INFO:nncf:Not adding activation input quantizer for operation: 271 __module.model.23/aten::sigmoid/Sigmoid
Statistics collection ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 4/4 • 0:00:00 • 0:00:00
WARNING:nncf:Dataset contains only 4 samples, smaller than the requested subset size 300.
Applying Fast Bias correction ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 80/80 • 0:00:01 • 0:00:00
OpenVINO: export success ✅ 6.2s, saved as '/localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s_int8_openvino_model/' (9.8 MB)

Export complete (6.6s)
Results saved to /localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s_int8_openvino_model
Predict:         yolo predict task=detect model=/localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s_int8_openvino_model/ imgsz=640 int8
Validate:        yolo val task=detect model=/localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s_int8_openvino_model/ imgsz=640 data=/usr/src/ultralytics/ultralytics/cfg/datasets/coco.yaml int8 
Visualize:       https://netron.app
[YOLO] Model ready: /localdisk/OpenVINO/dlstreamer_vlm/models/yolo11s_openvino/yolo11s.xml
[VLM] Exporting OpenGVLab/InternVL3_5-2B via optimum-cli (int4)...
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.48k/2.48k [00:00<00:00, 6.94MB/s]
configuration_internvl_chat.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.70k/4.70k [00:00<00:00, 11.2MB/s]
configuration_intern_vit.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.55k/5.55k [00:00<00:00, 13.9MB/s]
`torch_dtype` is deprecated! Use `dtype` instead!
modeling_internvl_chat.py: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16.5k/16.5k [00:00<00:00, 31.4MB/s]
conversation.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15.3k/15.3k [00:00<00:00, 31.2MB/s]
modeling_intern_vit.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18.2k/18.2k [00:00<00:00, 32.4MB/s]
FlashAttention2 is not installed.
model.safetensors: downloading bytes: ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.08GB, 15.8MB/s  
model.safetensors: reconstructing file: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.70GB / 4.70GB,  131MB/s  
Traceback (most recent call last):
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/commands/optimum_cli.py", line 219, in main
    service.run()
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/commands/export/openvino.py", line 468, in run
    main_export(
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/exporters/openvino/__main__.py", line 530, in main_export
    model = TasksManager.get_model_from_task(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/exporters/tasks.py", line 1217, in get_model_from_task
    model = model_class.from_pretrained(model_name_or_path, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 365, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4072, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/huggingface/modules/transformers_modules/OpenGVLab/InternVL3_5_hyphen_2B/7d7bd7bcc35b77b391883ebd7f9686cfa0e25cc0/modeling_internvl_chat.py", line 70, in __init__
    self.vision_model = InternVisionModel(config.vision_config)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/huggingface/modules/transformers_modules/OpenGVLab/InternVL3_5_hyphen_2B/7d7bd7bcc35b77b391883ebd7f9686cfa0e25cc0/modeling_intern_vit.py", line 378, in __init__
    self.encoder = InternVisionEncoder(config)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/huggingface/modules/transformers_modules/OpenGVLab/InternVL3_5_hyphen_2B/7d7bd7bcc35b77b391883ebd7f9686cfa0e25cc0/modeling_intern_vit.py", line 312, in __init__
    dpr = [x.item() for x in torch.linspace(0, config.drop_path_rate, config.num_hidden_layers)]
           ^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/torch/utils/_device.py", line 122, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/torch/utils/_device.py", line 122, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/torch/_meta_registrations.py", line 8550, in meta_local_scalar_dense
    raise RuntimeError("Tensor.item() cannot be called on meta tensors")
RuntimeError: Tensor.item() cannot be called on meta tensors
Traceback (most recent call last):
  File "/localdisk/OpenVINO/dlstreamer_vlm/export_models.py‎", line 95, in <module>
    main()
  File "/localdisk/OpenVINO/dlstreamer_vlm/export_models.py‎", line 90, in main
    export_vlm()
  File "/localdisk/OpenVINO/dlstreamer_vlm/export_models.py‎", line 72, in export_vlm
    run(
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/bin/optimum-cli', 'export', 'openvino', '--model', 'OpenGVLab/InternVL3_5-2B', '--task', 'image-text-to-text', '--trust-remote-code', '--weight-format', 'int4', '/localdisk/OpenVINO/dlstreamer_vlm/models/InternVL3_5-2B']' returned non-zero exit status 1.

The message FlashAttention2 is not installed. seems related to NVIDIA/CUDA, and might be a warning only.

This message seems causing it: RuntimeError: Tensor.item() cannot be called on meta tensors.

The following packages got installed:

$ pip install -r export_requirements.txt‎ 
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Requirement already satisfied: ultralytics==8.4.57 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 8)) (8.4.57)
Requirement already satisfied: nncf==3.0.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 11)) (3.0.0)
Requirement already satisfied: openvino==2026.2.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 14)) (2026.2.0)
Requirement already satisfied: huggingface_hub in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 18)) (1.23.0)
Requirement already satisfied: einops in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 21)) (0.8.2)
Requirement already satisfied: timm in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 22)) (1.0.28)
Requirement already satisfied: optimum[openvino] in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from -r export_requirements.txt‎ (line 17)) (2.2.0)
Requirement already satisfied: numpy>=1.23.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (2.2.6)
Requirement already satisfied: matplotlib>=3.3.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (3.11.0)
Requirement already satisfied: opencv-python>=4.6.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (5.0.0.93)
Requirement already satisfied: pillow>=7.1.2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (12.3.0)
Requirement already satisfied: pyyaml>=5.3.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (6.0.3)
Requirement already satisfied: requests>=2.23.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (2.34.2)
Requirement already satisfied: scipy>=1.4.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.18.0)
Requirement already satisfied: torch>=1.8.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (2.13.0+cpu)
Requirement already satisfied: torchvision>=0.9.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (0.28.0+cpu)
Requirement already satisfied: psutil>=5.8.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (7.2.2)
Requirement already satisfied: polars>=0.20.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.42.1)
Requirement already satisfied: ultralytics-thop>=2.0.18 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (2.0.20)
Requirement already satisfied: networkx<3.5.0,>=2.6 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (3.4.2)
Requirement already satisfied: ninja<1.14,>=1.10.0.post2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (1.13.0)
Requirement already satisfied: openvino-telemetry>=2023.2.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (2025.2.0)
Requirement already satisfied: packaging>=20.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (26.2)
Requirement already satisfied: pandas<2.4,>=1.1.5 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (2.3.3)
Requirement already satisfied: pydot<=3.0.4,>=1.4.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (3.0.4)
Requirement already satisfied: rich>=13.5.2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (15.0.0)
Requirement already satisfied: safetensors>=0.4.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (0.7.0)
Requirement already satisfied: scikit-learn>=0.24.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (1.9.0)
Requirement already satisfied: tabulate>=0.9.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (0.10.0)
Requirement already satisfied: transformers>=4.29 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from optimum[openvino]->-r export_requirements.txt‎ (line 17)) (5.0.0)
Requirement already satisfied: optimum-intel>=1.23.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from optimum-intel[openvino]>=1.23.0; extra == "openvino"->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (2.0.0)
Requirement already satisfied: click<9.0.0,>=8.4.2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (8.4.2)
Requirement already satisfied: filelock>=3.10.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (3.29.7)
Requirement already satisfied: fsspec>=2023.5.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (2026.6.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.5.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (1.5.1)
Requirement already satisfied: httpx<1,>=0.23.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (0.28.1)
Requirement already satisfied: tqdm>=4.42.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (4.68.4)
Requirement already satisfied: typing-extensions>=4.1.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from huggingface_hub->-r export_requirements.txt‎ (line 18)) (4.16.0)
Requirement already satisfied: anyio in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub->-r export_requirements.txt‎ (line 18)) (4.14.2)
Requirement already satisfied: certifi in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub->-r export_requirements.txt‎ (line 18)) (2026.6.17)
Requirement already satisfied: httpcore==1.* in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub->-r export_requirements.txt‎ (line 18)) (1.0.9)
Requirement already satisfied: idna in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from httpx<1,>=0.23.0->huggingface_hub->-r export_requirements.txt‎ (line 18)) (3.18)
Requirement already satisfied: h11>=0.16 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub->-r export_requirements.txt‎ (line 18)) (0.16.0)
Requirement already satisfied: contourpy>=1.0.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.3.3)
Requirement already satisfied: cycler>=0.10 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (4.63.0)
Requirement already satisfied: kiwisolver>=1.3.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.5.0)
Requirement already satisfied: pyparsing>=3 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (2.9.0.post0)
Requirement already satisfied: setuptools in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from optimum-intel>=1.23.0->optimum-intel[openvino]>=1.23.0; extra == "openvino"->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (83.0.0)
Requirement already satisfied: openvino-tokenizers>=2026.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from optimum-intel>=1.23.0->optimum-intel[openvino]>=1.23.0; extra == "openvino"->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (2026.2.0.0)
Requirement already satisfied: pytz>=2020.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from pandas<2.4,>=1.1.5->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (2026.2)
Requirement already satisfied: tzdata>=2022.7 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from pandas<2.4,>=1.1.5->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (2026.3)
Requirement already satisfied: polars-runtime-32==1.42.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from polars>=0.20.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.42.1)
Requirement already satisfied: charset_normalizer<4,>=2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from requests>=2.23.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (3.4.9)
Requirement already satisfied: urllib3<3,>=1.26 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from requests>=2.23.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (2.7.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from rich>=13.5.2->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (4.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from rich>=13.5.2->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (2.20.0)
Requirement already satisfied: joblib>=1.4.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from scikit-learn>=0.24.0->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (1.5.3)
Requirement already satisfied: narwhals>=2.0.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from scikit-learn>=0.24.0->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (2.24.0)
Requirement already satisfied: threadpoolctl>=3.5.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from scikit-learn>=0.24.0->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (3.6.0)
Requirement already satisfied: sympy>=1.13.3 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from torch>=1.8.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.14.0)
Requirement already satisfied: jinja2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from torch>=1.8.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (3.1.6)
Requirement already satisfied: regex!=2019.12.17 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from transformers>=4.29->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (2026.7.10)
Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from transformers>=4.29->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (0.22.2)
Requirement already satisfied: typer-slim in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from transformers>=4.29->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (0.24.0)
Requirement already satisfied: mdurl~=0.1 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=13.5.2->nncf==3.0.0->-r export_requirements.txt‎ (line 11)) (0.1.2)
Requirement already satisfied: six>=1.5 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from python-dateutil>=2.7->matplotlib>=3.3.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.17.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=1.8.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from jinja2->torch>=1.8.0->ultralytics==8.4.57->-r export_requirements.txt‎ (line 8)) (3.0.3)
Requirement already satisfied: typer>=0.24.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from typer-slim->transformers>=4.29->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (0.26.8)
Requirement already satisfied: shellingham>=1.3.0 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from typer>=0.24.0->typer-slim->transformers>=4.29->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (1.5.4)
Requirement already satisfied: annotated-doc>=0.0.2 in ./.cv_triggered_vlm-venv/lib/python3.12/site-packages (from typer>=0.24.0->typer-slim->transformers>=4.29->optimum[openvino]->-r export_requirements.txt‎ (line 17)) (0.0.4)

Do you see something obvious?

@brmarkus

Copy link
Copy Markdown

The script export_models.py is spawning

/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/bin/optimum-cli  export openvino --model OpenGVLab/InternVL3_5-2B --task image-text-to-text --trust-remote-code --weight-format int4 /localdisk/OpenVINO/dlstreamer_vlm
/models/InternVL3_5-2B

Which then shows the error:

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
`torch_dtype` is deprecated! Use `dtype` instead!
FlashAttention2 is not installed.
Traceback (most recent call last):
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/commands/optimum_cli.py", line 219, in main
    service.run()
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/commands/export/openvino.py", line 468, in run
    main_export(
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/exporters/openvino/__main__.py", line 530, in main_export
    model = TasksManager.get_model_from_task(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/optimum/exporters/tasks.py", line 1217, in get_model_from_task
    model = model_class.from_pretrained(model_name_or_path, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 365, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4072, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efuser/.cache/huggingface/modules/transformers_modules/OpenGVLab/InternVL3_5_hyphen_2B/7d7bd7bcc35b77b391883ebd7f9686cfa0e25cc0/modeling_internvl_chat.py", line 70, in __init__
    self.vision_model = InternVisionModel(config.vision_config)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efuser/.cache/huggingface/modules/transformers_modules/OpenGVLab/InternVL3_5_hyphen_2B/7d7bd7bcc35b77b391883ebd7f9686cfa0e25cc0/modeling_intern_vit.py", line 378, in __init__
    self.encoder = InternVisionEncoder(config)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/efuser/.cache/huggingface/modules/transformers_modules/OpenGVLab/InternVL3_5_hyphen_2B/7d7bd7bcc35b77b391883ebd7f9686cfa0e25cc0/modeling_intern_vit.py", line 312, in __init__
    dpr = [x.item() for x in torch.linspace(0, config.drop_path_rate, config.num_hidden_layers)]
           ^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/torch/utils/_device.py", line 122, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/torch/utils/_device.py", line 122, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/localdisk/OpenVINO/dlstreamer_vlm/.cv_triggered_vlm-venv/lib/python3.12/site-packages/torch/_meta_registrations.py", line 8550, in meta_local_scalar_dense
    raise RuntimeError("Tensor.item() cannot be called on meta tensors")
RuntimeError: Tensor.item() cannot be called on meta tensors

@brmarkus

Copy link
Copy Markdown

@tjanczak do you see differences in your local package versions compared to mine listed earlier?

@tjanczak

Copy link
Copy Markdown
Contributor Author

@brmarkus - I believe transformers library has to be pinned to an older version for InternVL3_5 - I'll check it and update export_requiremnts.txt as necessary:
transformers<5.0.0

Please note this PR is in 'draft' stage - shared with VIPPET for inclusion in this tool - it is not ready for productization yet

@brmarkus

brmarkus commented Jul 15, 2026

Copy link
Copy Markdown

@brmarkus - I believe transformers library has to be pinned to an older version for InternVL3_5 - I'll check it and update export_requiremnts.txt as necessary: transformers<5.0.0

Please note this PR is in 'draft' stage - shared with VIPPET for inclusion in this tool - it is not ready for productization yet

Will have a try as well and will let you know the results.

Another interesting thing:

When copy&pasting commands from the README like pip install -r export_requirements.txt and python3 export_models.py and bash cv_triggered_vlm.sh videos/pedestrians.mp4 then I get terminal errors like ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'export_requirements.txt'. I haven't checked with a HEX editor, but there might be some obscure Unicode characters contained, not visually obvious?
When I paste the command, delete the filename with DELETE-key and use tab-completion then the file can be found.

@brmarkus

Copy link
Copy Markdown

Yesss, I can confirm the models get downloaded, quantized/compressed successfully with "pinning" the "transformers" package:

# Model export dependencies for cv_triggered_vlm
# Run once before starting the pipeline: pip install -r export_requirements.txt

# IMPORTANT: CPU-only PyTorch — must appear before any torch-dependent package.
--extra-index-url https://download.pytorch.org/whl/cpu

transformers<5.0.0

# Ultralytics YOLO (for yolov11s detection model export)
ultralytics==8.4.57

# NNCF for INT8 quantization (required by ultralytics int8=True export)
nncf==3.0.0

# OpenVINO Python (pin to match DL Streamer runtime)
openvino==2026.2.0

# HuggingFace optimum for VLM model export (InternVL3_5-2B)
optimum[openvino]
huggingface_hub

# Required by InternVL3 model architecture
einops
timm

Uninstalling v5.0.0 and installing 4.57.6:

Downloading transformers-4.57.6-py3-none-any.whl (12.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 15.1 MB/s eta 0:00:00
Downloading huggingface_hub-0.36.2-py3-none-any.whl (566 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 566.4/566.4 kB 13.0 MB/s eta 0:00:00
Installing collected packages: huggingface_hub, transformers
  Attempting uninstall: huggingface_hub
    Found existing installation: huggingface_hub 1.23.0
    Uninstalling huggingface_hub-1.23.0:
      Successfully uninstalled huggingface_hub-1.23.0
  Attempting uninstall: transformers
    Found existing installation: transformers 5.0.0
    Uninstalling transformers-5.0.0:
      Successfully uninstalled transformers-5.0.0
Successfully installed huggingface_hub-0.36.2 transformers-4.57.6

The script export_models.py succeeds with === All models ready === now.
Let me continue with the remaining steps.

@brmarkus

brmarkus commented Jul 15, 2026

Copy link
Copy Markdown

Running the pipeline with the recommended Docker way fails with

WARNING: erroneous pipeline: no element "gvaproximitytrigger_py"

It looks like the script cv_triggered_vlm.sh‎ extends the ENV variable GST_PLUGIN_PATH instead of the ENV variable GVA_ELEMENTS_PATH.

By using the ENV variable GVA_ELEMENTS_PATH instead the custom element could be found - but failed to write results/output.mp4, for the JSON-file and for the JPEG files; therefore using -v "$(pwd)":/app:rw in the docker run command line)

Finally seeing the command line working:

GST_PLUGIN_PATH=${GST_PLUGIN_PATH}:/app/plugins GVA_ELEMENTS_PATH=/app/plugins GST_REGISTRY_FORK=no gst-launch-1.0 filesrc location=videos/pedestrians.mp4 ! decodebin3 caps=video/x-raw\(ANY\) ! gvadetect model=models/yolo11s_openvino/yolo11s.xml device=NPU ! queue ! tee name=t t. 
! queue flush-on-eos=true ! gvawatermark ! gvafpscounter !     vah264enc ! h264parse ! mp4mux fragment-duration=1000 !     filesink location=results/output.mp4 t. ! queue leaky=downstream max-size-buffers=1 flush-on-eos=true !     gvaproximitytrigger_py class-a=person class-b=bicycle distance=150 frames=10 !  
   videoconvertscale ! video/x-raw,width=796,height=448 !     gvagenai model-path=models/InternVL3_5-2B device=GPU         prompt="Are people riding on bicycles? Answer yes or no."         generation-config="max_new_tokens=10"         chunk-size=1 metrics=true !     gvametapublish file-format=json-lines file-p
ath=results/vlm_output.jsonl !     jpegenc ! multifilesink location=results/vlm_frame_%05d.jpeg async=false
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'vah264enc0': gst.va.display.handle=context, gst-display=(GstObject)"\(GstVaDisplayDrm\)\ vadisplaydrm2", description=(string)"Intel\(R\)\ Gen\ Graphics", path=(string)/dev/dri/renderD128;
Redistribute latency...
Redistribute latency...
Redistribute latency...
WARNING: from element /GstPipeline:pipeline0/GstGvaDetect:gvadetect0: 

System memory is being used for inference on device 'NPU'. For optimal performance, use VA memory in the pipeline:

vapostproc ! "video/x-raw(memory:VAMemory)" ! gvadetect device=NPU model=models/yolo11s_openvino/yolo11s.xml.

Additional debug info:
/home/dlstreamer/dlstreamer/src/monolithic/gst/inference_elements/base/gva_base_inference.cpp(1183): gva_base_inference_set_caps (): /GstPipeline:pipeline0/GstGvaDetect:gvadetect0:
System memory transfers are less efficient than VA memory for device 'NPU'. Consider using memory:VAMemory for better performance. 

Redistribute latency...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
FpsCounter(last 1.01sec): total=42.73 fps, number-streams=1, per-stream=42.73 fps
FpsCounter(average 1.01sec): total=42.73 fps, number-streams=1, per-stream=42.73 fps
FpsCounter(last 1.00sec): total=40.90 fps, number-streams=1, per-stream=40.90 fps
FpsCounter(average 2.01sec): total=41.82 fps, number-streams=1, per-stream=41.82 fps
FpsCounter(last 1.03sec): total=42.72 fps, number-streams=1, per-stream=42.72 fps
FpsCounter(average 3.04sec): total=42.12 fps, number-streams=1, per-stream=42.12 fps
FpsCounter(last 1.01sec): total=41.66 fps, number-streams=1, per-stream=41.66 fps
FpsCounter(average 4.05sec): total=42.01 fps, number-streams=1, per-stream=42.01 fps
Redistribute latency...68.0 %)
FpsCounter(last 1.01sec): total=41.46 fps, number-streams=1, per-stream=41.46 fps
FpsCounter(average 5.06sec): total=41.90 fps, number-streams=1, per-stream=41.90 fps
FpsCounter(last 1.01sec): total=41.50 fps, number-streams=1, per-stream=41.50 fps
FpsCounter(average 6.07sec): total=41.83 fps, number-streams=1, per-stream=41.83 fps
FpsCounter(last 1.01sec): total=41.78 fps, number-streams=1, per-stream=41.78 fps
FpsCounter(average 7.08sec): total=41.82 fps, number-streams=1, per-stream=41.82 fps
FpsCounter(last 1.01sec): total=41.69 fps, number-streams=1, per-stream=41.69 fps
FpsCounter(average 8.08sec): total=41.81 fps, number-streams=1, per-stream=41.81 fps
FpsCounter(last 1.01sec): total=41.78 fps, number-streams=1, per-stream=41.78 fps
FpsCounter(average 9.09sec): total=41.81 fps, number-streams=1, per-stream=41.81 fps
FpsCounter(last 1.02sec): total=42.14 fps, number-streams=1, per-stream=42.14 fps
FpsCounter(average 10.11sec): total=41.84 fps, number-streams=1, per-stream=41.84 fps
[proximity] Trigger fired at 7.09s - person near bicycle for 10 frames
FpsCounter(last 1.03sec): total=38.84 fps, number-streams=1, per-stream=38.84 fps
FpsCounter(average 11.14sec): total=41.56 fps, number-streams=1, per-stream=41.56 fps
FpsCounter(last 1.01sec): total=37.79 fps, number-streams=1, per-stream=37.79 fps
FpsCounter(average 12.15sec): total=41.25 fps, number-streams=1, per-stream=41.25 fps
Redistribute latency...70.3 %)
FpsCounter(last 1.01sec): total=35.81 fps, number-streams=1, per-stream=35.81 fps
FpsCounter(average 13.15sec): total=40.83 fps, number-streams=1, per-stream=40.83 fps
[proximity] Trigger fired at 9.33s - person near bicycle for 10 frames
FpsCounter(last 1.00sec): total=37.00 fps, number-streams=1, per-stream=37.00 fps
FpsCounter(average 14.15sec): total=40.56 fps, number-streams=1, per-stream=40.56 fps
FpsCounter(last 1.01sec): total=37.58 fps, number-streams=1, per-stream=37.58 fps
FpsCounter(average 15.16sec): total=40.36 fps, number-streams=1, per-stream=40.36 fps
FpsCounter(last 1.04sec): total=41.48 fps, number-streams=1, per-stream=41.48 fps
FpsCounter(average 16.20sec): total=40.44 fps, number-streams=1, per-stream=40.44 fps
FpsCounter(last 1.02sec): total=42.36 fps, number-streams=1, per-stream=42.36 fps
FpsCounter(average 17.21sec): total=40.55 fps, number-streams=1, per-stream=40.55 fps
FpsCounter(last 1.02sec): total=46.16 fps, number-streams=1, per-stream=46.16 fps
FpsCounter(average 18.23sec): total=40.86 fps, number-streams=1, per-stream=40.86 fps
FpsCounter(last 1.01sec): total=47.40 fps, number-streams=1, per-stream=47.40 fps
FpsCounter(average 19.24sec): total=41.21 fps, number-streams=1, per-stream=41.21 fps
FpsCounter(last 0.59sec): total=45.48 fps, number-streams=1, per-stream=45.48 fps
FpsCounter(overall 19.84sec): total=41.33 fps, number-streams=1, per-stream=41.33 fps
Got EOS from element "pipeline0".
Execution ended after 0:00:19.777961496
Setting pipeline to NULL ...
Freeing pipeline ...

Triggering two events:
image

and
image

In the video with bounding boxes:
image

and
image

@tjanczak

Copy link
Copy Markdown
Contributor Author

Thanks!
Model export requirements file has been updated with pinned dependencies.

We are not using GVA_ELEMENTS_PATH for custom elements; it should use regular GStreamer mechanisms (GST_PYTHON_PATH). Is it possible you have cleared local element cache in between two runs?

rm -rf ~/.cache/gstreamer-1.0/registry.*.bin

@brmarkus

Copy link
Copy Markdown

Thanks! Model export requirements file has been updated with pinned dependencies.

We are not using GVA_ELEMENTS_PATH for custom elements; it should use regular GStreamer mechanisms (GST_PYTHON_PATH). Is it possible you have cleared local element cache in between two runs?

rm -rf ~/.cache/gstreamer-1.0/registry.*.bin

No, haven't cleared the registry cache - but experimented elsewhise alot, started the Docker container image in interactive-terminal mode.
I could try again on another day using a clean environment again.

Do you see the same effect that copy&pasting commands from the README (e.g. from Github in the browser or the README opened in e.g. VisualStudioCode) into a terminal that files could not be found (maybe using different underscore characters or different spaces)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants