System One Model for Video Understanding and Interaction

Overview

VideoJev performs temporal grounding and multiple-choice video QA through direct candidate scoring, without text generation.

Try VideoJev-4B on ModelScope Studio ↗

videojev_promo_v4_music.mp4

Architecture

Results

_{Rankings as of 2026-10-17.}

Temporal grounding

Video question answering

Models

Model	Base model	Weight format	Usage
VideoJev-4B	Qwen3.5-4B	BF16 · Safetensors	Checkpoint & examples ↗ · ModelScope ↗
VideoJev-35B-A3B	Qwen3.6-35B-A3B	BF16 · Safetensors	Checkpoint & examples ↗ · ModelScope ↗

Quick start

Set up the environment, then run the 10-second demo:

import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoProcessor

video = hf_hub_download(
    repo_id="ZhuYuhan/VideoJev-Demo", filename="eating_spaghetti.mp4",
    repo_type="dataset", revision="7e2b9685bca592c3b929268f6fcf82c4ad74b267",
)
repo = "ZhuYuhan/VideoJev-4B"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16,
    device_map={"": "cuda:0"}, attn_implementation="flash_attention_2",
)
model.prepare_for_inference()

questions = [
    {"id": "grounding-1", "task": "grounding",
     "query": "The woman in green eats pasta."},
    {"id": "qa-1", "task": "choice",
     "query": "What is the woman in green eating?",
     "options": ["A. Pasta.", "B. Rice.", "C. Soup."]},
]
results = model.predict_video(
    processor, video, questions,
    fps=2.0, max_frames=2048, context=8192,
)
print(json.dumps(results, indent=2))

Use ZhuYuhan/VideoJev-35B-A3B for the larger model.

Inference output

Recorded 4B output:

[
  {
    "id": "grounding-1",
    "task": "grounding",
    "probabilities": [
      0.005761092994362116,
      0.010865824297070503,
      0.026081746444106102,
      0.9046059250831604,
      0.9879851937294006,
      0.9887627959251404,
      0.9896195530891418,
      0.9849089980125427,
      0.833530068397522,
      0.33037981390953064
    ],
    "interval": [
      3.0,
      10.0
    ]
  },
  {
    "id": "qa-1",
    "task": "choice",
    "probabilities": [
      0.9999161958694458,
      5.755242091254331e-05,
      2.617507561808452e-05
    ],
    "answer_index": 0,
    "answer": "A. Pasta."
  }
]

interval is in seconds; answer_index is zero-based. Probabilities use sigmoid for grounding and softmax for QA.

Grounding defaults to threshold=0.25. Pass threshold=... to predict_video or predict_features to override it.

Environment

Python 3.12, PyTorch 2.8 / CUDA 12.8, Triton 3.4, FlashAttention 2 and causal-conv1d. Install the matching CUDA packages first:

pip install -r requirements.txt
source scripts/env.sh

Training requires XTuner; 35B also requires its compiled grouped-GEMM/token-permutation extension. Set XTUNER_ROOT before sourcing the environment, or install XTuner in it.

Put the base models in models/Qwen3.5-4B and models/Qwen3.6-35B-A3B, or set VIDEOJEV_BASE_4B and VIDEOJEV_BASE_35B.

Data

Prepare your JSONL data; the command generates ready-to-train configs for either model.

Training

Stage	Trainable parameters	Grounding loss	QA loss
Head warmup	Head	BCE	CE
Joint	Language backbone + head	BCE + 2 × duration-weighted SoftIoU	CE

Vision stays frozen. Each stage runs for one epoch with a global batch of 64 packs.

Launch

Use the configs generated by data preparation:

torchrun --standalone --nproc-per-node=8 scripts/train.py \
  --config data/custom4b/configs/head_only.json --out runs/custom4b/head_only

torchrun --standalone --nproc-per-node=8 scripts/train.py \
  --config data/custom4b/configs/joint.json --out runs/custom4b/joint \
  --init-head runs/custom4b/head_only/head.pt

For 35B-A3B, use scripts/train_qwen36.py with its generated configs and scripts/env_qwen36.sh. See Data & training for preparation and GPU settings.

Checkpoints

4B exports joint/hf_checkpoint/ in Hugging Face format, preserving the base model's dtype (BF16). See Checkpoint export for recovery files and standalone 35B conversion.

Evaluation

source scripts/env.sh
python scripts/evaluate_hf.py \
  --checkpoint ZhuYuhan/VideoJev-4B \
  --manifest /path/to/eval.jsonl --out runs/eval4b

The same command supports ZhuYuhan/VideoJev-35B-A3B. QA uses a fixed 256K context. See Evaluation for benchmarks and distributed runs.

Verification

source scripts/env.sh
python -m pytest -q tests

Set VIDEOJEV_TEST_MODEL=/path/to/Qwen3.5-4B for tokenizer checks. Tests skip when required local inputs are absent.

Documentation

Paper · Data · Evaluation · Checkpoint export

Citation

@article{zhu2026videojev,
  title={VideoJev: System One Model for Video Understanding and Interaction},
  author={Zhu, Yuhan and Zeng, Xiangyu and Si, Qingyi and Yang, Chenxu and Li, Yifei and Qin, Luozheng and Zhou, Junhao and Yao, Dingyu and Zhang, Jun and Qin, Chuanyu and others},
  journal={arXiv preprint arXiv:2610.18615},
  year={2026}
}

Acknowledgements

Thanks to the teams behind Qwen for the open model checkpoints, XTuner for the training framework, and TimeLens-100K and TimeLens2-93K for the open training data.

Apache 2.0 · NOTICE