Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding
This repository contains the official implementation of SG-PVR, a reasoning-based reward model for text-to-video generation. Given a text prompt and a generated video, SG-PVR writes an interpretable reasoning trace and scores the video on semantic alignment (SA), visual quality (VQ), temporal quality (TQ), and physical / common-sense consistency (PC). Its core contribution is plan-and-verify reasoning grounded in a spatio-temporal scene graph: instead of judging the video holistically, the model plans the prompt as atomic, verifiable claims and verifies each one against structured visual evidence extracted from the video.
- Release the model checkpoint.
- Release the training code.
- Release the inference code.
- Release the evaluation code.
- Release the data construction code.
SG-PVR/
├── train.py # Training entry point (both stages)
├── inference.py # Reward inference on a benchmark
├── configs/
│ ├── stage1_scene_graph.yaml # Stage 1: scene graph generation
│ ├── stage2_reasoning.yaml # Stage 2: plan-and-verify reasoning SFT
│ └── deepspeed/ # ZeRO-2 / ZeRO-2 offload / ZeRO-3 offload
├── scripts/
│ └── merge_lora.py # Merge a LoRA adapter into the base model
└── src/
├── prompts.py # System prompts (main + ablations)
├── data/ # Datasets, tokenization, per-token loss weights, collators
├── models/ # Lightning modules (LoRA on Qwen3.5)
└── metrics/ # Scene graph / reward reasoning metrics
conda create -n sgpvr python=3.12 -y
conda activate sgpvr
pip install -r requirements.txtTraining has two stages. Both stages fine-tune Qwen3.5-9B with LoRA.
Set the /path/to/... entries in the configs before you run the commands below.
The annotation file is a parquet file with one row per video:
| Column | Description |
|---|---|
video_id |
Video file name without extension |
source |
Source subset. SA-V → sav/, PVD → pvd/ under video_root |
num_frames, video_fps |
Used to compute duration (filled from the video file if missing) |
objects |
[{"id": 0, "name": "dog", "attributes": ["brown"]}, ...] |
relationships |
[{"subject": 0, "predicate": "chasing", "object": 1, "time": [[0.0, 3.5]], "type": "motion"}, ...] |
python train.py --cfg_path configs/stage1_scene_graph.yaml
python scripts/merge_lora.py \
--base_model Qwen/Qwen3.5-9B \
--adapter_path checkpoints/stage1_scene_graph/version_0/final_adapter \
--output_path checkpoints/stage1_mergedSet model.name_or_path in configs/stage2_reasoning.yaml to the merged stage-1 model. The training file is a JSON (or JSONL) list:
[
{
"index": 0,
"video_path": "relative/or/absolute/path.mp4",
"conversations": [
{"from": "human", "value": "<prompt>A cat playing piano</prompt>\nEvaluate this video following the four-step procedure defined above. Begin your response with <think>."},
{"from": "gpt", "value": "<think><scene_graph>...</scene_graph><semantic>...</semantic><quality>...</quality></think>\n<answer>SA 4 VQ 4 TQ 5 PC 3</answer>"}
]
}
]The system prompt is added automatically. data.system_prompt selects it from src/prompts.py. Use final for SG-PVR. The other keys are the ablation variants.
python train.py --cfg_path configs/stage2_reasoning.yamlCUDA_VISIBLE_DEVICES=0 python inference.py \
--model_path KU-AGI/SG-PVR \
--meta_path /path/to/benchmark.json \
--video_root /path/to/videos \
--output_path ./results/output.jsonl| Argument | Description | Default |
|---|---|---|
--model_path |
Merged HF model (required) | - |
--processor_path |
Processor path | same as --model_path |
--adapter_path |
Optional LoRA adapter on top of --model_path |
- |
--meta_path |
Benchmark JSON metadata (required) | - |
--video_root |
Root directory of video files (required) | - |
--output_path |
Output JSONL path (required) | - |
--num_samples |
Number of samples (0 = all) | 0 |
--max_new_tokens |
Max generation length | 8192 |
--fps |
Video sampling FPS | 1.0 |
--min_frames / --max_frames |
Frame count bounds per video | 16 / 16 |
--shortest_edge / --longest_edge |
Video resize bounds (pixels) | 4096 / 4000000 |
Input. --meta_path is a list of entries. A .mp4 extension is appended when it is missing:
[
{"video_name": "clip_001", "prompt": "A cat playing piano"},
{"video_name": "clip_002.mp4", "prompt": "A sunset over the ocean"}
]Output. The output is a JSONL file with one record per sample. The final scores are in the <answer>SA n VQ n TQ n PC n</answer> line of output_text. src.metrics.reasoning.parse_scores parses that line.
{"index": 0, "sample_id": "...", "prompt": "...", "input_tokens": 1234, "output_tokens": 567, "output_text": "<think>..."}@article{kim2026sgpvr,
title = {Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding},
author = {Kim, Hyomin and Kim, Junghye and Chung, Joanie Hayoun and Oh, Yoonjin and Lee, Kyungjae and Lim, Sungbin and Kim, Sungwoong},
journal = {arXiv preprint arXiv:2606.11838},
year = {2026}
}