Skip to content
KU-AGIPublic

About

[EMNLP2026] Official implementation of "Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding"

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SG-PVR

Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding

arXiv Checkpoint

This repository contains the official implementation of SG-PVR, a reasoning-based reward model for text-to-video generation. Given a text prompt and a generated video, SG-PVR writes an interpretable reasoning trace and scores the video on semantic alignment (SA), visual quality (VQ), temporal quality (TQ), and physical / common-sense consistency (PC). Its core contribution is plan-and-verify reasoning grounded in a spatio-temporal scene graph: instead of judging the video holistically, the model plans the prompt as atomic, verifiable claims and verifies each one against structured visual evidence extracted from the video.

TODO

  • Release the model checkpoint.
  • Release the training code.
  • Release the inference code.
  • Release the evaluation code.
  • Release the data construction code.

Repository Structure

SG-PVR/
├── train.py                     # Training entry point (both stages)
├── inference.py                 # Reward inference on a benchmark
├── configs/
│   ├── stage1_scene_graph.yaml  # Stage 1: scene graph generation
│   ├── stage2_reasoning.yaml    # Stage 2: plan-and-verify reasoning SFT
│   └── deepspeed/               # ZeRO-2 / ZeRO-2 offload / ZeRO-3 offload
├── scripts/
│   └── merge_lora.py            # Merge a LoRA adapter into the base model
└── src/
    ├── prompts.py               # System prompts (main + ablations)
    ├── data/                    # Datasets, tokenization, per-token loss weights, collators
    ├── models/                  # Lightning modules (LoRA on Qwen3.5)
    └── metrics/                 # Scene graph / reward reasoning metrics

1. Setup

conda create -n sgpvr python=3.12 -y
conda activate sgpvr
pip install -r requirements.txt

2. Training

Training has two stages. Both stages fine-tune Qwen3.5-9B with LoRA. Set the /path/to/... entries in the configs before you run the commands below.

2.1 Stage 1: Scene Graph Generation

The annotation file is a parquet file with one row per video:

Column Description
video_id Video file name without extension
source Source subset. SA-V → sav/, PVD → pvd/ under video_root
num_frames, video_fps Used to compute duration (filled from the video file if missing)
objects [{"id": 0, "name": "dog", "attributes": ["brown"]}, ...]
relationships [{"subject": 0, "predicate": "chasing", "object": 1, "time": [[0.0, 3.5]], "type": "motion"}, ...]
python train.py --cfg_path configs/stage1_scene_graph.yaml

python scripts/merge_lora.py \
    --base_model Qwen/Qwen3.5-9B \
    --adapter_path checkpoints/stage1_scene_graph/version_0/final_adapter \
    --output_path checkpoints/stage1_merged

2.2 Stage 2: Plan-and-Verify Reasoning

Set model.name_or_path in configs/stage2_reasoning.yaml to the merged stage-1 model. The training file is a JSON (or JSONL) list:

[
  {
    "index": 0,
    "video_path": "relative/or/absolute/path.mp4",
    "conversations": [
      {"from": "human", "value": "<prompt>A cat playing piano</prompt>\nEvaluate this video following the four-step procedure defined above. Begin your response with <think>."},
      {"from": "gpt", "value": "<think><scene_graph>...</scene_graph><semantic>...</semantic><quality>...</quality></think>\n<answer>SA 4 VQ 4 TQ 5 PC 3</answer>"}
    ]
  }
]

The system prompt is added automatically. data.system_prompt selects it from src/prompts.py. Use final for SG-PVR. The other keys are the ablation variants.

python train.py --cfg_path configs/stage2_reasoning.yaml

3. Inference

CUDA_VISIBLE_DEVICES=0 python inference.py \
    --model_path KU-AGI/SG-PVR \
    --meta_path /path/to/benchmark.json \
    --video_root /path/to/videos \
    --output_path ./results/output.jsonl
Argument Description Default
--model_path Merged HF model (required) -
--processor_path Processor path same as --model_path
--adapter_path Optional LoRA adapter on top of --model_path -
--meta_path Benchmark JSON metadata (required) -
--video_root Root directory of video files (required) -
--output_path Output JSONL path (required) -
--num_samples Number of samples (0 = all) 0
--max_new_tokens Max generation length 8192
--fps Video sampling FPS 1.0
--min_frames / --max_frames Frame count bounds per video 16 / 16
--shortest_edge / --longest_edge Video resize bounds (pixels) 4096 / 4000000

Input. --meta_path is a list of entries. A .mp4 extension is appended when it is missing:

[
  {"video_name": "clip_001", "prompt": "A cat playing piano"},
  {"video_name": "clip_002.mp4", "prompt": "A sunset over the ocean"}
]

Output. The output is a JSONL file with one record per sample. The final scores are in the <answer>SA n VQ n TQ n PC n</answer> line of output_text. src.metrics.reasoning.parse_scores parses that line.

{"index": 0, "sample_id": "...", "prompt": "...", "input_tokens": 1234, "output_tokens": 567, "output_text": "<think>..."}

Citation

@article{kim2026sgpvr,
  title   = {Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding},
  author  = {Kim, Hyomin and Kim, Junghye and Chung, Joanie Hayoun and Oh, Yoonjin and Lee, Kyungjae and Lim, Sungbin and Kim, Sungwoong},
  journal = {arXiv preprint arXiv:2606.11838},
  year    = {2026}
}

About

[EMNLP2026] Official implementation of "Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding"

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages