Skip to content
MCG-NJUPublic

About

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

Resources

Stars

152 stars

Watchers

2 watching

Forks

Latest commit

 

History

29 Commits

Folders and files

Repository files navigation

TimeLens2 — Generalist video temporal grounding with multimodal LLMs

TimeLens2 Homepage TimeLens2 Paper on arXiv TimeLens2 Models & Data

Overview · Models and data · Codebase · Reproduction

Find the moments that matter.
TimeLens2 turns natural-language queries into precise, traceable evidence intervals on the video timeline.

TimeLens2 as a generalist video temporal-grounding model across seven benchmarks
One model family across seven temporal-grounding settings, from short indoor actions to long-form and egocentric video.


Overview

TimeLens2 is a generalist video temporal-grounding MLLM. Given a video and a natural-language description or question, it finds when the supporting visual evidence occurs and returns one or more temporal intervals. A single model handles short and long videos, single and repeated events, descriptive and question-form queries, and both third-person and egocentric footage through a unified generative interface.

Scope Training Output
Short and long videos · third-person and egocentric footage Verified SFT data · temporal-grounding GRPO One or more precise temporal intervals

TimeLens2 treats temporal evidence as a set of intervals throughout training. Its supervised stage uses verified single- and multi-span annotations from TimeLens2-93K. Its GRPO stage combines temporal IoU with a matching-free temporal Wasserstein reward. The SFT, GRPO, and evaluation code are released in this repository.

Models and data

TimeLens2 uses two training stages. SFT learns evidence search and variable-cardinality interval generation; GRPO then calibrates the predicted intervals with temporal rewards.

Resource Training stage Description
TimeLens2-2B SFT + GRPO 2B checkpoint · 44.5 average mIoU
TimeLens2-4B SFT + GRPO 4B checkpoint · 47.7 average mIoU
TimeLens2-8B SFT + GRPO 8B checkpoint · 48.0 average mIoU
TimeLens2-2B-SFT SFT 2B checkpoint for rollout generation and GRPO reproduction
TimeLens2-4B-SFT SFT 4B checkpoint for rollout generation and GRPO reproduction
TimeLens2-8B-SFT SFT 8B checkpoint for rollout generation and GRPO reproduction
TimeLens2-93K Dataset 23,793 videos · 93,232 temporal-grounding instances

The repository includes ready-to-use annotations and rollout data for the SFT and GRPO recipes. To reproduce the second stage, download the matching -SFT checkpoint and set it as MODEL_PATH for rollout generation or GRPO training. Video files are distributed separately through the linked Hugging Face dataset and must be downloaded before training. To use another framework for SFT or RL, download the public data and convert it to that framework's required format.

Codebase

The official training and evaluation code is organized around the three stages used in the project:

Stage Purpose Guide Main entry points
sft/ XTuner-based supervised fine-tuning SFT guide train_sft_4b.sh · train_sft_8b.sh
grpo/ Off-policy rollout and GRPO GRPO guide rollout_timelens2.sh · train_grpo_4b.sh · train_grpo_8b.sh
evaluation/ VLMEvalKit with the TimeLens2 grounding entry Evaluation guide run_grounding.sh

Reproduction

  1. Prepare the videos. Download TimeLens2-93K, TimeLens-100K, and Ego4D-NLQ. The SFT and GRPO annotations are already included in this repository.
  2. Run supervised fine-tuning. Launch train_sft_4b.sh or train_sft_8b.sh.
  3. Prepare rollouts. Use the bundled results, or regenerate them with rollout_timelens2.sh. Its single configuration includes both timelens2-93k and timelens-100k.
  4. Run GRPO. Launch train_grpo_4b.sh or train_grpo_8b.sh. Both default to the bundled official rollout files for both sources.
  5. Evaluate. Run run_grounding.sh.

Each module has a focused README with installation, data layout, environment variables, and launch examples. Paths are configurable and may contain environment variables.

License and acknowledgements

The project is released under the Apache License 2.0. The SFT, evaluation, and GRPO modules contain code derived from InternLM/xtuner, open-compass/VLMEvalKit, and TencentARC/TimeLens, respectively. We thank their authors for open-sourcing these projects.

Citation

@article{zhu2026timelens2,
  title={TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs},
  author={Zhu, Yuhan and Ma, Changlian and Zeng, Xiangyu and Li, Xinhao and Zhang, Zhiqiu and Li, Songze and Zhang, Jun and Jiang, Tianxiang and Yang, Yuandong and Yan, Ziang and others},
  journal={arXiv preprint arXiv:2607.17423},
  year={2026}
}

About

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

Resources

Stars

152 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages