Yiran Wang1,*,β , Xingyilang Yin1,2,6,*, Junfu Pu1,*, Guangzhi Wang1,*, Kaifeng Li2,
Mingyu Ouyang1,3, Huiqiang Sun1,4,
Lingen Li1,5, Cheng Cheng1, Wangbo Yu1,
Honghao Chen1, Xiaodong Cun1,2,β, Chi-Man Pun6, Zhiguo Cao4, Ying Shan1
1ARC Lab, Tencent IEG Β 2GVC Lab, Great Bay University Β 3NUS Β 4HUST Β 5MMLab, CUHK Β 6University of Macau
β Project Lead Β Β * Equal Contribution Β Β β Corresponding Author
- [2026.11.11] ποΈ GameHorizon-Annotator is released β the automated pipeline that builds our multi-horizon instruction pyramid! Along with it, we release GameHorizon-Annotator-Demo and the keybind maps for all 21 titles so you can run the whole pipeline right away.
- [2026.10.26] β³ Due to our company's internal approval process, the release will be delayed by one week. We will start releasing next week, beginning with the Annotator. Sorry for the delay and thanks for your patience! π
- [2026.09.22] π The paper and project page of GameHorizon Suite are released!
- We will progressively release the code, benchmark, and data starting around 2026.10.25. Stay tuned! β
We are actively preparing the release of our code, benchmark, and data. Star the repo to follow our progress.
- π Project page
- π Technical report / paper
- π Public leaderboard
- ποΈ GameHorizon-Annotator β annotation toolkit & pipeline
- π GameHorizon-Bench (offline) β MCQ suite & evaluation code
- π GameHorizon-Bench (online) β online task environments
- π― GameHorizon-Data β 5,000-hour AAA gameplay corpus
We introduce GameHorizon, a large-scale data and evaluation suite spanning multiple horizons and AAA games. It serves as a unified yardstick across a broad range of model types. It consists of three key components. GameHorizon-Annotator automatically produces a three-level pyramid of short-horizon operations, medium-horizon goals, and long-horizon strategies. GameHorizon-Data contains 5,000 hours of gameplay across 21 game titles, with temporally aligned videos, actions, and multi-horizon instructions. GameHorizon-Bench provides reproducible offline and stepwise online testing. The offline track contains thousands of standardized MCQs across three primary tasks and diagnostic variants. The online track tests order-dependent causal and order-flexible thematic tasks via the verifiable subtasks for failure localization.
| Component | What it is | Highlights |
|---|---|---|
| ποΈ Annotator | A scalable and automated annotation pipeline | L1 β L2 β L3 instruction pyramid |
| π― Data | A large-scale AAA gameplay corpus | 5,000 hours Β· AAA-focused Β· 21 titles Β· temporally-aligned videos, actions & multi-horizon instructions |
| π Bench | Reproducible offline + stepwise online track | 5,000 offline MCQs (3 primary + 10 variant tasks) Β· 20 online tasks / 62 subtasks Β· 47 models benchmarked |
GameHorizon-Annotator turns gameplay video plus temporally aligned keyboard-and-mouse actions into the three-level instruction pyramid β short-horizon operations β medium-horizon goals β long-horizon strategies.
cd GameHorizon-Annotator
uv sync # or: pip install -e .
export OPENAI_API_KEY="your-key"
export OPENAI_BASE_URL="https://your-provider/v1"
python -m annotator.cli.run_full_pipeline \
--game minecraft \
--video ./data/gameplay.mp4 \
--annotations ./data/annotations.json \
--workdir ./outputs/minecraft-sceneLevel-1/2/3 instructions land in ./outputs/minecraft-scene/level_{1,2,3}_instructions.json.
To try it without preparing your own recording, we release six demo clips β Apex Legends, Cyberpunk 2077, Elden Ring, Elden Ring Nightreign, GTA V and Minecraft β each with its video, aligned keyboard-mouse actions and annotated instructions, alongside the keybind maps for all 21 titles: π€ GameHorizon-Data
π Input format, pipeline stages and all options β GameHorizon-Annotator/README.md
Accuracies are reported as percentages. Overall is the mean across the three tasks.
Bold = best, underline = second-best. Models are grouped into four tiers by Overall score.
|
Tier 3 Β· Rank 23β33
|
Tier 4 Β· Rank 34β44
|
Each entry reports the success rate (%) with the number of passed tasks in parentheses.
The offline and online rankings show a clear positive association.
| Online | Model | Offline | Short-Horizon Subtasks | Long-Horizon Tasks | Causal | Thematic |
|---|---|---|---|---|---|---|
| π₯ | GPT-6-Astra | 1 | 66.1Β (41/62) | 45.0Β (9/20) | 40.0Β (4/10) | 50.0Β (5/10) |
| π₯ | Gemini 3.6 Flash | 4 | 56.5Β (35/62) | 30.0Β (6/20) | 30.0Β (3/10) | 30.0Β (3/10) |
| π₯ | Kimi-K3 | 6 | 46.8Β (29/62) | 10.0Β (2/20) | 20.0Β (2/10) | 0.0Β (0/10) |
| 4 | GPT-5.6 Terra | 14 | 37.1Β (23/62) | 10.0Β (2/20) | 20.0Β (2/10) | 0.0Β (0/10) |
| 5 | GPT-5.6 Luna | 17 | 33.9Β (21/62) | 10.0Β (2/20) | 10.0Β (1/10) | 10.0Β (1/10) |
| 6 | MiniMax-M3 | 21 | 29.0Β (18/62) | 10.0Β (2/20) | 10.0Β (1/10) | 10.0Β (1/10) |
| 7 | GLM-5V-Turbo | 27 | 27.4Β (17/62) | 5.0Β (1/20) | 10.0Β (1/10) | 0.0Β (0/10) |
| 8 | Qwen3.5-397B-A17B | 29 | 19.4Β (12/62) | 5.0Β (1/20) | 10.0Β (1/10) | 0.0Β (0/10) |
| 9 | Step3-VL-10B | 32 | 14.5Β (9/62) | 5.0Β (1/20) | 10.0Β (1/10) | 0.0Β (0/10) |
| 10 | Qwen3.6-35B-A3B | 34 | 11.3Β (7/62) | 5.0Β (1/20) | 10.0Β (1/10) | 0.0Β (0/10) |
| 11 | InternVL3.5-8B | 38 | 3.2Β (2/62) | 0.0Β (0/20) | 0.0Β (0/10) | 0.0Β (0/10) |
| 12 | UI-TARS-1.5-7B | 41 | 1.6Β (1/62) | 0.0Β (0/20) | 0.0Β (0/10) | 0.0Β (0/10) |
If you find GameHorizon Suite useful, please consider citing:
@article{GameHorizonSuite2026,
title={GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay},
author={Yiran Wang and Xingyilang Yin and Junfu Pu and Guangzhi Wang and Kaifeng Li and Mingyu Ouyang and Huiqiang Sun and Lingen Li and Cheng Cheng and Wangbo Yu and Honghao Chen and Xiaodong Cun and Chi-Man Pun and Zhiguo Cao and Ying Shan},
year={2026},
journal={arXiv preprint arXiv:2609.25001},
url={https://arxiv.org/abs/2609.25001},
}
