Skip to content

About

The official repo of GameHorizon, a unified data and evaluation suite that measures AAA gameplay capabilities at different temporal horizons for diverse model families.

Resources

Stars

602 stars

Watchers

7 watching

Forks

Latest commit

Β 

History

56 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

GameHorizon Suite β€” Multi-Horizon Data and Evaluation in Gameplay


Yiran Wang1,*,†, Xingyilang Yin1,2,6,*, Junfu Pu1,*, Guangzhi Wang1,*, Kaifeng Li2, Mingyu Ouyang1,3, Huiqiang Sun1,4,
Lingen Li1,5, Cheng Cheng1, Wangbo Yu1, Honghao Chen1, Xiaodong Cun1,2,βœ‰, Chi-Man Pun6, Zhiguo Cao4, Ying Shan1

1ARC Lab, Tencent IEG Β  2GVC Lab, Great Bay University Β  3NUS Β  4HUST Β  5MMLab, CUHK Β  6University of Macau
† Project Lead Β Β  * Equal Contribution Β Β  βœ‰ Corresponding Author


πŸ“’ News

  • [2026.11.11] πŸ—οΈ GameHorizon-Annotator is released β€” the automated pipeline that builds our multi-horizon instruction pyramid! Along with it, we release GameHorizon-Annotator-Demo and the keybind maps for all 21 titles so you can run the whole pipeline right away.
  • [2026.10.26] ⏳ Due to our company's internal approval process, the release will be delayed by one week. We will start releasing next week, beginning with the Annotator. Sorry for the delay and thanks for your patience! πŸ™
  • [2026.09.22] πŸ“„ The paper and project page of GameHorizon Suite are released!
  • We will progressively release the code, benchmark, and data starting around 2026.10.25. Stay tuned! ⭐

πŸš€ Open-Source Plan

We are actively preparing the release of our code, benchmark, and data. Star the repo to follow our progress.

πŸ“– Overview

GameHorizon Suite overview

We introduce GameHorizon, a large-scale data and evaluation suite spanning multiple horizons and AAA games. It serves as a unified yardstick across a broad range of model types. It consists of three key components. GameHorizon-Annotator automatically produces a three-level pyramid of short-horizon operations, medium-horizon goals, and long-horizon strategies. GameHorizon-Data contains 5,000 hours of gameplay across 21 game titles, with temporally aligned videos, actions, and multi-horizon instructions. GameHorizon-Bench provides reproducible offline and stepwise online testing. The offline track contains thousands of standardized MCQs across three primary tasks and diagnostic variants. The online track tests order-dependent causal and order-flexible thematic tasks via the verifiable subtasks for failure localization.

Component What it is Highlights
πŸ—οΈ Annotator A scalable and automated annotation pipeline L1 β†’ L2 β†’ L3 instruction pyramid
🎯 Data A large-scale AAA gameplay corpus 5,000 hours · AAA-focused · 21 titles · temporally-aligned videos, actions & multi-horizon instructions
πŸ“Š Bench Reproducible offline + stepwise online track 5,000 offline MCQs (3 primary + 10 variant tasks) Β· 20 online tasks / 62 subtasks Β· 47 models benchmarked

πŸ—οΈ GameHorizon-Annotator

GameHorizon-Annotator pipeline

GameHorizon-Annotator turns gameplay video plus temporally aligned keyboard-and-mouse actions into the three-level instruction pyramid β€” short-horizon operations β†’ medium-horizon goals β†’ long-horizon strategies.

Quick Start

cd GameHorizon-Annotator
uv sync                                   # or: pip install -e .

export OPENAI_API_KEY="your-key"
export OPENAI_BASE_URL="https://your-provider/v1"

python -m annotator.cli.run_full_pipeline \
  --game minecraft \
  --video ./data/gameplay.mp4 \
  --annotations ./data/annotations.json \
  --workdir ./outputs/minecraft-scene

Level-1/2/3 instructions land in ./outputs/minecraft-scene/level_{1,2,3}_instructions.json.

To try it without preparing your own recording, we release six demo clips β€” Apex Legends, Cyberpunk 2077, Elden Ring, Elden Ring Nightreign, GTA V and Minecraft β€” each with its video, aligned keyboard-mouse actions and annotated instructions, alongside the keybind maps for all 21 titles: πŸ€— GameHorizon-Data

πŸ“– Input format, pipeline stages and all options β†’ GameHorizon-Annotator/README.md

πŸ† Leaderboard

Offline Results β€” Primary Tasks (T1 / T2 / T3)

Accuracies are reported as percentages. Overall is the mean across the three tasks.

Bold = best, underline = second-best. Models are grouped into four tiers by Overall score.

Average (all 44 models) β€” T1 57.3 Β· T2 65.1 Β· T3 71.6 Β· Overall 64.7

πŸ₯‡ Tier 1 Β· Rank 1–11

#ModelT1T2T3Overall
πŸ₯‡GPT-6-Astra69.479.691.580.2
πŸ₯ˆGemini 3.8 Flash65.981.284.877.3
πŸ₯‰Gemini 3.7 Flash66.280.183.976.7
4Gemini 3.6 Flash62.379.184.675.3
5GPT-5.6 Sol65.377.581.574.8
6Kimi-K364.479.379.774.5
7Gemini 3.5 Flash64.977.480.874.4
8GPT-5.564.678.976.273.2
9Gemini 3.1 Pro64.575.878.272.8
10Doubao-Seed-2.1-Turbo63.179.475.572.7
11Doubao-Seed-2.1-Pro62.976.776.672.1

Tier 2 Β· Rank 12–22

#ModelT1T2T3Overall
12Doubao-Seed-2.0-Pro59.274.980.071.4
13Claude Fable 562.972.778.171.2
14GPT-5.6 Terra62.175.674.470.7
15Qwen3.7-Plus58.875.971.568.7
16Qwen3.8-27B59.977.468.168.5
17GPT-5.6 Luna61.670.771.167.8
17Kimi-K2.661.273.269.067.8
19Doubao-Seed-2.0-Lite55.269.076.366.8
20Gemma 4 31B-IT53.868.676.666.3
21MiniMax-M357.868.570.665.6
22Qwen3-VL-235B-A22B-Thinking58.271.665.965.2

Tier 3 Β· Rank 23–33

#ModelT1T2T3Overall
23Claude Opus 4.859.162.173.164.8
24Claude Sonnet 554.457.782.064.7
25Step-3.7-Flash57.571.363.464.1
26Qwen3-VL-235B-A22B-Instruct55.162.173.163.4
27GLM-5V-Turbo53.465.470.863.2
28GPT-5.258.958.970.662.8
29Qwen3.5-397B-A17B53.360.173.562.3
30BAGEL-7B-MoT51.757.375.161.4
31Qwen2.5-VL-32B-Instruct54.858.069.760.8
32Step3-VL-10B55.464.761.760.6
33SenseNova-U1-8B-MoT48.259.267.558.3

Tier 4 Β· Rank 34–44

#ModelT1T2T3Overall
34Qwen3.6-35B-A3B53.352.966.357.5
35Qwen3-Omni-30B-A3B-Instruct52.555.663.757.3
36GELab-Zero-4B-Preview51.846.872.757.1
37Qwen2.5-VL-7B-Instruct52.054.363.756.7
38InternVL3.5-8B51.850.265.555.8
39GLM-4.1V-9B-Thinking54.954.855.355.0
40GPT-4o55.746.361.354.4
41UI-TARS-1.5-7B49.147.162.953.0
42Ovis-U1-3B44.739.659.247.8
43InternVL3.5-2B45.140.354.746.0
44InternVL-U-4B46.137.949.844.6

Online Results β€” Long Horizon Tasks and Short-Horizon Subtasks

Each entry reports the success rate (%) with the number of passed tasks in parentheses.

The offline and online rankings show a clear positive association.

Online Model Offline Short-Horizon Subtasks Long-Horizon Tasks Causal Thematic
πŸ₯‡ GPT-6-Astra 1 66.1Β (41/62) 45.0Β (9/20) 40.0Β (4/10) 50.0Β (5/10)
πŸ₯ˆ Gemini 3.6 Flash 4 56.5Β (35/62) 30.0Β (6/20) 30.0Β (3/10) 30.0Β (3/10)
πŸ₯‰ Kimi-K3 6 46.8Β (29/62) 10.0Β (2/20) 20.0Β (2/10) 0.0Β (0/10)
4 GPT-5.6 Terra 14 37.1Β (23/62) 10.0Β (2/20) 20.0Β (2/10) 0.0Β (0/10)
5 GPT-5.6 Luna 17 33.9Β (21/62) 10.0Β (2/20) 10.0Β (1/10) 10.0Β (1/10)
6 MiniMax-M3 21 29.0Β (18/62) 10.0Β (2/20) 10.0Β (1/10) 10.0Β (1/10)
7 GLM-5V-Turbo 27 27.4Β (17/62) 5.0Β (1/20) 10.0Β (1/10) 0.0Β (0/10)
8 Qwen3.5-397B-A17B 29 19.4Β (12/62) 5.0Β (1/20) 10.0Β (1/10) 0.0Β (0/10)
9 Step3-VL-10B 32 14.5Β (9/62) 5.0Β (1/20) 10.0Β (1/10) 0.0Β (0/10)
10 Qwen3.6-35B-A3B 34 11.3Β (7/62) 5.0Β (1/20) 10.0Β (1/10) 0.0Β (0/10)
11 InternVL3.5-8B 38 3.2Β (2/62) 0.0Β (0/20) 0.0Β (0/10) 0.0Β (0/10)
12 UI-TARS-1.5-7B 41 1.6Β (1/62) 0.0Β (0/20) 0.0Β (0/10) 0.0Β (0/10)

πŸ“œ Citation

If you find GameHorizon Suite useful, please consider citing:

@article{GameHorizonSuite2026,
      title={GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay}, 
      author={Yiran Wang and Xingyilang Yin and Junfu Pu and Guangzhi Wang and Kaifeng Li and Mingyu Ouyang and Huiqiang Sun and Lingen Li and Cheng Cheng and Wangbo Yu and Honghao Chen and Xiaodong Cun and Chi-Man Pun and Zhiguo Cao and Ying Shan},
      year={2026},
      journal={arXiv preprint arXiv:2609.25001},
      url={https://arxiv.org/abs/2609.25001}, 
}

Β© 2026 ARC Lab, Tencent Β· GameHorizon Suite

About

The official repo of GameHorizon, a unified data and evaluation suite that measures AAA gameplay capabilities at different temporal horizons for diverse model families.

Resources

Stars

602 stars

Watchers

7 watching

Forks

Releases

Packages

Contributors

Languages