Dohwan Ko1
Jinyoung Park2
Seoung Choi2
Sanghyeok Lee2
Seohyun Lee2
Hyunwoo J. Kim2
1Korea University 2KAIST
Official implementation of MoE-GRPO (CVPR 2026).
Standard MoE architectures activate experts by taking a deterministic top-K over gating scores. This restricts the exploration of expert combinations, overlooks better routings, and overfits the model to a small subset of experts. MoE-GRPO instead formulates expert selection as a sequential decision-making problem and optimizes the routing policy with reinforcement learning.
The training objective has two complementary terms:
-
Token-GRPO — the usual token-level GRPO objective, computed under the sampled expert routing
$E^i$ of each rollout. - Gate-GRPO — a clipped surrogate over the gating network's probability of the experts it actually selected, using the same group-normalized advantage. This turns a single binary reward into dense supervision at every layer and token position.
Because the gating networks are trained from scratch, there is no pretrained routing policy to regularize toward, so MoE-GRPO uses no reference model and no KL term.
Modality-aware router guidance. Exploring the full sequential expert-selection space is slow and unstable. We compute a per-expert modality-awareness score from how often each expert is selected by vision vs. text tokens, deactivate the bottom 25% of experts for each modality by setting their gating scores to
At inference, expert selection is deterministic top-K.
conda create -n moe-grpo python=3.12
conda activate moe-grpo
bash setup.shTraining uses 100K multiple-choice visual instruction-tuning examples sampled from OneThinker. Place the annotation JSON and the media files under src/r1-v/data/:
src/r1-v/data/
├── OneThinker/
│ ├── image-video-mc_100k.json # annotation file (--dataset_name)
│ ├── <images...>
│ └── <videos...>
└── annotations/
└── <benchmark>.json # evaluation annotations
Each annotation entry holds question, options, answer, and either image or video (a path relative to --data_root/<dataset_name>/).
Evaluation videos and images should be downloaded from each benchmark's official release and placed under src/r1-v/evaluation/data/ following the paths referenced in the annotation JSONs.
Put the base checkpoint at src/r1-v/pretrained/InternVL3_5-1B (InternVL3.5-1B).
bash src/scripts/run_grpo.shThe dense InternVL3.5-1B is upcycled into an MoE at load time — each layer's FFN is copied into
| Base model | InternVL3.5-1B (1.3B activated / 2.9B total) |
| Experts |
|
| Rollouts |
|
| Router guidance | modality-aware, bottom 25% deactivated per modality |
| Reward | accuracy (1 if the predicted option letter matches, else 0) |
| Optimizer | lr 1e-6, cosine schedule, 25K steps, batch size 4 |
| Hardware | 4 GPUs, DeepSpeed ZeRO-2, ~1 day |
| Inputs | up to 8 images / 8 video frames |
bash src/r1-v/eval_internvl.sh <gpu_id> "<checkpoint_name>" "<datasets>"For example:
bash src/r1-v/eval_internvl.sh 0 "moe-grpo" "mme mmbench mmstar mmt-bench ai2d videomme mlvu longvideobench mvbench"Supported benchmarks:
- Image — MME, MMBench, MMStar, MMT-Bench, AI2D, SEED-Bench (image), MME-RealWorld
- Video — VideoMME, MLVU, LongVideoBench, MVBench, EgoSchema, SEED-Bench (video), TempCompass, PerceptionTest
Per-benchmark results are written to src/r1-v/evaluation/results/<dataset>/<checkpoint>.json.
This codebase builds on R1-V and Video-R1. The base model is InternVL3.5.
@inproceedings{ko2026moegrpo,
title = {MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models},
author = {Ko, Dohwan and Park, Jinyoung and Choi, Seoung and Lee, Sanghyeok and Lee, Seohyun and Kim, Hyunwoo J.},
booktitle = {CVPR},
year = {2026}
}