Skip to content
mlvlabPublic

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models

Dohwan Ko1   Jinyoung Park2   Seoung Choi2   Sanghyeok Lee2   Seohyun Lee2   Hyunwoo J. Kim2
1Korea University    2KAIST

arXiv CVPR 2026

Official implementation of MoE-GRPO (CVPR 2026).

Overview

Standard MoE architectures activate experts by taking a deterministic top-K over gating scores. This restricts the exploration of expert combinations, overlooks better routings, and overfits the model to a small subset of experts. MoE-GRPO instead formulates expert selection as a sequential decision-making problem and optimizes the routing policy with reinforcement learning.

The training objective has two complementary terms:

  • Token-GRPO — the usual token-level GRPO objective, computed under the sampled expert routing $E^i$ of each rollout.
  • Gate-GRPO — a clipped surrogate over the gating network's probability of the experts it actually selected, using the same group-normalized advantage. This turns a single binary reward into dense supervision at every layer and token position.

Because the gating networks are trained from scratch, there is no pretrained routing policy to regularize toward, so MoE-GRPO uses no reference model and no KL term.

Modality-aware router guidance. Exploring the full sequential expert-selection space is slow and unstable. We compute a per-expert modality-awareness score from how often each expert is selected by vision vs. text tokens, deactivate the bottom 25% of experts for each modality by setting their gating scores to $-\infty$, and sample $K$ experts from the remainder. This keeps exploration on modality-consistent experts while preserving enough diversity for policy learning.

At inference, expert selection is deterministic top-K.

Installation

conda create -n moe-grpo python=3.12
conda activate moe-grpo
bash setup.sh

Data preparation

Training uses 100K multiple-choice visual instruction-tuning examples sampled from OneThinker. Place the annotation JSON and the media files under src/r1-v/data/:

src/r1-v/data/
├── OneThinker/
│   ├── image-video-mc_100k.json     # annotation file (--dataset_name)
│   ├── <images...>
│   └── <videos...>
└── annotations/
    └── <benchmark>.json             # evaluation annotations

Each annotation entry holds question, options, answer, and either image or video (a path relative to --data_root/<dataset_name>/).

Evaluation videos and images should be downloaded from each benchmark's official release and placed under src/r1-v/evaluation/data/ following the paths referenced in the annotation JSONs.

Put the base checkpoint at src/r1-v/pretrained/InternVL3_5-1B (InternVL3.5-1B).

Training

bash src/scripts/run_grpo.sh

The dense InternVL3.5-1B is upcycled into an MoE at load time — each layer's FFN is copied into $N$ experts with small Gaussian noise for symmetry breaking — and only the MoE blocks (experts + gate) are trained.

Base model InternVL3.5-1B (1.3B activated / 2.9B total)
Experts $N = 8$, $K = 2$ per token, all layers
Rollouts $G = 8$, greedy decoding, stochasticity only in expert selection
Router guidance modality-aware, bottom 25% deactivated per modality
Reward accuracy (1 if the predicted option letter matches, else 0)
Optimizer lr 1e-6, cosine schedule, 25K steps, batch size 4
Hardware 4 GPUs, DeepSpeed ZeRO-2, ~1 day
Inputs up to 8 images / 8 video frames

Evaluation

bash src/r1-v/eval_internvl.sh <gpu_id> "<checkpoint_name>" "<datasets>"

For example:

bash src/r1-v/eval_internvl.sh 0 "moe-grpo" "mme mmbench mmstar mmt-bench ai2d videomme mlvu longvideobench mvbench"

Supported benchmarks:

  • Image — MME, MMBench, MMStar, MMT-Bench, AI2D, SEED-Bench (image), MME-RealWorld
  • Video — VideoMME, MLVU, LongVideoBench, MVBench, EgoSchema, SEED-Bench (video), TempCompass, PerceptionTest

Per-benchmark results are written to src/r1-v/evaluation/results/<dataset>/<checkpoint>.json.

Acknowledgements

This codebase builds on R1-V and Video-R1. The base model is InternVL3.5.

Citation

@inproceedings{ko2026moegrpo,
  title     = {MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models},
  author    = {Ko, Dohwan and Park, Jinyoung and Choi, Seoung and Lee, Sanghyeok and Lee, Seohyun and Kim, Hyunwoo J.},
  booktitle = {CVPR},
  year      = {2026}
}

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages