Kaname Yokoyama1 Β· Norimichi Ukita1
1 Toyota Technological Institute
A monocular real video with the estimated ViTPose 2D keypoints overlaid. The caption underneath is generated by our 2D Motion Interface β no 3D pose estimation anywhere in the pipeline.
- π Plug-and-play, at very low training cost. The interface is an external module: no modification and no fine-tuning of the pretrained MoLM.
E_2Dhas 9.6M parameters (~22 h) and the real-video adapterA_realonly 0.3M (~1 h), both on a single A100 40GB. - π Comparable to 3D motion inputs. On HumanML3D motion captioning, Avg Drop vs 3D Input is β0.2% (TM2T), +0.4% (MotionGPT), and β2.8% (MG-MotionLLM).
- π₯ Better than training MoLMs from scratch on 2D. 2D Scratch drops β25.8% (TM2T) and β12.8% (MG-MotionLLM) versus 3D Input β our interface substantially reduces that gap at a fraction of the compute.
- π¬ 2D and 3D motions align in both latent and token space. Discrete token agreement with 3D-derived tokens reaches 44.8% / 59.6% / 67.1% (Top-1/2/3), against a 0.20% random baseline for a codebook of size 512.
- π₯ On real monocular video, 2D beats estimated 3D. 2D Input w/
A_realoutperforms all 3D Input settings on almost all metrics across all three MoLMs β while being far cheaper: ViTPose (Base) needs 17.2 GFLOPs/frame versus 250.2 for WHAM. - π New monocular real-world video motion dataset. 132 videos from 10 participants (14,878 frames / 743.9 s), paired with estimated 2D (ViTPose) and 3D (TRACE, WHAM) motions and HumanML3D captions; 86.4% of videoβcaption pairs were human-verified as semantically consistent. Publicly released.
π Table of Contents
A new 2D motion encoder E_2D is trained to map 2D motion into the same latent space as the MoLM's own 3D encoder, with a simple L1 loss on the latents. The codebook, decoder, and language model stay frozen and are reused as-is, so 2D motion is tokenized by the original codebook and captioned by the pretrained language model β with no 3D pose estimation in the loop.
E_2D is trained on clean 2D projections of mocap, while real video adds perspective distortion and pose-estimation noise. A small real-video adapter A_real (0.3M params) inserted before E_2D absorbs that gap. Since real videos have no ground-truth 3D, it is trained on pseudo-real pairs: AMASS motions rendered from random viewpoints, run through ViTPose, and paired with their own 3D.
Both tables report Avg Drop, the average percentage difference from the reference 3D setting across all metrics (higher is better, β is the reference row itself).
1 Β· Motion captioning on HumanML3D β Avg Drop vs 3D Input. Replacing 3D inputs with our 2D interface costs almost nothing, while training the MoLM from scratch on 2D costs a great deal.
| Base model | 2D Scratch | Ours (2D Motion Interface) |
|---|---|---|
| TM2T | β25.8% | β0.2% |
| MotionGPT | not reported β | +0.4% |
| MG-MotionLLM | β12.8% | β2.8% |
β The MotionGPT 2D-Scratch result is omitted because the official implementation could not be stably retrained; the same reproducibility issue is open in the official repository.
2 Β· Motion captioning on monocular real videos β Avg Drop vs Ref 3D Input. A_real is our real-video adapter; A_3D is the same adapter applied to WHAM's estimated 3D motion for a fair comparison.
| Base model | 3D (TRACE) | 3D (WHAM) | 2D | 3D (WHAM) w/ A_3D |
2D w/ A_real |
|---|---|---|---|---|---|
| TM2T | β37.7% | β35.9% | β18.5% | β10.8% | +4.3% |
| MotionGPT | β40.3% | β34.1% | β31.4% | β21.1% | β18.5% |
| MG-MotionLLM | β50.7% | β35.6% | β46.7% | β34.9% | β17.5% |
2D Input w/ A_real outperforms all 3D Input settings on almost all metrics across all models β and does so far more cheaply: excluding human detection, ViTPose (Base) requires 17.2 GFLOPs/frame against 250.2 for WHAM.
Note
Ref 3D Input is the original HumanML3D motion sampled when collecting each video. It is not the strict ground-truth 3D motion of what the participant actually performed, so it serves as a reference point rather than a hard upper bound.
Full per-metric tables are in Reproducing Key Results.
| Directory | Base Model | Tasks | Role in this work |
|---|---|---|---|
2DMotionGPT/ |
MotionGPT | M2T | Trains the 2D encoder and both adapters β the source of the shared weights |
2DTM2T/ |
TM2T | M2T | Reuses the 2D encoder and adapters above |
2DMG-MotionLLM/ |
MG-MotionLLM | M2T, M2DT | Trains its own 2D encoder and adapters independently |
M2T = motion-to-text Β· M2DT = motion-to-detailed-text
Each sub-project uses its own conda environment β the PyTorch versions are mutually incompatible, so all three are needed to reproduce everything.
| Sub-project | Env name | Python | PyTorch | CUDA |
|---|---|---|---|---|
2DMotionGPT/ |
mgpt_2d |
3.10 | 2.9.0 | 11.8 |
2DTM2T/ |
tm2t |
3.7 | 1.13.1 | 10.1 |
2DMG-MotionLLM/ |
mg-motionllm |
3.9 | 2.8.0 | 11.8 |
cd <sub-project> # 2DMotionGPT | 2DTM2T | 2DMG-MotionLLM
conda env create -f environment.yml
conda activate <env-name> # mgpt_2d | tm2t | mg-motionllmOne more environment is needed only to run ViTPose on your own videos (Caption your own video) β the released datasets already ship their 2D keypoints, so reproducing the paper does not need it:
| Purpose | Env name | Python | PyTorch | CUDA | Key packages |
|---|---|---|---|---|---|
| ViTPose 2D pose estimation | openmmlab |
3.8 | 1.9.1 | 11.1 | mmpose 1.1.0 Β· mmdet 3.1.0 Β· mmcv 2.0.1 |
cd 2DMotionGPT
conda env create -f environment_openmmlab.ymlAll data lives under 2DMotionGPT/dataset/:
2DMotionGPT/dataset/
βββ HumanML3D/ # HumanML3D
β βββ new_joint_vecs/ texts/ Mean.npy Std.npy
β βββ train.txt val.txt test.txt all.txt
β βββ finemotion_texts/ # FineMotion β 2DMG-MotionLLM only
βββ real_world_dataset_ver2/ # our real-world video dataset (2D / ViTPose)
βββ gt_upper_bound_wham/ # WHAM 3D motions of the same real videos
βββ humanml3d_for_render/ # pseudo-real ViTPose 2D pairs β A_real training
βββ humanml3d_for_render_wham/ # WHAM 3D pairs β A_3D training
2DTM2T/ and 2DMG-MotionLLM/ need no copy of their own β symlink them to the same data:
# from the repository root
mkdir -p 2DTM2T/dataset 2DMG-MotionLLM/dataset
ln -s ../../2DMotionGPT/dataset/HumanML3D 2DTM2T/dataset/HumanML3D
ln -s ../../2DMotionGPT/dataset/HumanML3D 2DMG-MotionLLM/dataset/HumanML3DFollow the HumanML3D repository to download and prepare the dataset. Expected structure:
HumanML3D/
βββ new_joint_vecs/
βββ texts/
βββ Mean.npy
βββ Std.npy
βββ train.txt
βββ val.txt
βββ test.txt
βββ all.txt
Needed for the Motion-to-Detailed-Text task. Download BPMSD_auto.zip and BPMSD_human.zip from the FineMotion repository and place them under dataset/HumanML3D/finemotion_texts/:
dataset/HumanML3D/
βββ finemotion_texts/
β βββ BPMSD_auto.zip
β βββ BPMSD_auto.json
β βββ BPMSD_human.zip
β βββ BPMSD_human.json
βββ ...
Motion clips were randomly sampled from the HumanML3D test set and shown to participants, who imitated them while being recorded with a single RGB camera (iPhone SE 2nd generation), varying the relative viewpoint between subject and camera. Each video is paired with estimated 2D motion (ViTPose), estimated 3D motion (TRACE and WHAM), and the original HumanML3D caption of the sampled clip. All performers participated with informed consent.
| Videos | 132, from 10 participants (male and female, 20sβ50s) |
| Frames / duration | 14,878 frames Β· 743.9 s |
| Captions | original HumanML3D captions of the sampled clips |
| Human verification | 3 annotators; 86.4% of videoβcaption pairs judged semantically consistent |
Expected structure:
dataset/
βββ HumanML3D/
βββ gt_upper_bound_wham/
βββ humanml3d_for_render/
βββ humanml3d_for_render_wham/
βββ real_world_dataset_ver2/
GloVe embeddings, evaluators, and base checkpoints come from the original repositories:
# MotionGPT
cd 2DMotionGPT
bash prepare/prepare_t5.sh
bash prepare/download_t2m_evaluators.sh
bash prepare/download_pretrained_models.sh
# MG-MotionLLM
cd 2DMG-MotionLLM
bash prepare/prepare_t5.sh
bash prepare/download_glove.sh
bash prepare/download_vqvae.shFor MG-MotionLLM, also download the baseline checkpoints and place them in 2DMG-MotionLLM/:
Once Installation, the real-world dataset, and our checkpoints are in place, this single command reproduces the headline real-world result:
cd 2DMotionGPT
conda activate mgpt_2d
python evaluate_with_realworld.py \
--cfg configs/config_h3d_stage1.yaml --nodebug \
--estimated_motion_dir ./dataset/real_world_dataset_ver2/pred \
--vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
--adapter_ckpt ./checkpoints/adapter/MotionGPT_2DEncoder/adapter-residual-bs64-lr1e-3/best_adapter_epoch690_valloss1.2251.tar \
--adapter_type residual \
--mode both--mode both evaluates with and without the adapter in one pass, so you get the comparison directly.
demo/ runs the whole thing on an arbitrary monocular video β ViTPose 2D pose estimation, then captioning through A_real β E_2D β MotionGPT:
cd 2DMotionGPT
python demo/demo.py --video /path/to/your_video.mp4 --gpu_id 0The driver only needs the standard library β it runs pose estimation in the openmmlab environment and captioning in mgpt_2d, since mmpose (torch 1.9) and MotionGPT (torch 2.9) cannot share one environment. Interpreters are found automatically; override with POSE_PYTHON / CAPTION_PYTHON if needed. Outputs land in demo/outputs/<video name>/: keypoints.json, overlay.mp4 (skeleton overlay), caption.json, pose_meta.json.
Only the weight bundle in space/bundle/ is required β no HumanML3D, no evaluators. The video is resampled to 20 fps automatically; use --start / --duration to pick a segment, since the model captions at most 196 frames (9.8 s). Best results come from a single person, full body in frame, roughly static camera.
Running the two stages separately
# stage 1 β openmmlab env
python demo/run_vitpose_single.py --video in.mp4 --out_json kp.json --out_video overlay.mp4
# stage 2 β mgpt_2d env
python demo/caption_from_json.py --json kp.json --mode both2DMotionGPT β evaluate 2D encoder and 3D baseline side by side:
cd 2DMotionGPT
conda activate mgpt_2d
# 3D baseline
python evaluate_motionGPT.py --gpu_id 7 --eval_3d \
--output results/eval_3d_bs1.json
# 2D encoder
python evaluate_motionGPT.py --gpu_id 7 \
--vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
--output results/eval_2d_bs1.jsonπ Paper Results β Table 1, MotionGPT (HumanML3D test set)
| Method | R-Prec Top-1β | Top-2β | Top-3β | MM-Distβ | BLEU-1β | BLEU-4β | ROUGE-Lβ | CIDErβ | BERTScoreβ | Avg Drop vs 3Dβ |
|---|---|---|---|---|---|---|---|---|---|---|
| 3D Input | 0.516 | 0.707 | 0.803 | 2.994 | 0.429 | 0.062 | 0.345 | 0.079 | 0.316 | β |
| Ours | 0.523 Β±0.002 |
0.717 Β±0.003 |
0.802 Β±0.003 |
2.985 Β±0.015 |
0.431 Β±0.000 |
0.063 Β±0.001 |
0.346 Β±0.001 |
0.077 Β±0.001 |
0.319 Β±0.001 |
+0.4% |
Ours is the mean Β± std over three runs. 2D Scratch is not reported for MotionGPT: the official implementation could not be stably retrained, an issue also open in the official repository.
2DMG-MotionLLM β evaluate 3D baseline and 2D encoder sequentially:
cd 2DMG-MotionLLM
# 3D baseline + 2D encoder (2D encoder is auto-detected from checkpoints/2d_vq_train/t2m/)
# M2T
python eval_m2t.py --model_name ./m2t-ft-from-GSPretrained-base
# M2DT
python eval_m2dt.py --model_name ./m2dt-ft-from-GSPretrained-base
# 2D scratch (full VQ-VAE trained from scratch on 2D data)
# M2T
python eval_m2t.py --model_name ./m2t-2drecon-100k/checkpoint-100000 \
--vqvae_2drecon_ckpt ../2DMotionGPT/checkpoints/2d_vqvae_full/MotionGPT_2DEncoder/full_recon_2d-l1_smooth-bs64-lr0.0001/best_vqvae_epoch2990_valloss0.0354.tar
# M2DT
python eval_m2dt.py --model_name ./m2dt-2drecon/checkpoint-300000 \
--vqvae_2drecon_ckpt ../2DMotionGPT/checkpoints/2d_vqvae_full/MotionGPT_2DEncoder/full_recon_2d-l1_smooth-bs64-lr0.0001/best_vqvae_epoch2990_valloss0.0354.tarπ Paper Results β Tables 1 & 2, MG-MotionLLM
M2T β motion captioning on the HumanML3D test set (Table 1):
| Method | R-Prec Top-1β | Top-2β | Top-3β | MM-Distβ | BLEU-1β | BLEU-4β | ROUGE-Lβ | CIDErβ | BERTScoreβ | Avg Drop vs 3Dβ |
|---|---|---|---|---|---|---|---|---|---|---|
| 3D Input | 0.583 | 0.787 | 0.873 | 2.571 | 0.507 | 0.091 | 0.402 | 0.095 | 0.386 | β |
| 2D Scratch | 0.487 | 0.667 | 0.758 | 3.289 | 0.494 | 0.078 | 0.385 | 0.074 | 0.364 | β12.8% |
| Ours | 0.576 | 0.770 | 0.855 | 2.701 | 0.499 | 0.087 | 0.396 | 0.090 | 0.379 | β2.8% |
M2DT β motion-to-detailed-text on the FineMotion test set (Table 2):
| Level | Method | BLEU-1β | BLEU-4β | BLEU-7β | ROUGE-Lβ | BERTScoreβ | Avg Drop vs 3Dβ |
|---|---|---|---|---|---|---|---|
| Sequence | 3D Input | 0.828 | 0.667 | 0.536 | 0.651 | 0.523 | β |
| Sequence | 2D Scratch | 0.814 | 0.637 | 0.497 | 0.629 | 0.483 | β4.9% |
| Sequence | Ours | 0.828 | 0.665 | 0.533 | 0.648 | 0.517 | β0.5% |
| Snippet | 3D Input | 0.672 | 0.478 | 0.354 | 0.606 | 0.509 | β |
| Snippet | 2D Scratch | 0.653 | 0.456 | 0.329 | 0.588 | 0.487 | β4.4% |
| Snippet | Ours | 0.666 | 0.473 | 0.349 | 0.601 | 0.502 | β1.1% |
Ours rows are the mean over three runs (Β± β€0.021 on every metric). The M2DT protocol follows MG-MotionLLM and reports BLEU-7 rather than CIDEr.
Requires the real-world video dataset (see Dataset Preparation).
2DMotionGPT:
cd 2DMotionGPT
# ViTPose 2D + 2D Adapter (main result)
python evaluate_with_realworld.py \
--cfg configs/config_h3d_stage1.yaml --nodebug \
--estimated_motion_dir ./dataset/real_world_dataset_ver2/pred \
--vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
--adapter_ckpt ./checkpoints/adapter/MotionGPT_2DEncoder/adapter-residual-bs64-lr1e-3/best_adapter_epoch690_valloss1.2251.tar \
--adapter_type residual \
--mode both
# WHAM 3D + 3D Adapter
python evaluate_with_realworld_3d_adapter.py \
--cfg configs/config_h3d_stage1.yaml --nodebug \
--estimated_motion_dir ./dataset/gt_upper_bound_wham \
--adapter_ckpt ./checkpoints/adapter3d/MotionGPT_2DEncoder/adapter3d-residual-bs64-lr1e-3/best_adapter3d_epoch880_valloss1.2810.tar \
--split test \
--mode bothπ Paper Results β Table 4, MotionGPT (monocular real-world video motion dataset)
| Method | BLEU-1β | BLEU-4β | ROUGE-Lβ | CIDErβ | BERTScoreβ | Avg Drop vs Ref 3Dβ |
|---|---|---|---|---|---|---|
| Ref 3D Input | 0.461 | 0.069 | 0.405 | 0.149 | 0.352 | β |
| 3D Input (TRACE) | 0.369 | 0.034 | 0.315 | 0.044 | 0.218 | β40.3% |
| 3D Input (WHAM) | 0.380 | 0.035 | 0.339 | 0.056 | 0.264 | β34.1% |
| 2D Input | 0.361 | 0.047 | 0.321 | 0.075 | 0.236 | β31.4% |
3D Input (WHAM) w/ A_3D |
0.408 | 0.046 | 0.365 | 0.102 | 0.285 | β21.1% |
2D Input w/ A_real |
0.435 | 0.045 | 0.367 | 0.105 | 0.306 | β18.5% |
Bold marks the best value among the estimated-motion rows (Ref 3D Input is a reference, not a competitor).
2DMG-MotionLLM:
cd 2DMG-MotionLLM
# ViTPose 2D + 2D Adapter (main result)
python evaluate_with_realworld.py \
--estimated_motion_dir ../2DMotionGPT/dataset/real_world_dataset_ver2/pred \
--adapter_ckpt ./checkpoints/adapter/MG-MotionLLM/best_adapter_epoch2520_valloss1.1380.pt \
--adapter_type residual \
--meta_dir ../2DMotionGPT/deps/t2m/t2m/VQVAEV3_CB1024_CMT_H1024_NRES3/meta \
--data_root ./dataset/HumanML3D \
--split test \
--mode both \
--gpu_id 0
# WHAM 3D + 3D Adapter
python evaluate_with_realworld_3d_adapter.py \
--estimated_motion_dir ../2DMotionGPT/dataset/gt_upper_bound_wham \
--adapter_ckpt ./checkpoints/adapter3d/MG-MotionLLM/best_adapter3d_epoch610_valloss1.1204.pt \
--meta_dir ../2DMotionGPT/deps/t2m/t2m/VQVAEV3_CB1024_CMT_H1024_NRES3/meta \
--data_root ./dataset/HumanML3D \
--split test \
--gpu_id 0 \
--mode bothπ Paper Results β Table 4, MG-MotionLLM (monocular real-world video motion dataset)
| Method | BLEU-1β | BLEU-4β | ROUGE-Lβ | CIDErβ | BERTScoreβ | Avg Drop vs Ref 3Dβ |
|---|---|---|---|---|---|---|
| Ref 3D Input | 0.556 | 0.117 | 0.450 | 0.233 | 0.436 | β |
| 3D Input (TRACE) | 0.385 | 0.033 | 0.310 | 0.058 | 0.240 | β50.7% |
| 3D Input (WHAM) | 0.433 | 0.064 | 0.350 | 0.095 | 0.308 | β35.6% |
| 2D Input | 0.392 | 0.044 | 0.321 | 0.062 | 0.264 | β46.7% |
3D Input (WHAM) w/ A_3D |
0.469 | 0.036 | 0.378 | 0.106 | 0.352 | β34.9% |
2D Input w/ A_real |
0.507 | 0.093 | 0.411 | 0.151 | 0.374 | β17.5% |
Three steps: (1) train the 2D motion encoder E_2D, (2) train the real-video adapter A_real for ViTPose inputs, (3) train the adapter A_3D for estimated-3D inputs (the fair-comparison arm).
Tip
The 2D interface requires no training for 2DTM2T. It reuses E_2D and the adapters trained in 2DMotionGPT/ β which is the whole point of the plug-and-play design. (The TM2T base model itself was retrained once with the matched VQ-VAE architecture; see Repository Layout.)
2DMotionGPT (encoder-only; also used by 2DTM2T):
cd 2DMotionGPT
python train_2d_vqvae_ver4_coco_normalized.py \
--cfg configs/config_h3d_stage1.yaml --nodebug2DMG-MotionLLM (trains its own 2D encoder independently):
cd 2DMG-MotionLLM
python 2d_vq_train.py --dataname t2m2DMotionGPT (adapter also used by 2DTM2T):
cd 2DMotionGPT
python -m train_adapter \
--cfg configs/config_h3d_stage1.yaml --nodebug \
--vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
--estimated_motion_dir ./dataset/humanml3d_for_render2DMG-MotionLLM:
cd 2DMG-MotionLLM
python -u train_adapter.py \
--vqvae_2d_ckpt ./checkpoints/2d_vq_train/t2m/best_2dvq_epoch1881_ratio0.5557.pt \
--estimated_motion_dir ../2DMotionGPT/dataset/humanml3d_for_render \
--meta_dir ../2DMotionGPT/deps/t2m/t2m/VQVAEV3_CB1024_CMT_H1024_NRES3/meta \
--data_root ./dataset/HumanML3D \
--ckpt_save_dir ./checkpoints/adapter/MG-MotionLLM \
--max_epochs 3000 --batch_size 64 --val_every 10 \
--adapter_type residual \
--wandb_project MG-MotionLLM-Adapter \
--gpu_id 72DMotionGPT (adapter also used by 2DTM2T):
cd 2DMotionGPT
python train_adapter_3d.py \
--cfg configs/config_h3d_stage1.yaml --nodebug \
--estimated_motion_dir ./dataset/humanml3d_for_render_wham2DMG-MotionLLM:
cd 2DMG-MotionLLM
python train_adapter_3d.py \
--estimated_motion_dir ../2DMotionGPT/dataset/humanml3d_for_render_wham \
--val_every 10 \
--val_text_every 100- Over-generalized captions. On real videos, MotionGPT and MG-MotionLLM show a larger Avg Drop vs Ref 3D than TM2T. A likely cause is that newer MoLMs generate more detailed descriptions, while the adapter's correction smooths the motion sequence and suppresses fine-grained differences, pushing the representation toward an average motion pattern. Decoding the 2D-derived tokens back through the VQ-VAE decoder confirms this: with the adapter, an S-shaped walking trajectory is no longer preserved in the reconstructed root trajectory.
- Imitation error in the dataset. Because videos were collected by asking participants to imitate HumanML3D motions, discrepancies between the reference motion and the performed motion can cause semantic mismatches (86.4% of pairs were verified as consistent).
- Scope of the 2D-vs-3D conclusion. The comparison holds under the evaluated monocular pose-estimation setting (TRACE, WHAM, ViTPose), and the analysis is conducted where 2D features are explicitly aligned to the 3D latent space. It does not establish that 3D-specific information is unnecessary in general.
If you find this work useful, please consider citing:
@inproceedings{yokoyama2026plugandplay,
title = {A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models},
author = {Yokoyama, Kaname and Ukita, Norimichi},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops,
Human-Centric Motion Intelligence Workshop (HCMIW)},
note = {Oral Presentation},
year = {2026}
}This work builds on the following excellent projects β we thank the authors for releasing their code and data:

