Official implementation of PlanTalk, a future-grounded speech-to-plan-to-motion framework for holistic co-speech gesture generation.
- [2026-09] Release training and inference code for PlanTalk.
- [Coming Soon] Release pretrained future motion planner and PlanTalk checkpoints.
- [Coming Soon] Release evaluation scripts and qualitative examples on BEAT2.
We recommend using a dedicated Conda environment.
conda create -n PlanTalk python=3.8 -y
conda activate PlanTalkInstall PyTorch and the required dependencies:
pip install -r requirements.txtPlease ensure that the CUDA and PyTorch versions are compatible with your local environment.
PlanTalk is primarily evaluated on the BEAT2 dataset using SMPL-X motion representations.
Please download BEAT2 following the official dataset instructions and organize it as:
PlanTalk/
├── BEAT2/
│ └── beat_english_v2.0.0/
├── configs/
├── dataloaders/
├── models/
├── weights/
├── utils/
└── train.py
The original dataset is not redistributed in this repository.
PlanTalk requires several pretrained components, including:
- motion representation models,
- speech feature encoder,
- future motion-plan representation model,
- speech-to-plan predictor,
- holistic motion generator.
After downloading the released checkpoints, organize the weights as:
weights/
├── pretrained_vq/
├── future_plan/
│ └── PlanTalk_future_plan_vq.bin
├── speech_encoder/
├── base_motion/
└── PlanTalk/
└── PlanTalk.bin
Pretrained checkpoints will be released after publication.
PlanTalk follows a staged training strategy.
The first stage learns a discrete representation from future motion windows.
Conceptually,
Future Motion Window
↓
Motion Plan Encoder
↓
Vector Quantization
↓
Discrete Future Plan
The learned codebook provides a compact representation of upcoming motion organization.
Example:
python train.py \
--config configs/PlanTalk_future_plan.yaml \
--train_planAfter training, save the best future-plan representation model under:
weights/future_plan/
Given speech context, the planner predicts the discrete motion plan corresponding to the upcoming motion window.
Speech Context
↓
Speech Encoder
↓
Planner
↓
Predicted Future Plan
Example:
python train.py \
--config configs/PlanTalk.yaml \
--train_planner \
--plan_vq_ckpt weights/future_plan/PlanTalk_future_plan_vq.binThe predicted future plan is used by both the event-aware semantic gate and hierarchical plan-conditioned modulation modules.
┌──────────────────────┐
Speech ────────────→│ Speech Representation│
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Future Plan Predictor│
└──────────┬───────────┘
│
Predicted Future Plan
┌─────┴─────┐
▼ ▼
Event-Aware Plan-Conditioned
Semantic Gate Modulation
\ /
\ /
▼ ▼
Holistic Motion
Example training command:
python train.py \
--config configs/PlanTalk.yaml \
--plan_vq_ckpt weights/future_plan/PlanTalk_future_plan_vq.binTo evaluate a trained PlanTalk model:
python train.py \
--config configs/PlanTalk.yaml \
--test_state \
--load_ckpt weights/PlanTalk/PlanTalk_best.bin \
--plan_vq_ckpt weights/future_plan/PlanTalk_future_plan_vq.binThe evaluation includes metrics for:
- motion distribution quality,
- gesture diversity,
- speech-motion synchronization,
- semantic relevance,
- motion reconstruction,
- semantic event prediction,
- future-plan prediction.
Representative metrics include:
FGD
BC
DIV
SRGR
MSE
LVD
Event F1
Plan Acc@1
Plan Acc@5
At inference time, PlanTalk requires speech only.
No future motion or oracle motion-plan information is used.
The inference pipeline is:
Speech
↓
Speech Representation
↓
Future Plan Prediction
↓
Event-Aware Semantic Gating
+
Plan-Conditioned Motion Generation
↓
SMPL-X Motion
Example:
python train.py \
--config configs/PlanTalk.yaml \
--inference \
--audio_infer_path demo/example.wav \
--load_ckpt weights/PlanTalk/PlanTalk_best.bin \
--plan_vq_ckpt weights/future_plan/PlanTalk_future_plan_vq.binThe main repository structure is expected to be:
PlanTalk/
├── BEAT2/
├── configs/
│ ├── PlanTalk.yaml
│ └── PlanTalk_future_plan.yaml
├── dataloaders/
├── datasets/
├── models/
│ ├── motion representations
│ ├── future motion planner
│ ├── event-aware semantic gate
│ └── PlanTalk motion generator
├── utils/
├── weights/
│ ├── pretrained_vq/
│ ├── future_plan/
│ └── PlanTalk/
├── demo/
├── train.py
└── README.md
Generated motions are represented using SMPL-X parameters.
We recommend using the official SMPL-X model together with Blender or the BEAT2 visualization tools to render generated motions.
The repository will also provide scripts for generating qualitative comparison videos.
If you find this work useful for your research, please consider citing:
@article{zhang2026planttalk,
title = {PlanTalk: Future-Grounded Motion Planning for Co-Speech Gesture Generation},
author = {Jiye Zhang, Guibiao Liao, Dingwei Liu, Gaolin Yang, Xiuhua Jiang, and Jiangbo Xu},
year = {2026}
}The BibTeX entry will be updated after publication.
Our implementation builds upon several excellent open-source projects and prior work in holistic co-speech gesture generation.
We sincerely thank the authors of BEAT2, EMAGE, SemTalk, and other related projects for releasing their datasets, models, and code.
Please follow the licenses of the corresponding datasets, SMPL-X models, pretrained speech models, and third-party repositories.
The license for the PlanTalk source code will be provided upon release.