This repository provides the official implementation of Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration, accepted to ICML 2026. Polyphonia is a zero-shot music editing framework for stem-specific timbre transfer in polyphonic mixtures. Instead of relying only on semantic cross-attention, which often lacks sufficient spectral resolution in dense music mixtures, Polyphonia introduces Acoustic-Informed Attention Calibration to inject probabilistic acoustic priors into the diffusion editing process. This enables precise target-stem timbre transfer while preserving non-target accompaniment and overall musical coherence.
🎧 Demo page: https://polyphonia2026.github.io/polyphonia-demo/
We recommend using a clean Conda environment.
git clone https://github.com/Ltx-Leif/polyphonia.git
cd polyphonia
conda create -n polyphonia python=3.8.20 -y
conda activate polyphonia
pip install -r requirements.txtThe provided requirements.txt pins the main dependencies, including:
torch==2.4.1+cu124
torchaudio==2.4.1+cu124
torchvision==0.19.1+cu124
diffusers==0.32.0
transformers==4.46.3
The current dependency file is configured for CUDA 12.4 PyTorch wheels. If your CUDA version or hardware environment differs, please install the PyTorch build that matches your system first, and then adjust the PyTorch-related entries in requirements.txt accordingly.
polyphonia/
├── PolyEvalPrompts/
│ ├── musdb18_test/
│ └── musicdelta/
├── sample/
├── run_sample_exp.py # Inference script for single-sample editing
├── edit.py # Core Polyphonia editor and attention calibration mechanisms
├── my_audioldm2.py # AudioLDM2 loading and attention-layer access
├── audio_utils.py # Audio loading and acoustic mask generation
├── utils.py # Attention calibration utilities and visualization helpers
├── requirements.txt
└── README.md
We release PolyEvalPrompts, a standardized benchmark prompt set designed for evaluating stem-specific timbre transfer in polyphonic music. Existing music-editing prompt sets are often too coarse-grained for this task, where the model must modify a specific target stem while preserving the accompaniment and all non-target stems. PolyEvalPrompts is constructed to provide semantically rich source descriptions and systematically defined editing tasks for dense musical mixtures.
The benchmark is organized as follows:
PolyEvalPrompts/
├── musdb18_test/ # Prompt entries for the MUSDB18-HQ test subset
└── musicdelta/ # Prompt entries for the MusicDelta dataset
PolyEvalPrompts is built through a two-stage pipeline:
-
Acoustic Analysis
We use Qwen-Audio to analyze the acoustic content of each mixture and generate comprehensive metadata, including instrumentation, playing techniques, and overall musical style. This step ensures that the source prompt is semantically rich and acoustically grounded. -
Task Synthesis
Conditioned on the extracted metadata, we use Qwen3 to synthesize 15 distinct editing tasks for each musical piece. These tasks span a wide range of timbre-transfer scenarios, from subtle instrument substitutions to more radical timbre-style transformations.
Each benchmark entry contains a source description and multiple edit prompts. A representative format is shown below:
Music Sample: [Zeppelin]
Source Prompt:
A recording of a distorted electric guitar melody, a solid bass line and driving energetic drums.
Edit Prompt 1 (Guitar → Violin):
A recording of a fast expressive violin, a solid bass line and driving energetic drums.
Edit Prompt 2 (Guitar → Cello):
A recording of a powerful cello, a solid bass line and driving energetic drums.
For evaluation, PolyEvalPrompts is applied to the MUSDB18-HQ test subset and MusicDelta, resulting in 1,170 stem-specific timbre transfer tasks. In the paper, Polyphonia is benchmarked against representative zero-shot and pretrained music editing paradigms, including global inversion methods, structural steering methods, and pretrained autoregressive models.
Note: This repository provides the PolyEvalPrompts prompt files. The corresponding audio data should be prepared separately according to the licenses and usage terms of MUSDB18-HQ and MusicDelta.
The default inference script expects a sample directory with the following structure:
sample/
├── mixture.wav
├── vocals.wav
├── bass.wav
├── drums.wav
├── other.wav
└── prompt_multi.json
mixture.wavis required.- Stem files such as
vocals.wav,bass.wav,drums.wav, andother.wavare used to construct acoustic soft masks. prompt_multi.jsondefines one or more editing tasks.
A prompt_multi.json example:
{
"vocals2violin": {
"Original Prompt": "A recording of soulful male vocals, deep bass, atmospheric synth and punchy drums.",
"Baseline Prompt": "A recording of a smooth violin, deep bass, atmospheric synth and punchy drums.",
"Vocals Softmask Prompt": "a smooth violin",
"Bass Softmask Prompt": "deep bass",
"Drums Softmask Prompt": "punchy drums",
"Others Softmask Prompt": "atmospheric synth"
}
}The task name is used to infer the edited stem. For example:
vocals2violinedits thevocalsstem.drums2percussionedits thedrumsstem.bass2synthedits thebassstem.- Other task names default to the
otherstem.
python run_sample_exp.py \
--sample_dir ./sample \
--output_dir ./sample/output \
--device cuda \
--precision fp16 \
--seed 928 \
--visualizeThis command will:
- Load
./sample/mixture.wav. - Parse all editing tasks from
./sample/prompt_multi.json. - Generate acoustic soft masks from the available stem files.
- Perform AudioLDM2 inversion once for the input mixture.
- Run Polyphonia editing for each task.
- Save edited audio files to
./sample/output/. - Save spectrogram visualizations when
--visualizeis enabled.
Expected outputs:
sample/output/
├── vocals2violin.wav
└── vocals2violin.png # only generated when --visualize is enabled
python run_sample_exp.py \
--sample_dir ./sample \
--output_dir ./sample/output \
--task_key vocals2violin \
--device cuda \
--precision fp16 \
--seed 928 \
--mask_type energy \
--visualizeOnly the task named vocals2violin in prompt_multi.json will be executed.
If you find this repository useful for your research, please cite our paper:
@misc{li2026polyphoniazeroshottimbretransfer,
title={Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration},
author={Haowen Li and Tianxiang Li and Yi Yang and Boyu Cao and Qi Liu},
year={2026},
eprint={2605.10203},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2605.10203},
}This project builds upon several excellent open-source projects and pretrained models. In particular, we thank the authors and maintainers of AudioLDM2, Diffusers, Transformers, PyTorch, Librosa, and related audio-generation toolkits for making their code and models publicly available.
We also thank the broader music generation and audio editing research community for providing valuable open-source implementations and evaluation resources that have supported reproducible research in zero-shot music editing.