Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

Diffusion-Based Image Generation with Multimodal Guidance

Abstract

This report presents a systematic study on the architectural evolution, sampling mechanics, and multimodal guidance of diffusion-based generative systems evaluated on the CIFAR-10 dataset. To address the inherent trade-offs between generative quality, inference latency, and geometric structural fidelity in color image synthesis, our research proceeds across three evolutionary stages.

In Stage 1, we establish a foundational exploration of neural network backbones (UNet vs. Diffusion Transformer, DiT), sampling paradigms (DDPM, DPM-Solver, and Flow Matching), and image quality enhancement mechanisms, specifically Classifier-Free Guidance (CFG) and Edge-Aware Structural Loss (EASL).

In Stage 2, we construct a rigorous benchmark matrix comprising seven distinct model configurations (M1--M7) to systematically isolate the contributions of backbone design, sampler speed, and structural constraints. Empirical evaluations across Fréchet Inception Distance (FID), Inception Score (IS), sampling latency, and edge sharpness identify M6 (UNet backbone with Flow Matching, enhanced by CFG and EASL) as the optimal baseline architecture.

In Stage 3, we extend M6 into a multimodal framework by incorporating a pre-trained Text Encoder and Cross-Attention modules, facilitating fine-grained conditional generation controlled by continuous semantic embeddings and dynamic CFG scales. Experimental results confirm that our evolutionary pipeline significantly optimizes stroke fidelity, class stability, and cross-modal alignment.

Key Features

  • Stage 2 Benchmark (M1-M7): Complete training and evaluation scripts for comparing UNet/DiT backbones and DDPM/DPM-Solver/Flow Matching samplers.
  • Stage 3 Multimodal Fine-tuning: Cross-Attention integration and CLIP Text Encoder conditional sampling for the optimal M6 baseline.
  • Evaluation Suite: Integrated scripts for calculating FID, IS, and Edge Sharpness.

Requirements

  • Python 3.10+
  • PyTorch (Configured with CUDA for optimal training throughput on high-end hardware like the RTX 5070)
  • Torchvision

Repository Structure

Please refer to the stage2_benchmark/ and stage3_multimodal/ directories for detailed model implementations, training loops, and sampling scripts.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages