Week 7: Latent Diffusion

Stable Diffusion: Making it fast by working in compressed space.

Overview

Pixel-space diffusion is slow. Latent Diffusion Models (LDMs) use a Variational Autoencoder (VAE) to compress images such that a 512x512 image becomes a 64x64 latent tensor. We then diffuse this small tensor.

Theory & Concepts

Perceptual Compression

The VAE removes high-frequency details that aren't perceptually important, while keeping the semantic content.

Conditioning

How do we guide generation? We use Cross-Attention in the U-Net. Text embeddings (from CLIP) are injected into the U-Net layers to steer the denoising.

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Latent Compression

Slide to compress the image into Latent Space and back. Notice the loss of detail.

➡️

Original (Pixel Space)

💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
# Training Loop Sketch
# 1. Encode image to latent
z = vae.encode(image).latent_dist.sample() * 0.18215

# 2. Add noise
noise = torch.randn_like(z)
z_noisy = scheduler.add_noise(z, noise, t)

# 3. Predict noise (conditioned on text)
pred_noise = unet(z_noisy, t, encoder_hidden_states=prompts)

# 4. Loss
loss = F.mse_loss(pred_noise, noise)

Weekly Project

Latent Pipeline Mockup

Simulate the LDM pipeline using mock components.

Step-by-Step Implementation

  • Create the `LatentDiffusion` class.
  • Mock the VAE encoder/decoder.
  • Mock the CLIP text encoder.
  • Trace the tensor shapes through the entire pipeline.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.