Week 5: Diffusion Math

From Noise to Clarity: Understanding the Forward and Reverse Processes.

Overview

Diffusion models (like DALL-E 2 and Stable Diffusion) work by destroying data with noise and learning to repair it. This week covers the fundamental probability theory and the Reparameterization Trick.

Theory & Concepts

The Forward Process (q)

We gradually add Gaussian noise. The amount of noise at step t is controlled by a variance schedule β_t.

x_t = √(1 - β_t) * x_{t-1} + √(β_t) * ε

Crucially, we can jump to step t in one go:

x_t = √(α_bar) * x_0 + √(1 - α_bar) * ε

The Reverse Process (p)

We train a neural network to predict the noise ε that was added. If we can subtract the noise, we recover the image.

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Noise Schedule

Slide t from 0 to 1000 to see how the signal is destroyed.

💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
def q_sample(self, x_start, t, noise=None):
    if noise is None:
        noise = torch.randn_like(x_start)
        
    sqrt_alpha_cumprod_t = self.sqrt_alphas_cumprod[t]
    sqrt_one_minus_alpha_cumprod_t = self.sqrt_one_minus_alphas_cumprod[t]
    
    return sqrt_alpha_cumprod_t * x_start +            sqrt_one_minus_alpha_cumprod_t * noise

Weekly Project

Diffusion Scheduler

Implement the noise schedules and samplers.

Step-by-Step Implementation

  • Implement linear beta schedule.
  • Pre-compute alpha bars.
  • Create the q_sample function (Forward process).
  • Visualize the degradation of an image over timesteps.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.