Week 10: Mixture of Experts

Sparse models that scale to trillions of parameters.

Overview

In an MoE, different parts of the network specialize. For each token, a Gating Network chooses the Top-K experts to process it.

Theory & Concepts

Formula: y = Sum( Gate(x)_i * Expert_i(x) )

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Token Routing

Watch how tokens are routed to specific experts based on their color.

💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
gates = self.gate(x)
top_k_weights, top_k_indices = torch.topk(gates, k=2)

Weekly Project

MoE Layer

Implement a sparse MoE layer from scratch.

Step-by-Step Implementation

  • Gating Net.
  • Sparse Dispatch.
  • Load Balancing Loss.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.