Week 10: Mixture of Experts
Sparse models that scale to trillions of parameters.
Overview
In an MoE, different parts of the network specialize. For each token, a Gating Network chooses the Top-K experts to process it.
Theory & Concepts
Formula: y = Sum( Gate(x)_i * Expert_i(x) )
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Token Routing
Watch how tokens are routed to specific experts based on their color.
💡 Experiment: Try different values to see how the system reacts in real-time.
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
Python
gates = self.gate(x)
top_k_weights, top_k_indices = torch.topk(gates, k=2)
Weekly Project
MoE Layer
Implement a sparse MoE layer from scratch.
Step-by-Step Implementation
- Gating Net.
- Sparse Dispatch.
- Load Balancing Loss.
🚀 Challenge: Can you optimize this further? Check the
exercises.py file for bonus tasks.