Week 4 Capstone: The Router
Building the brain of a Mixture-of-Experts (MoE) system.
Overview
A Router is a classifier that decides which model should handle a given query. In this capstone, you will build a system that can distinguish between Chat, Coding, and Math queries, dispatching them to simulated expert models.
Theory & Concepts
Embeddings for Classification
We use a pre-trained BERT model to extract semantic features from the input text.
The [CLS] token output (vector size 768) serves as a summary of the entire sentence.
The Routing Layer
A simple linear layer projects the 768-dim vector to N_EXPERTS logits.
We train this using Cross Entropy Loss on a synthetic dataset of queries.
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Live Router
Type a query. The system will categorize it and route it to the correct simulated expert.
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
class Router(nn.Module):
def __init__(self, num_experts=3):
super().__init__()
self.bert = AutoModel.from_pretrained('prajjwal1/bert-tiny')
self.classifier = nn.Linear(128, num_experts) # TinyBERT dim is 128
def forward(self, input_ids, attention_mask):
outputs = self.bert(input_ids, attention_mask=attention_mask)
# Use [CLS] token embedding
cls_emb = outputs.last_hidden_state[:, 0, :]
logits = self.classifier(cls_emb)
return logits
def route(self, text):
inputs = tokenizer(text, return_tensors='pt')
logits = self(inputs['input_ids'], inputs['attention_mask'])
expert_id = torch.argmax(logits, dim=-1)
return expert_id
Weekly Project
Build the Router
Train a BERT-based classifier on a custom dataset.
Step-by-Step Implementation
- Generate a synthetic dataset (queries labeled 0, 1, 2).
- Define the
Routerclass using TinyBERT. - Train for 5 epochs using AdamW.
- Implement the
dispatch()function that calls fake experts.
exercises.py file for bonus tasks.