Week 4 Capstone: The Router

Building the brain of a Mixture-of-Experts (MoE) system.

Overview

A Router is a classifier that decides which model should handle a given query. In this capstone, you will build a system that can distinguish between Chat, Coding, and Math queries, dispatching them to simulated expert models.

Theory & Concepts

Embeddings for Classification

We use a pre-trained BERT model to extract semantic features from the input text.

The [CLS] token output (vector size 768) serves as a summary of the entire sentence.

The Routing Layer

A simple linear layer projects the 768-dim vector to N_EXPERTS logits.

logits = Linear(768, 3) # [Chat, Code, Math]

We train this using Cross Entropy Loss on a synthetic dataset of queries.

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Live Router

Type a query. The system will categorize it and route it to the correct simulated expert.

Other
Code
Math
💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
class Router(nn.Module):
    def __init__(self, num_experts=3):
        super().__init__()
        self.bert = AutoModel.from_pretrained('prajjwal1/bert-tiny')
        self.classifier = nn.Linear(128, num_experts) # TinyBERT dim is 128
        
    def forward(self, input_ids, attention_mask):
        outputs = self.bert(input_ids, attention_mask=attention_mask)
        # Use [CLS] token embedding
        cls_emb = outputs.last_hidden_state[:, 0, :]
        logits = self.classifier(cls_emb)
        return logits
        
    def route(self, text):
        inputs = tokenizer(text, return_tensors='pt')
        logits = self(inputs['input_ids'], inputs['attention_mask'])
        expert_id = torch.argmax(logits, dim=-1)
        return expert_id

Weekly Project

Build the Router

Train a BERT-based classifier on a custom dataset.

Step-by-Step Implementation

  • Generate a synthetic dataset (queries labeled 0, 1, 2).
  • Define the Router class using TinyBERT.
  • Train for 5 epochs using AdamW.
  • Implement the dispatch() function that calls fake experts.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.