Week 1: PyTorch Foundations

Tensors, Autograd, and Neural Networks.

Overview

PyTorch is a powerful deep learning framework that provides Tensors (multi-dimensional arrays with GPU acceleration), Autograd (automatic differentiation), and Dynamic Computation Graphs.

In this week, we build neural networks from first principles, avoiding high-level abstractions like nn.Linear until we understand the math underneath.

Theory & Concepts

Sensors & Operations

Tensors are the fundamental building blocks. They are similar to NumPy arrays but can run on GPUs.

x = torch.tensor([[1, 2], [3, 4]])

Autograd: The Engine of Deep Learning

PyTorch automatically tracks operations on tensors to build a computational graph. When you call .backward(), it computes gradients using the Chain Rule.

The Chain Rule

∂y/∂x = (∂y/∂h) * (∂h/∂x)

Gradients flow backwards from the output loss to the input parameters.

Concept Check

Make sure you grasp the core equations before moving to code.

Interactive Lab: Computational Graph

A visualization of how data flows forward (blue) and gradients flow backward (red).

x Input x² Op +3 y
💡 Experiment: Try different values to see how the system reacts in real-time.

Code Implementation

Core Logic

Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.

Python
class ManualLinear(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        # Xavier initialization
        self.weight = nn.Parameter(
            torch.randn(out_features, in_features) * (2.0 / in_features) ** 0.5
        )
        self.bias = nn.Parameter(torch.zeros(out_features))
    
    def forward(self, x):
        # x: (batch_size, in_features)
        # weight: (out_features, in_features)
        return x @ self.weight.T + self.bias

Weekly Project

MLP from Scratch

Build a Multi-Layer Perceptron to classify MNIST digits without using nn.Linear.

Step-by-Step Implementation

  • Implement ManualLinear using nn.Parameter.
  • Build an Architecture: 784 → 256 → 128 → 10.
  • Implement the Training Loop with optimizer.zero_grad() and loss.backward().
  • Achieve >95% accuracy on MNIST.
🚀 Challenge: Can you optimize this further? Check the exercises.py file for bonus tasks.