Week 1: PyTorch Foundations
Tensors, Autograd, and Neural Networks.
Overview
PyTorch is a powerful deep learning framework that provides Tensors (multi-dimensional arrays with GPU acceleration), Autograd (automatic differentiation), and Dynamic Computation Graphs.
In this week, we build neural networks from first principles, avoiding high-level abstractions like nn.Linear until we understand the math underneath.
Theory & Concepts
Sensors & Operations
Tensors are the fundamental building blocks. They are similar to NumPy arrays but can run on GPUs.
Autograd: The Engine of Deep Learning
PyTorch automatically tracks operations on tensors to build a computational graph. When you call .backward(), it computes gradients using the Chain Rule.
The Chain Rule
∂y/∂x = (∂y/∂h) * (∂h/∂x)
Gradients flow backwards from the output loss to the input parameters.
Concept Check
Make sure you grasp the core equations before moving to code.
Interactive Lab: Computational Graph
A visualization of how data flows forward (blue) and gradients flow backward (red).
Code Implementation
Core Logic
Below is the critical implementation from project.py. Analyze how the theory transforms into PyTorch code.
class ManualLinear(nn.Module):
def __init__(self, in_features, out_features):
super().__init__()
# Xavier initialization
self.weight = nn.Parameter(
torch.randn(out_features, in_features) * (2.0 / in_features) ** 0.5
)
self.bias = nn.Parameter(torch.zeros(out_features))
def forward(self, x):
# x: (batch_size, in_features)
# weight: (out_features, in_features)
return x @ self.weight.T + self.bias
Weekly Project
MLP from Scratch
Build a Multi-Layer Perceptron to classify MNIST digits without using nn.Linear.
Step-by-Step Implementation
- Implement
ManualLinearusingnn.Parameter. - Build an Architecture: 784 → 256 → 128 → 10.
- Implement the Training Loop with
optimizer.zero_grad()andloss.backward(). - Achieve >95% accuracy on MNIST.
exercises.py file for bonus tasks.