Day 10 · Backpropagation: How a Network Learns

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ Day 9: forward pass and loss ☐ Day 4: one gradient-descent update ☐ Day 1, Math 7: the three derivative rules If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Follow backpropagation through a hidden layer with numbers you can check by hand, then put it in a full training loop and verify it with gradient checking.

You will be able to:

• Compute every number of a forward and backward pass by hand
• Write the training loop
• Train a full 2-layer network
• Check your gradients numerically
• Debug common network problems

Backpropagation With a Hidden Layer — Every Number Worked Out

The single-neuron example earlier in Days 9–10 shows the idea. What confuses most beginners is the next step: how does the error reach a weight in the first layer, when that weight never touches the output directly? Below is the smallest network that shows it, computed by hand. Every number was checked against code.

1. The tiny network

input x = [1, 2]  ──▶  hidden layer (2 neurons, ReLU)  ──▶  output (1 neuron,
    sigmoid)  ──▶  prediction a2
                         Z1 = x @ W1 + b1                      Z2 = A1 @ W2 + b2
target y = 1   ·   loss = binary cross-entropy   ·   learning rate = 0.5

2. Forward pass (left to right)

• Hidden, before activation: Z1 = x @ W1 + b1 = [1·0.5 + 2·0.2 + 0.1, 1·(−0.3) + 2·0.4 + 0.0] = [1.0, 0.5]
• ReLU keeps positives: A1 = [1.0, 0.5] (both are positive, so nothing is zeroed).
• Output, before activation: Z2 = 1.0·0.6 + 0.5·(−0.5) + 0.2 = 0.55
• Sigmoid: a2 = 1 / (1 + e −0.55 ) = 0.634 → the network says “63% likely class 1”.
• Loss: −ln(a2) = 0.455 (target is 1, so we want a2 closer to 1).

3. Backward pass (right to left) — “who is to blame?”

Read each line as a question the network asks about the mistake, starting at the output and walking backward.

4. Update and check

Update rule: parameter ← parameter − lr × gradient with lr = 0.5.

• W2 → [0.783, -0.409] b2 → 0.383
• W1 → [[0.610, -0.391], [0.420, 0.217]]
• b1 → [0.210, -0.091]
• New forward pass: a2 = 0.843 (was 0.634) and loss = 0.171 (was 0.455). One step, and the loss fell by more than half.

What to notice ① The error travels backward through the same weights that carried the signal forward (W2ᵀ). That is all “back-propagation” means. ② If a hidden neuron’s Z1 had been ≤ 0, ReLU would have blocked the blame and its weights would not change on this example (a “dead ReLU” problem when it happens for every example). ③ Shape check: each gradient has the same shape as its parameter. dW1 is (2, 2) exactly like W1.

5. Reproduce it, then break it

import numpy as np
x  = np.array([[1., 2.]]);  y = np.array([[1.]])
W1 = np.array([[.5, -.3], [.2, .4]]);  b1 = np.array([[.1, 0.]])
W2 = np.array([[.6], [-.5]]);          b2 = np.array([[.2]])
relu = lambda z: np.maximum(0, z);  sigmoid = lambda z: 1 / (1 + np.exp(-z))
Z1 = x @ W1 + b1;  A1 = relu(Z1);  Z2 = A1 @ W2 + b2;  A2 = sigmoid(Z2)   # forward
dZ2 = A2 - y;  dW2 = A1.T @ dZ2;  db2 = dZ2
# backward
dZ1 = (dZ2 @ W2.T) * (Z1 > 0);  dW1 = x.T @ dZ1;  db1 = dZ1
print(A2, dW1)          # a2 ≈ 0.634 ; dW1 ≈ [[-0.220, 0.183], [-0.439, 0.366]]

# Gradient check: nudge one weight and measure the loss change directly
def loss(W1_):
    a = sigmoid(relu(x @ W1_ + b1) @ W2 + b2);  return float(-np.log(a[0, 0]))
e = 1e-6;  Wp = W1.copy();  Wp[0, 0] += e;  Wm = W1.copy();  Wm[0, 0] -= e
print((loss(Wp) - loss(Wm)) / (2 * e), dW1[0, 0])
# the two numbers should match

Your turn (2 minutes)

Change the target to y = 0 and predict, before running anything, the sign of dZ2 and whether W2 will go up or down. Answer: dZ2 = a2 − 0 is positive, so the update pushes the output down.

8. The Training Loop

repeat for each epoch:
    1. Forward pass   → predictions
    2. Compute loss
    3. Backward pass  → gradients
    4. Update weights

These are the same 4 steps at the heart of training most neural networks.

Worked Problem · A Complete Tiny Training Loop

Problem. Train a single weight w so that y = w·x fits the data x = [1, 2, 3], y = [2, 4, 6]. Use learning rate 0.05 for 20 epochs and watch the loss.

1. Every epoch repeats the same four steps: forward pass (predictions), loss, backward pass (gradient), update.
2. Forward: pred = w·x. Loss = mean((pred − y)²).
3. Backward: gradient = mean(2 × (pred − y) × x).
4. Update: w = w − lr × gradient.
5. Record the loss each epoch and compare the first and last.

import numpy as np
x = np.array([1., 2., 3.]); y = np.array([2., 4., 6.])
w, lr, losses = 0.0, 0.05, []
for epoch in range(20):
    pred = w * x
    losses.append(((pred - y) ** 2).mean())
    grad = (2 * (pred - y) * x).mean()
    w -= lr * grad
print(losses[0] > losses[-1], abs(w - 2) < 0.1)

True True

Answer. Both checks print True: the loss shrinks and w ends close to the true value 2.

Your Turn · Order Of A Training Step

Problem. Put these in the right order: update weights, compute loss, forward pass, backward pass. Why must the gradients be reset between steps in PyTorch?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Forward pass: compute predictions from the inputs.
2. Compute the loss: compare predictions with the labels.
3. Backward pass: compute the gradient of the loss for every weight.
4. Update the weights with the optimizer.
5. PyTorch adds new gradients to the old ones, so you call optimizer.zero_grad() first in each step. Otherwise old gradients pile up.

Answer. Forward pass → loss → backward pass → update weights, with the gradients zeroed at the start of every step.

Code walkthrough: the loop above, line by line

import numpy as np
x = np.array([1., 2., 3.]); y = np.array([2., 4., 6.])
w, lr, losses = 0.0, 0.05, []
for epoch in range(20):
    pred = w * x
    losses.append(((pred - y) ** 2).mean())
    grad = (2 * (pred - y) * x).mean()
    w -= lr * grad
print(round(losses[0], 2), round(losses[-1], 4), round(w, 3))

18.67 0.0 2.0

Why that output? The first loss (18.67) is large because w started at 0. Each epoch moves w closer to 2, so the loss shrinks toward 0 and w ends near 2.

Common error: forgetting the update line w -= lr * grad. The loss then prints the same number every epoch, because w never changes. Fix: check that the weight is actually modified inside the loop.

9. ★ ASSIGNMENT — Full 2-Layer NN on Moons (complete code)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
# ---------- Data ----------
X, y = make_moons(n_samples = 1000, noise = 0.2, random_state = 42)
y = y.reshape( - 1, 1)                                 # shape (1000, 1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2,
    random_state = 42)
# ---------- Helper functions ----------
def relu(z):            return np.maximum(0, z)
def relu_grad(z):       return (z > 0).astype(float)
def sigmoid(z):         return 1 / (1 + np.exp( - z))
# ---------- Initialize ----------
np.random.seed(0)
n_in, n_hidden, n_out = 2, 16, 1
W1 = np.random.randn(n_in, n_hidden) * np.sqrt(2 / n_in)
b1 = np.zeros((1, n_hidden))
W2 = np.random.randn(n_hidden, n_out) * np.sqrt(2 / n_hidden)
b2 = np.zeros((1, n_out))
lr = 0.1
epochs = 3000
train_losses, test_losses = [], []
eps = 1e-8                                              # avoid log(0)
def forward(X):
    Z1 = X @ W1 + b1
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2
    A2 = sigmoid(Z2)
    return Z1, A1, Z2, A2
def bce(y, a):
    return - np.mean(y * np.log(a + eps) + (1 - y) * np.log(1 - a + eps))
# ---------- Training loop ----------
for epoch in range(epochs):
    # 1. forward
    Z1, A1, Z2, A2 = forward(X_train)
    # 2. loss
    loss = bce(y_train, A2)
    # 3. backward
    n = len(X_train)
    dZ2 = A2 - y_train
    dW2 = A1.T @ dZ2 / n
    db2 = dZ2.sum(axis = 0, keepdims = True) / n
    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * relu_grad(Z1)
    dW1 = X_train.T @ dZ1 / n
    db1 = dZ1.sum(axis = 0, keepdims = True) / n
    # 4. update
    W1 -= lr * dW1 ; b1 -= lr * db1
    W2 -= lr * dW2 ; b2 -= lr * db2
    # record for plots
    train_losses.append(loss)
    test_losses.append(bce(y_test, forward(X_test)[3]))
    if epoch % 500 == 0:
        print(f"epoch { epoch :4d} | train loss { loss :.4f} ")
# ---------- Evaluate ----------
def accuracy(X, y):
    pred = (forward(X)[3] > 0.5).astype(int)
    return (pred == y).mean()
print("Train acc:", accuracy(X_train, y_train))

print("Test  acc:", accuracy(X_test,  y_test))       # expect ~ 0.95+
# ---------- Plot loss ----------
plt.plot(train_losses, label = "train") ; plt.plot(test_losses, label = "test")
plt.xlabel("epoch") ; plt.ylabel("BCE loss") ; plt.legend() ; plt.show()
# ---------- Plot decision boundary ----------
xx, yy = np.meshgrid(np.linspace( - 2, 3, 300), np.linspace( - 1.5, 2, 300))
grid = np.c_[xx.ravel(), yy.ravel()]
Z = forward(grid)[3].reshape(xx.shape)
plt.contourf(xx, yy, Z, levels = 50, cmap = "RdBu", alpha = 0.6)
plt.scatter(X_test[:,0], X_test[:,1], c = y_test.ravel(), cmap = "RdBu", edgecolor
    = "k")
plt.title("Decision boundary") ; plt.show()

What you should see: loss goes down smoothly; accuracy ≈ 95–98%; decision boundary curves around the moons.

Line-by-line understanding

• make_moons — creates the dataset
• relu_grad — derivative of ReLU: 1 where z>0
• A2 - y_train — output error (nice sigmoid+BCE simplification)
• .T — transpose so shapes multiply correctly
• keepdims=True — keeps (1, n) shape so it matches the bias shape
• eps — prevents log(0) = -inf

Experiments to try (write findings in README)

1. n_hidden = 2, 4, 16, 64 — what changes?
2. lr = 0.001, 0.01, 0.1, 1.0
3. Replace ReLU with tanh
4. Remove the hidden layer (i.e. logistic regression) — accuracy drops to ~85% since the boundary can only be a straight line.

10. Gradient Checking (prove your backprop is right)

Numerical estimate of gradient: (L(w+ε) − L(w−ε)) / (2ε)

def grad_check(param, analytic, idx, eps = 1e-5):
    old = param[idx]
    param[idx] = old + eps ; lp = bce(y_train, forward(X_train)[3])
    param[idx] = old - eps ; lm = bce(y_train, forward(X_train)[3])
    param[idx] = old
    numeric = (lp - lm) / (2 * eps)
    print("analytic", analytic[idx], "numeric", numeric)
    # should match to ~6 decimals

Call it once (before updating weights) with e.g. grad_check(W1, dW1, (0,0)).

Worked Problem · Check A Gradient Numerically

Problem. For f(w) = w², the analytic gradient is 2w. Check it at w = 3 with the two-sided formula (f(w+ε) − f(w−ε)) ÷ 2ε, using ε = 0.001.

1. Analytic gradient: 2 × 3 = 6.
2. f(3.001) = 9.006001 and f(2.999) = 8.994001.
3. Difference = 0.012, divided by 2ε = 0.002, gives 6.
4. If the two numbers agree closely, your backprop code is probably correct.

f = lambda w: w ** 2
w, e = 3.0, 1e-3
print(round((f(w + e) - f(w - e)) / (2 * e), 4))

6.0

Answer. Numerical gradient 6.0, matching the analytic 6.

Your Turn · Check A Loss Gradient

Problem. For L = (w·x − y)² with x = 2, y = 1, w = 0.8, compute the analytic gradient 2(w·x − y)·x and compare it with the numerical one.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. w·x = 1.6 and the error is 1.6 − 1 = 0.6.
2. Analytic gradient = 2 × 0.6 × 2 = 2.4.
3. Compute the numerical gradient with ε = 1e-6 and compare using the relative difference.

L = lambda w: (w * 2 - 1) ** 2
w, e = 0.8, 1e-6
num = (L(w + e) - L(w - e)) / (2 * e); ana = 2 * (w * 2 - 1) * 2
print(round(ana, 4), round(num, 4), abs(num - ana) / abs(ana) < 1e-6)

2.4 2.4 True

Answer. Analytic 2.4, numerical 2.4, and the check prints True.

11. Debugging Guide

Symptom Likely cause

Loss = nan Learning rate too high; log(0) (add eps); exp overflow Loss not decreasing Wrong gradient (use gradient check); lr too small; bad init Loss decreases then stalls at 0.69 Predicting 0.5 always —

dead ReLUs / lr too small Shape mismatch Print .shape of everything; gradient shape must equal parameter shape Accuracy 50% Bug in labels shape (n,) vs (n,1) (broadcasting turns into (n,n)!)

Symptom Likely cause

Works on train, poor Overfitting → fewer on test neurons, regularize

12. Vocabulary Cheat Sheet

13. Practice Questions

1. Why can’t a network with only linear layers learn moons?
2. Compute ReLU(−3), ReLU(4). → 0, 4
3. If y=1 and ŷ=0.01, is BCE loss large or small? Why?
4. What is the shape of dW1 if X is (800,2) and the hidden layer has 16 neurons? → (2, 16)
5. Why not initialize weights to zero?
6. State the chain rule in one sentence.
7. What are the 4 steps of the training loop?

14. Quick Summary

• Neuron: a = f(w·x + b); layer: A = f(XW + b)
• Activations add non-linearity (ReLU hidden, sigmoid output)
• Loss (BCE) measures wrongness; punishes confident mistakes
• Backprop = chain rule backward; gradients have the same shape as parameters
• Training loop: forward → loss → backward → update

Next: Day 11 — PyTorch (it automates steps 1–4 for you!)

Worked Problem · The Chain Rule On One Neuron

Problem. One neuron: z = w·x + b, a = sigmoid(z), loss L = (a − y)². With x = 1, y = 1, w = 0.5, b = 0, find dL/dw and update w with learning rate 1.

1. Forward: z = 0.5, a = sigmoid(0.5) ≈ 0.6225, L = (0.6225 − 1)² ≈ 0.1425.
2. dL/da = 2(a − y) = 2 × (−0.3775) ≈ −0.755.
3. da/dz = a(1 − a) = 0.6225 × 0.3775 ≈ 0.235.
4. dz/dw = x = 1.
5. Multiply the links of the chain: dL/dw = −0.755 × 0.235 × 1 ≈ −0.1774.
6. Update: w = 0.5 − 1 × (−0.1774) = 0.6774. The weight rises, so a moves toward y = 1.
7. Verify the gradient with a tiny nudge in code.

import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))
x, y, w, b = 1.0, 1.0, 0.5, 0.0
a = sig(w * x + b)
grad = 2 * (a - y) * a * (1 - a) * x
eps = 1e-6
num = (((sig((w + eps) * x + b) - y) ** 2) - ((sig((w - eps) * x + b) - y) ** 2)) / \
    (2 * eps)
print(round(grad, 4), round(num, 4), round(w - 1 * grad, 4))

-0.1774 -0.1774 0.6774

Answer. dL/dw ≈ −0.1774, confirmed by the numerical check, and the new weight is 0.6774.

Your Turn · The Same Neuron With Y = 0

Problem. Use the same neuron and x = 1, w = 0.5, b = 0, but now the label is y = 0. Find dL/dw and update w with learning rate 0.5.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Forward: a = 0.6225 as before.
2. dL/da = 2(a − 0) = 1.245.
3. da/dz = 0.235 and dz/dw = 1, so dL/dw = 1.245 × 0.235 ≈ 0.2926.
4. Update: w = 0.5 − 0.5 × 0.2926 ≈ 0.3537.
5. The gradient is positive, so w goes down, pushing a toward 0.

import numpy as np
sig = lambda z: 1 / (1 + np.exp(-z))
w = 0.5; a = sig(w)
g = 2 * (a - 0) * a * (1 - a)
print(round(g, 4), round(w - 0.5 * g, 4))

0.2926 0.3537

Answer. dL/dw ≈ 0.2926 and the new weight is 0.3537.

⚡ Quick Recap — Days 9–10

6.1 The Neuron

A neuron = weighted sum + activation.

z = w₁x₁ + w₂x₂ + ⋯ + b and a = f(z)

Layer of neurons: Z = X @ W + b, A = f(Z).

6.2 Activation Functions

Without activations, a deep network = just one linear model. Activations add non-linearity.

Derivatives (needed for backprop):

• ReLU′ = 1 if z>0 else 0
• Sigmoid′ = a(1−a)

6.3 Forward Propagation

Data flows input → hidden → output.

X → [Z1 = X·W1 + b1] → [A1 = ReLU(Z1)] → [Z2 = A1·W2 + b2] → [A2 = sigmoid(Z2)] = prediction

6.4 Loss

Binary Cross-Entropy (for 0/1 classification):

L = −(1/n) · Σ [ y·log(ŷ) + (1 − y)·log(1 − ŷ) ]

Punishes confident wrong answers heavily. Multi-class → Categorical Cross-Entropy. Regression → MSE.

6.5 Backpropagation ★

Idea: Use the chain rule to find how much each weight contributed to the error, going backward from output to input.

Analogy: A team fails a project — trace back who contributed how much to the failure.

For 2-layer net (sigmoid + BCE, output error simplifies):

dZ2 = A2 - y                       # output error
dW2 = A1.T @ dZ2 / n
db2 = dZ2.sum(axis=0) / n
dA1 = dZ2 @ W2.T
dZ1 = dA1 * (Z1 > 0)               # ReLU derivative
dW1 = X.T @ dZ1 / n
db1 = dZ1.sum(axis=0) / n

Update: W -= lr * dW

6.6 ★ Full 2-Layer NN (Moons Dataset)

import numpy as np
from sklearn.datasets import make_moons
X, y = make_moons(1000, noise = 0.2, random_state = 42)
y = y.reshape( - 1, 1)
np.random.seed(0)
n_in, n_hidden, n_out = 2, 16, 1
W1 = np.random.randn(n_in, n_hidden) * np.sqrt(2 / n_in) # He init
b1 = np.zeros((1, n_hidden))
                                        np.sqrt(2 / n_hidden)
b2 = np.zeros((1, n_out))
relu = lambda z: np.maximum(0, z)
sigmoid = lambda z: 1 / (1 + np.exp( - z))
lr, losses = 0.1, []
for epoch in range(2000):
    # forward
    Z1 = X @ W1 + b1 ; A1 = relu(Z1)
    Z2 = A1 @ W2 + b2 ; A2 = sigmoid(Z2)
    # loss
    eps = 1e-8
    loss = - np.mean(y * np.log(A2 + eps) + (1 - y) * np.log(1 - A2 + eps))
    losses.append(loss)
    # backward
    n = len(X)
    dZ2 = A2 - y
    dW2 = A1.T @ dZ2 / n ; db2 = dZ2.mean(axis = 0, keepdims = True)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
    dW1 = X.T @ dZ1 / n ;  db1 = dZ1.mean(axis = 0, keepdims = True)
    # update
    W1 -= lr * dW1 ; b1 -= lr * db1
    W2 -= lr * dW2 ; b2 -= lr * db2
acc = ((A2 > 0.5) == y).mean()

6.7 Debugging Checklist

• Loss is nan → lr too high, or log(0) (add epsilon)
• Loss not decreasing → wrong gradient, lr too small, bad init
• Check shapes at every step (print them!)
• Start with tiny data and overfit it (sanity test)
• Gradient checking: compare with numerical gradient (L(w+ε)−L(w−ε))/2ε

6.8 Training Loop (the universal pattern)

for epoch:
    forward → loss → backward → update

This exact 4-step loop is what PyTorch automates on Day 11.

Beginner checkpoint — Days 9–10 ■ Explain the day’s main idea without using the textbook wording. ■ Write one tiny example from memory. ■ Identify one common failure mode. ■ Explain how today’s topic connects to the next day. ■ Write one question you still cannot answer.

Move on when you can… ☐ do a forward and a backward pass for one neuron by hand ☐ follow the hidden-layer worked example number by number ☐ write a training loop (forward, loss, backward, update) ☐ check a gradient numerically ☐ name what to check when the loss does not fall If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.