Day 12 · MNIST, Dropout, Early Stopping, CNN

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ Day 11: tensors, nn.Module, training loop ☐ Day 5: overfitting and early stopping ☐ Day 2: reshaping If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Train a network on real images (handwritten digits), learn two anti-overfitting tools (dropout, early stopping), then meet the CNN — the network designed for images.

Big picture — read this first A CNN learns local visual patterns through filters. Early layers often capture simple patterns; deeper layers combine them into more complex features. Padding controls spatial size, stride controls movement, and pooling reduces spatial dimensions. Dropout and early stopping are different regularization strategies: one changes training behavior, the other stops training when validation performance stops improving. Transfer learning reuses representations learned on a larger dataset and adapts them to a new task.

1. The MNIST dataset

• 70,000 grayscale images of handwritten digits 0–9
• Each image = 28 × 28 pixels; each pixel = brightness from 0 (black) to 255 (white)
• 60,000 train + 10,000 test
• Task: 10-class classification

An image is just a grid of numbers:
  0   0   0  ...  0
  0  12 180 255 ...
  ...

So image shape = (1, 28, 28) → (channels, height, width). Grayscale has 1 channel; color (RGB) has 3.

Batch shape in PyTorch: (B, C, H, W) = (batch, channels, height, width) → e.g. (64, 1, 28, 28).

import torch, torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, random_split
tf = transforms.Compose([
    transforms.ToTensor(),                        # 0-255 → 0.0-1.0, shape (1,28,28)
    transforms.Normalize((0.1307,), (0.3081,))
    # subtract mean, divide std (MNIST's known values)
])
full_train = datasets.MNIST("data", train = True, download = True, transform = tf)
test_set   = datasets.MNIST("data", train = False, download = True, transform = tf)

train_set, val_set = random_split(full_train, [55000, 5000],
                                  generator = torch.Generator().manual_seed(42))
train_loader = DataLoader(train_set, batch_size = 64, shuffle = True)
val_loader   = DataLoader(val_set,   batch_size = 256)
test_loader  = DataLoader(test_set,  batch_size = 256)

Load it Why normalize? Inputs centered around 0 with similar scale → gradient descent trains faster and more stably.

Worked Problem · Size Of Mnist

Problem. MNIST has 60,000 training images of 28 × 28 pixels. How many numbers is that, and how much memory do they need as 32-bit floats?

1. One image: 28 × 28 = 784 pixels.
2. All images: 60,000 × 784 = 47,040,000 numbers.
3. A 32-bit float takes 4 bytes: 47,040,000 × 4 = 188,160,000 bytes ≈ 179 MB.

n = 60000 * 28 * 28
print(n, n * 4, round(n * 4 / 1024 ** 2, 1))

47040000 188160000 179.4

Answer. 47,040,000 numbers, about 179 MB as floats.

2. MLP — Multi-Layer Perceptron

An MLP is a network of fully connected (Linear) layers — what you built on Days 9–10/7.

For images: flatten first (1, 28, 28) → flatten → 784 numbers → Linear layers.

28×28 image → Flatten → 784 → Linear(784→256) → ReLU
                              → Linear(256→128) → ReLU
                              → Linear(128→10)  → 10 raw scores (logits)

Output: 10 numbers, one per digit. Highest = predicted digit. Loss: CrossEntropyLoss (contains softmax; labels are integers 0–9).

class MLP(nn.Module):
    def __init__ (self, p_drop = 0.3):
        super(). __init__ ()
        self.net = nn.Sequential(
            nn.Flatten(),                      # (B,1,28,28) → (B,784)
            nn.Linear(784, 256), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(256, 128), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(128, 10),
        )
    def forward(self, x): return self.net(x)

Parameters: 784×256+256 + 256×128+128 + 128×10+10 ≈ 235k.

Weakness of MLP for images Flattening throws away the 2D structure: the network doesn’t know pixel (5,5) is next to pixel (5,6). It also needs a separate weight for every pixel connection.

Softmax (for multi-class) Turns 10 raw scores into probabilities that sum to 1:

softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ e.g. [2, 1, 0] → [0.665, 0.245, 0.090]

CrossEntropyLoss = softmax + negative log of the probability given to the correct class. Get probabilities yourself: torch.softmax(logits, dim=1); predicted class: logits.argmax(dim=1).

Worked Problem · Count An Mlp'S Parameters

Problem. An MLP maps 784 → 128 → 64 → 10. How many learnable parameters does it have?

1. Each layer has in × out weights plus out biases.
2. 784 → 128: 784 × 128 + 128 = 100,480.
3. 128 → 64: 128 × 64 + 64 = 8,256.
4. 64 → 10: 64 × 10 + 10 = 650.
5. Add them: 100,480 + 8,256 + 650 = 109,386.

lin = lambda i, o: i * o + o
print(lin(784, 128) + lin(128, 64) + lin(64, 10))

109386

Answer. 109,386 parameters.

3. Dropout

Problem Networks can co-adapt: neurons rely on specific other neurons → memorize training data → overfit.

Idea During training, randomly switch off a fraction p of neurons in each forward pass (set output to 0).

Training pass 1:  ● ○ ● ● ○      (○ = dropped)
Training pass 2:  ○ ● ● ○ ●

• Forces every neuron to be useful on its own — like training a team where random members are absent, so nobody can slack or depend on one star player
• It’s like training many smaller networks and averaging them

nn.Dropout(p = 0.3)    # 30% dropped

Practical points

• Typical p: 0.2–0.5 (higher for big layers)
• Only active in model.train(); in model.eval() it does nothing (all neurons used). ← Forgetting eval() = noisy/wrong validation
• Place after the activation of hidden layers, not on the output layer

How these problems were checked. The numbers in the worked problems below were calculated and verified with NumPy when this book was built. The PyTorch calls named in the steps do the same math. The PyTorch code in the lessons was not run in the build environment, so run it in your own notebook.

Worked Problem · Inverted Dropout

Problem. Activations are [1, 2, 3, 4], dropout probability 0.5, and the random mask keeps [1, 0, 1, 0]. Apply inverted dropout.

1. Multiply by the mask: [1, 0, 3, 0]. Dropped units become 0.
2. Scale the survivors by 1 ÷ (1 − p) = 2 so the expected total stays the same: [2, 0, 6, 0].
3. At test time dropout is off (model.eval()), so no mask and no scaling is applied.

import numpy as np
x = np.array([1., 2., 3., 4.]); mask = np.array([1, 0, 1, 0])
print(x * mask / (1 - 0.5))

[2. 0. 6. 0.]

Answer. [2. 0. 6. 0.].

Your Turn · A Different Keep Probability

Problem. With dropout p = 0.2 the survivors are scaled by what factor? Apply it to survivors [4, 8].

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Keep probability = 1 − 0.2 = 0.8.
2. Scale factor = 1 ÷ 0.8 = 1.25.
3. 4 × 1.25 = 5 and 8 × 1.25 = 10.

print([v / (1 - 0.2) for v in (4, 8)])

[5.0, 10.0]

Answer. Factor 1.25, giving [5.0, 10.0].

4. Early Stopping

Problem With more epochs, training loss keeps falling, but validation loss eventually starts rising — overfitting has begun.

loss
 |\
 | \  train ↘↘↘↘↘↘↘↘

|  \_
|    \___   val ↘ ↗↗↗   ← overfitting begins here
|        ‾‾‾‾‾/
+--------------|--------- epochs
         STOP HERE (best val loss)

Idea Track validation loss. If it hasn’t improved for patience epochs, stop and restore the best weights.

class EarlyStopping:
    def __init__ (self, patience = 3, min_delta = 0.0):
        self.patience, self.min_delta = patience, min_delta
        self.best, self.wait = float("inf"), 0
        self.best_state = None
    def step(self, val_loss, model):
        if val_loss < self.best - self.min_delta:
            self.best, self.wait = val_loss, 0
            self.best_state = {k: v.clone() for k, v in model.state_dict().items()}
            return False                       # keep training
        self.wait += 1
        return self.wait >= self.patience      # True → stop

Use: after each epoch if es.step(val_loss, model): break; at the end model.load_state_dict(es.best_state).

Worked Problem · Run Early Stopping By Hand

Problem. Validation losses by epoch are [0.9, 0.7, 0.6, 0.62, 0.63, 0.64, 0.5]. With patience 3, when does training stop and which epoch's weights do you keep?

1. Track the best loss and how many epochs have passed without a new best.
2. Epoch 0: 0.9 (best). Epoch 1: 0.7 (best). Epoch 2: 0.6 (best).
3. Epochs 3, 4, 5 (0.62, 0.63, 0.64) are all worse than 0.6, so the counter reaches 3.
4. Training stops at epoch 5 and we restore the weights from epoch 2. The later 0.5 is never seen.
5. Simulate it.

losses = [0.9, 0.7, 0.6, 0.62, 0.63, 0.64, 0.5]
best, best_ep, wait = 1e9, -1, 0
for ep, l in enumerate(losses):
    if l < best: best, best_ep, wait = l, ep, 0
    else: wait += 1
    if wait == 3:
        print("stop at", ep, "keep epoch", best_ep); break

stop at 5 keep epoch 2

Answer. Stop at epoch 5 and keep epoch 2. A longer patience would have found the 0.5, which is the trade-off.

Your Turn · Patience 2

Problem. Validation losses are [1.0, 0.8, 0.85, 0.7, 0.72, 0.75] and the patience is 2. When does it stop and what is the best epoch?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Epoch 1: 0.8 is the best. Epoch 2: 0.85 is worse (wait 1).
2. Epoch 3: 0.7 is a new best, so the counter resets to 0.
3. Epoch 4: 0.72 (wait 1). Epoch 5: 0.75 (wait 2). Patience is used up, so training stops at epoch 5.
4. The best epoch is 3.

losses = [1.0, 0.8, 0.85, 0.7, 0.72, 0.75]
best, best_ep, wait = 1e9, -1, 0
for ep, l in enumerate(losses):
    if l < best: best, best_ep, wait = l, ep, 0
    else: wait += 1
    if wait == 2:
        print("stop at", ep, "keep epoch", best_ep); break

stop at 5 keep epoch 3

Answer. Stop at epoch 5, keep epoch 3.

5. Reading Training Curves

Worked Problem · Spot Where Overfitting Starts

Problem. Training losses are [0.9, 0.5, 0.3, 0.2] and validation losses are [0.9, 0.55, 0.5, 0.6]. At which epoch should you stop?

1. Training loss keeps falling, which alone is not proof of a good model.
2. Validation loss falls until epoch 2 (0.5) and then rises to 0.6.
3. The best epoch is the one with the lowest validation loss.
4. After it, the model is memorizing the training data.

import numpy as np
val = [0.9, 0.55, 0.5, 0.6]
print(int(np.argmin(val)))

2

Answer. Epoch 2 (counting from 0). Stop there, or restore those weights with early stopping.

6. Complete training function (reusable for MLP and CNN)

device = "cuda" if torch.cuda.is_available() else "cpu"
def evaluate(model, loader, criterion):
    model.eval() ; loss = correct = n = 0
    with torch.no_grad():
        for xb, yb in loader:
            xb, yb = xb.to(device), yb.to(device)
            out = model(xb)
            loss += criterion(out, yb).item() * len(xb)
            correct += (out.argmax(1) == yb).sum().item()
            n += len(xb)
    return loss / n, correct / n
def fit(model, epochs = 30, lr = 1e-3, patience = 3):
    model.to(device)
    criterion = nn.CrossEntropyLoss()
    opt = torch.optim.Adam(model.parameters(), lr = lr)
    es = EarlyStopping(patience)
    hist = {"tr_loss": [], "va_loss": [], "va_acc": []}
    for ep in range(epochs):
        model.train() ; tl = n = 0
        for xb, yb in train_loader:
            xb, yb = xb.to(device), yb.to(device)
            opt.zero_grad()
            loss = criterion(model(xb), yb)
            loss.backward() ; opt.step()
            tl += loss.item() * len(xb) ; n += len(xb)
        va_loss, va_acc = evaluate(model, val_loader, criterion)
        hist["tr_loss"].append(tl / n) ; hist["va_loss"].append(va_loss) ; \
            hist["va_acc"].append(va_acc)
        print(f"ep { ep + 1 :2d} | train { tl / n :.4f} | val { va_loss :.4f} | val "
          f"acc { va_acc :.4f} ")
        if es.step(va_loss, model): print("Early stop") ; break
    model.load_state_dict(es.best_state)
    return hist
mlp = MLP()
hist_mlp = fit(mlp)
print("MLP test:", evaluate(mlp, test_loader, nn.CrossEntropyLoss()))
# ≈ (0.07, 0.978)

PART B — CNN (Convolutional Neural Networks)

(Suggested: about 45 minutes for the CNN. Focus on the idea + the small model below.)

7. Why CNNs?

Images have structure: 1. Local patterns: edges, corners, curves are small local things 2. Same pattern can appear anywhere: a “loop” in a 6 can be at any position

CNNs exploit both, small shared filters and keeping the 2D layout.

Worked Problem · Weight Sharing Saves Parameters

Problem. Compare the parameters of a dense layer from a 28 × 28 image to 128 units with a convolutional layer of 16 filters of size 3 × 3 on one channel.

1. Dense: every one of 784 inputs connects to each of 128 units: 784 × 128 + 128 = 100,480.
2. Convolution: each filter has 3 × 3 = 9 weights plus 1 bias, reused at every position: 16 × 9 + 16 = 160.
3. The convolution is about 600 times smaller, and it also respects the 2-D layout.

print(784 * 128 + 128, 16 * 9 + 16, round((784 * 128 + 128) / 160))

100480 160 628

Answer. 100,480 versus 160 parameters, about 628 times fewer.

8. Convolution — the core operation

A filter (kernel) is a small grid of numbers (e.g. 3×3). Slide it across the image; at each position, multiply overlapping numbers and add them up → one output number.

Example (1D-ish intuition with 3×3 on a tiny image) Vertical-edge filter:

[-1 0 1]
[-1 0 1]
[-1 0 1]

Where the image goes from dark (left) to bright (right), the output is large → “there’s a vertical edge here”.

Image patch        Filter          Output cell
[0 0 9]           [-1 0 1]
[0 0 9]     ⊙     [-1 0 1]   =  (0*-1 + 0*0 + 9*1)*3 = 27   ← edge detected!
[0 0 9]           [-1 0 1]

The filter values are LEARNED by backprop — the network discovers useful filters itself.

Key terms

Output size formula

out = floor((W − K + 2P) / S) + 1

W=28, K=3, P=1, S=1 → (28 − 3 + 2)/1 + 1 = 28 (same size with padding=1) W=28, K=3, P=0, S=1 → 26

Layers stack: simple → complex

• Layer 1 filters: edges, lines
• Layer 2: corners, curves (combos of edges)
• Deeper: parts of digits, then whole digits

Parameters comparison

• Conv2d(1 → 16, kernel 3): 16×(1×3×3)+16 = 160 parameters — a Linear layer on 784 inputs to 16 neurons would need 12,560.

Worked Problem · One Convolution By Hand

Problem. Image [[1, 2, 3], [4, 5, 6], [7, 8, 9]] and a 2 × 2 filter [[1, 0], [0, -1]]. Slide the filter with stride 1 and no padding.

1. The output size is (3 − 2 + 1) = 2, so the result is 2 × 2.
2. At each position multiply the 2 × 2 patch by the filter and add everything up.
3. Top-left patch [[1,2],[4,5]]: 1×1 + 2×0 + 4×0 + 5×(−1) = 1 − 5 = −4.
4. Top-right patch [[2,3],[5,6]]: 2 − 6 = −4. Bottom-left [[4,5],[7,8]]: 4 − 8 = −4. Bottom-right [[5,6],[8,9]]: 5 − 9 = −4.
5. Check with a loop.

import numpy as np
img = np.arange(1, 10).reshape(3, 3); k = np.array([[1, 0], [0, -1]])
out = np.array([[(img[i:i+2, j:j+2] * k).sum() for j in range(2)] for i in
    range(2)])
print(out)

[[-4 -4]
 [-4 -4]]

Answer. Every output value is −4: this filter measures the difference along a diagonal, and the image rises steadily.

Your Turn · Output Size Formula

Problem. Use output = floor((W − K + 2P) ÷ S) + 1 for a 28-pixel-wide image with (a) K=3, P=1, S=1, (b) K=5, P=0, S=1, (c) K=3, P=0, S=2.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. (a) (28 − 3 + 2) ÷ 1 + 1 = 28. Padding 1 with a 3×3 kernel keeps the size.
2. (b) (28 − 5 + 0) ÷ 1 + 1 = 24.
3. (c) (28 − 3 + 0) ÷ 2 + 1 = 12.5 → floor 12, then + 1 = 13.

import math
f = lambda W, K, P, S: math.floor((W - K + 2 * P) / S) + 1
print(f(28, 3, 1, 1), f(28, 5, 0, 1), f(28, 3, 0, 2))

28 24 13

Answer. 28, 24, 13.

9. Pooling

Shrinks feature maps: MaxPool2d(2) takes the maximum in each 2×2 block → halves height and width.

[1 3 | 2 1]                   [4 | 2]
[4 2 | 0 1]   MaxPool 2×2 →   [4 | 3]
-----+-----
[0 1 | 3 2]
[2 4 | 1 1]

Benefits: fewer computations, keeps the strongest signal, small shift tolerance (digit slightly moved is still recognized).

Worked Problem · 2 × 2 Max Pooling

Problem. Max-pool this 4 × 4 image with a 2 × 2 window and stride 2: [[1,3,2,1],[4,6,5,2],[7,2,1,0],[3,8,4,9]].

1. Split the image into four 2 × 2 blocks.
2. Top-left [[1,3],[4,6]] → max 6. Top-right [[2,1],[5,2]] → max 5.
3. Bottom-left [[7,2],[3,8]] → max 8. Bottom-right [[1,0],[4,9]] → max 9.
4. The 4 × 4 image became 2 × 2.

import numpy as np
a = np.array([[1,3,2,1],[4,6,5,2],[7,2,1,0],[3,8,4,9]])
print(a.reshape(2, 2, 2, 2).max(axis=(1, 3)))

[[6 5]
 [8 9]]

Answer. [[6 5] [8 9]].

Your Turn · Average Pooling

Problem. Apply 2 × 2 average pooling to the same image.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Top-left: (1+3+4+6) ÷ 4 = 3.5. Top-right: (2+1+5+2) ÷ 4 = 2.5.
2. Bottom-left: (7+2+3+8) ÷ 4 = 5. Bottom-right: (1+0+4+9) ÷ 4 = 3.5.

import numpy as np
a = np.array([[1,3,2,1],[4,6,5,2],[7,2,1,0],[3,8,4,9]])
print(a.reshape(2, 2, 2, 2).mean(axis=(1, 3)))

[[3.5 2.5]
 [5.  3.5]]

Answer. [[3.5 2.5] [5. 3.5]]. Max pooling keeps the strongest signal, average pooling the typical one.

10. CNN architecture pattern

[Conv → ReLU → Pool] × few →  Flatten  →  Linear → ReLU → Dropout → Linear (classes)
   feature extraction                          classification

class CNN(nn.Module):
    def __init__ (self, p_drop = 0.3):
        super(). __init__ ()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size = 3, padding = 1), nn.ReLU(),
                nn.MaxPool2d(2), # (B,16,14,14)
            nn.Conv2d(16, 32, kernel_size = 3, padding = 1), nn.ReLU(),
                nn.MaxPool2d(2), # (B,32,7,7)
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),                       # (B, 32*7*7 = 1568)
            nn.Linear(32 * 7 * 7, 128), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(128, 10),
        )
    def forward(self, x):
        return self.classifier(self.features(x))

Small CNN for MNIST

Shape tracking (do this by hand!)

Debug trick to find the flatten size: print(model.features(torch.zeros(1,1,28,28)).shape).

Train it with the same function:

cnn = CNN()
hist_cnn = fit(cnn)
print("CNN test:", evaluate(cnn, test_loader, nn.CrossEntropyLoss()))
# ≈ (0.03, 0.99)

Worked Problem · Shape And Parameter Flow

Problem. A CNN takes a (1, 28, 28) image: Conv(1→16, 3×3, padding 1) → MaxPool 2 → Conv(16→32, 3×3, padding 1) → MaxPool 2 → Flatten → Linear(→128) → Linear(128→10). Track the shapes and count the parameters.

1. Conv 1 with padding 1 keeps the size: (16, 28, 28). Pool 2 halves it: (16, 14, 14).
2. Conv 2 gives (32, 14, 14), and the pool gives (32, 7, 7). Flatten: 32 × 7 × 7 = 1568.
3. Conv parameters = out × (in × K × K) + out. Conv 1: 16 × 9 + 16 = 160. Conv 2: 32 × 144 + 32 = 4,640.
4. Linear 1568 → 128: 1568 × 128 + 128 = 200,832. Linear 128 → 10: 1,290.
5. Add them up.

conv = lambda i, o, k: o * (i * k * k) + o
lin = lambda i, o: i * o + o
print(32 * 7 * 7, conv(1, 16, 3) + conv(16, 32, 3) + lin(1568, 128) + lin(128, 10))

1568 206922

Answer. Flatten size 1568 and 206,922 parameters. Most parameters sit in the first linear layer.

Your Turn · A Wider First Layer

Problem. How many parameters does Conv(3 → 64, 3×3) have for a color image?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Each of the 64 filters has 3 × 3 × 3 = 27 weights, plus one bias.
2. 64 × 27 + 64 = 1,728 + 64 = 1,792.

print(64 * (3 * 3 * 3) + 64)

1792

Answer. 1,792.

11. ★ Mini Project 2 — MNIST: MLP vs CNN

To-do list

1. ✓ Load MNIST, split 55k/5k/10k
2. ✓ Build MLP with dropout
3. ✓ Train with train/val loop + early stopping
4. ✓ Build small CNN
5. ✓ Train the same way
6. ✓ Compare

Comparison table for README

Count params: sum(p.numel() for p in model.parameters()).

Plots to include

1. Train/val loss curves for both models
2. Confusion matrix of the CNN on test set
3. Some misclassified digits:

import matplotlib.pyplot as plt
model = cnn.eval() ; X, y = next(iter(test_loader)) ; X, y = X.to(device), \
    y.to(device)
pred = model(X).argmax(1)
wrong = (pred != y).nonzero().flatten()[:8]
for i, idx in enumerate(wrong):
    plt.subplot(2, 4, i + 1) ; plt.imshow(X[idx].cpu().squeeze(), cmap = "gray")
    plt.title(f"true { y[idx] . item() } pred { pred[idx] . item() } ") ; \
        plt.axis("off")
plt.show()

1. (Optional) visualize learned first-layer filters: cnn.features[0].weight.detach().cpu() → shape (16,1,3,3)

Conclusions to write

• CNN beats MLP even with a similar or smaller parameter count because it keeps spatial structure and shares weights
• Dropout + early stopping controlled overfitting (show curves)
• Errors are mostly on sloppy/ambiguous digits (4↔9, 3↔5)

12. Common Errors

13. Practice Questions

1. What is the shape of one MNIST batch of 64 in PyTorch? → (64, 1, 28, 28)
2. Why does dropout do nothing in eval() mode?
3. Explain early stopping in two sentences.
4. Output size of Conv2d with input 32×32, kernel 5, padding 0, stride 1? → 28
5. What does MaxPool2d(2) do to a (B,16,14,14) tensor? → (B,16,7,7)
6. conv layers use so few weights vs a Linear layer?
7. What is a feature map?

14. Quick Summary

• Images: (B, C, H, W). Normalize inputs
• MLP: flatten → linear layers (loses spatial info)
• Dropout: random neuron shutoff in training (regularizer); Early stopping: stop when val loss stops improving
• CNN: conv filters (learned, shared) + pooling → detects local patterns
• Pattern: [Conv-ReLU-Pool]×n → Flatten → Linear → classes
• CNN ≈ 99% vs MLP ≈ 98% on MNIST

Next: Day 13 — Attention & Transformers (most important day)

⚡ Quick Recap — Day 12

8.1 MLP Architecture

MLP (Multi-Layer Perceptron) = stacked fully-connected layers.

Image 28×28 → Flatten (784) → Linear(784,256) → ReLU → Linear(256,128) → ReLU →
    Linear(128,10) →
logits

• Input: 784 pixels. Output: 10 scores (digits 0–9).
• Loss: CrossEntropyLoss (includes softmax).

⊠ Flattening destroys spatial structure (doesn’t know pixels are neighbors).

8.2 Dropout

Randomly turns off neurons (e.g. 20–50%) during training so the network can’t rely on any one neuron → less overfitting.

nn.Dropout(p = 0.3)

• Active in model.train(), disabled in model.eval().

8.3 Early Stopping

Stop training when validation loss stops improving.

best, patience, wait = float("inf"), 5, 0
for epoch in range(100):
    train() ; val_loss = validate()
    if val_loss < best:
        best, wait = val_loss, 0
        torch.save(model.state_dict(), "best.pt")
    else :
        wait += 1
        if wait >= patience: break
model.load_state_dict(torch.load("best.pt"))

8.4 Training vs Validation Curves — How to Read

8.5 CNN Fundamentals

Why CNN? Images have local patterns (edges → shapes → objects). CNNs small shared filters and the 2-D layout.

Convolution & Filters A small filter (kernel), e.g. 3×3, slides across the image; at each spot it multiplies & sums → produces a feature map.

• Different filters detect different features (edges, curves, textures).
• Filter values are learned.
• Weight sharing: same filter used everywhere → efficient.

Output size: floor((W − K + 2P) / S) + 1

Feature Maps Output of a conv layer: one 2D map per filter. Early layers → edges; deeper layers → complex shapes.

Pooling Downsample feature maps (e.g. MaxPool 2×2 → halves size). Keeps strongest signal, reduces computation, adds small translation invariance.

8.6 Small CNN for MNIST

class CNN(nn.Module):
    def __init__ (self):
        super(). __init__ ()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding = 1), nn.ReLU(), nn.MaxPool2d(2), # 28→14
            nn.Conv2d(16, 32, 3, padding = 1), nn.ReLU(), nn.MaxPool2d(2), # 14→7
        )
        self.classifier = nn.Sequential(
            nn.Flatten(), nn.Linear(32 * 7 * 7, 128), nn.ReLU(),
            nn.Dropout(0.3), nn.Linear(128, 10)
        )
    def forward(self, x):
        return self.classifier(self.features(x))

Shape flow: (B,1,28,28) → (B,16,14,14) → (B,32,7,7) → (B,1568) → (B,10).

8.7 Data Loading

from torchvision import datasets, transforms
tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,),
    (0.3081,))])
train = datasets.MNIST("data", train = True, download = True, transform = tf)

Split 55k/5k for train/val; test set at the end.

8.8 Expected Results

• MLP ≈ 97–98%
• 9%+ (better use of image structure)
• Write conclusion: CNN wins because it keeps spatial structure.

Tip (45-min CNN limit): copy the small CNN above, reuse your training loop, just change the model.

Beginner checkpoint — Day 12 ■ Explain the day’s main idea without using the textbook wording. ■ Write one tiny example from memory. ■ Identify one common failure mode. ■ Explain how today’s topic connects to the next day. ■ Write one question you still cannot answer.

Move on when you can… ☐ train an MLP on MNIST and read the learning curves ☐ explain dropout and early stopping ☐ compute a convolution output size ☐ explain what pooling does ☐ compare the parameter counts of an MLP and a CNN If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.