Day 9 · Neural Networks From Scratch

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ Day 2 and Day 1: matrix multiplication and shapes ☐ Day 4: loss and gradient descent ☐ Day 1: probability (softmax) and derivative intuition If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Understand exactly what happens inside a neural network by building one with only NumPy. No magic. After today, PyTorch will make total sense.

Big picture — read this first A neuron computes a weighted combination of inputs, adds a bias, and applies an activation function. Without nonlinear activations, stacking linear layers would still behave like one linear transformation. Backpropagation is the chain rule applied through the computation graph, so each parameter receives information about how it contributed to the loss. Start with a two-layer network on paper before relying on a framework.

Key idea Take your time today. This is the hardest and most valuable day. Do the hand calculations with a pen!

1. Why neural networks?

Linear regression can only draw straight lines. Real data is often curved and complex (images, language). Neural networks can learn any complex pattern by stacking simple units.

Today’s data: the moons dataset — two interlocking half-circles. No straight line can separate them:

  ● ● ●
●       ●   ○ ○ ○
         ○       ○
 ○ ○ ○     ● ●

A neural network can learn the curved boundary.

Worked Problem · A Network Solves What A Line Cannot

Problem. XOR gives 0,1,1,0 for inputs (0,0), (0,1), (1,0), (1,1). No single straight line separates the 1s from the 0s. Show a tiny network with two hidden ReLU units that gets all four right.

1. Let h1 = ReLU(x1 + x2) and h2 = ReLU(x1 + x2 − 1).
2. Let the output be h1 − 2·h2.
3. (0,0): h1 = 0, h2 = 0 → 0. (0,1): h1 = 1, h2 = 0 → 1. (1,0): 1. (1,1): h1 = 2, h2 = 1 → 2 − 2 = 0.
4. The hidden layer bent the space so a simple rule works. Depth adds this power.

import numpy as np
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
s = X.sum(axis=1)
h1 = np.maximum(0, s); h2 = np.maximum(0, s - 1)
print((h1 - 2 * h2).tolist())

[0, 1, 1, 0]

Answer. [0, 1, 1, 0], exactly XOR.

2. The Neuron — the building block

A neuron does 2 things:

Step 1 — Weighted sum: z = w₁x₁ + w₂x₂ + ... + b Step 2 — Activation: a = f(z) (a non-linear function)

x₁ ──(w₁)──┐
x₂ ──(w₂)──┼──► Σ + b ──► f( ) ──► a (output)
x₃ ──(w₃)──┘

Analogy: a neuron is like a voter. It weighs each input by importance (weights), adds a personal leaning (bias), and decides how strongly to “fire” (activation).

• Weights (w): importance of each input
• Bias (b): shifts the threshold (how easy it is to fire)

2.1 Layers

Many neurons side-by-side = a layer. Each neuron in a layer sees ALL inputs.

Input layer     Hidden layer      Output layer
   x₁ ────────► ( h₁ ) ─────┐
     ╲  ╳       ( h₂ ) ─────┼──► ( ŷ )
   x₂ ────────► ( h₃ ) ─────┘
                ( h₄ )
   (2 inputs)   (4 neurons)      (1 output)

• Input layer: just the data
• Hidden layer(s): where features are learned (not directly seen)
• Output layer: the prediction
• “Deep” learning = many hidden layers

2.2 Layer in matrix form (why Days 2–3 mattered!)

For all n samples at once:

Z = X @ W + b

Worked Problem · One Neuron

Problem. A neuron has inputs x = [2, -1], weights w = [0.5, 0.3] and bias b = 0.1. Compute z = w·x + b and the outputs after sigmoid and ReLU.

1. Multiply and add: 0.5×2 + 0.3×(−1) = 1.0 − 0.3 = 0.7.
2. Add the bias: z = 0.7 + 0.1 = 0.8.
3. ReLU keeps positive numbers: 0.8.
4. Sigmoid = 1 ÷ (1 + e^−0.8) ≈ 0.690.

import numpy as np
x = np.array([2, -1]); w = np.array([0.5, 0.3]); b = 0.1
z = w @ x + b
print(round(z, 3), round(max(0, z), 3), round(1 / (1 + np.exp(-z)), 3))

0.8 0.8 0.69

Answer. z = 0.8, ReLU = 0.8, sigmoid ≈ 0.69.

Your Turn · A Negative Pre-Activation

Problem. A neuron has x = [1, 3], w = [-2, 0.5] and b = 0. Find z, then the ReLU and sigmoid outputs.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. z = −2×1 + 0.5×3 + 0 = −2 + 1.5 = −0.5.
2. ReLU(−0.5) = 0 because negative values are cut to zero.
3. Sigmoid(−0.5) = 1 ÷ (1 + e^0.5) ≈ 0.378.

import numpy as np
z = -2 * 1 + 0.5 * 3
print(z, max(0, z), round(1 / (1 + np.exp(-z)), 3))

-0.5 0 0.378

Answer. z = −0.5, ReLU 0, sigmoid ≈ 0.378.

3. Activation Functions — why we need them

If we skip activations, two layers collapse into one: (X W₁) W₂ = X (W₁W₂) — still just linear! No matter how many layers, we’d only learn straight lines. Activations add the bend (non-linearity).

3.1 ReLU (Rectified Linear Unit) — default for hidden layers

ReLU(z) = max(0, z)

 a │      /
   │     /
───┼────/────── z
   │ 0

• Negative → 0; positive → unchanged
• Fast, simple, works well
• Derivative: 1 if z > 0, else 0

3.2 Sigmoid — for binary output

σ(z) = 1 / (1 + e^(−z))

 1 │        ____
   │      /
.5 │_____/         ← squashes any number into (0, 1)
 0 │___/

• Output = probability
• Examples: σ(0)=0.5, σ(2)=0.88, σ(−2)=0.12
• Derivative: σ(z)(1 − σ(z))

3.3 Others

Rule of thumb: hidden layers → ReLU; binary output → sigmoid; multi-class output → softmax; regression output → none (linear).

Worked Problem · Compare Three Activations

Problem. Compute sigmoid, tanh and ReLU for z = −2, 0 and 2.

1. Sigmoid squeezes values into (0, 1): 1 ÷ (1 + e^−z).
2. Tanh squeezes values into (−1, 1).
3. ReLU is max(0, z): negatives become 0 and positives stay.
4. Compute all three for each z.

import numpy as np
z = np.array([-2., 0., 2.])
print(np.round(1 / (1 + np.exp(-z)), 3))
print(np.round(np.tanh(z), 3))
print(np.maximum(0, z))

[0.119 0.5   0.881]
[-0.964  0.     0.964]
[0. 0. 2.]

Answer. Sigmoid [0.119 0.5 0.881], tanh [-0.964 0. 0.964], ReLU [0. 0. 2.].

Your Turn · Why We Need Activations

Problem. Show that two linear layers without an activation are the same as one linear layer.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Layer 1: h = x @ W1. Layer 2: out = h @ W2. Together: out = (x @ W1) @ W2.
2. Matrix multiplication can be regrouped: (x @ W1) @ W2 = x @ (W1 @ W2).
3. So the two layers collapse into the single matrix W1 @ W2 and can only draw straight lines.
4. A non-linear activation between the layers stops the collapse.

import numpy as np
x = np.array([[1., 2.]]); W1 = np.array([[1., 2.], [3., 4.]]); W2 = np.array([[1.],
    [-1.]])
print(np.allclose((x @ W1) @ W2, x @ (W1 @ W2)))

True

Answer. True. Without an activation, depth adds no power.

4. Forward Propagation — making a prediction

Data flows forward from input to output.

For our 2-layer network:

Z1 = X  @ W1 + b1        (linear, hidden)
A1 = ReLU(Z1)            (activation)
Z2 = A1 @ W2 + b2        (linear, output)
A2 = sigmoid(Z2)         (probability of class 1) = ŷ

4.1 Worked example — ONE neuron by hand

Input x = [1, 2], weights w = [0.5, 0.2], bias b = 0.1.

• z = 0.5×1 + 0.2×2 + 0.1 = 1.0
• a = sigmoid(1.0) = 1/(1 + e⁻¹) = 0.731 Prediction: 73.1% chance of class 1.

Worked Problem · Forward Pass By Hand

Problem. A network has input x = [1, 2], hidden weights W1 = [[0.5, -0.5], [0.25, 0.75]], hidden bias [0, -0.5], output weights W2 = [[1], [2]] and output bias 0.1. The hidden layer uses ReLU. Compute the output.

1. Hidden pre-activation: x @ W1 = [1×0.5 + 2×0.25, 1×(−0.5) + 2×0.75] = [1.0, 1.0].
2. Add the hidden bias: [1.0 + 0, 1.0 − 0.5] = [1.0, 0.5].
3. ReLU leaves positives unchanged: h = [1.0, 0.5].
4. Output: h @ W2 = 1.0×1 + 0.5×2 = 2.0, then add 0.1 to get 2.1.

import numpy as np
x = np.array([[1., 2.]])
W1 = np.array([[0.5, -0.5], [0.25, 0.75]]); b1 = np.array([0., -0.5])
W2 = np.array([[1.], [2.]]); b2 = np.array([0.1])
h = np.maximum(0, x @ W1 + b1)
print(h)
print(h @ W2 + b2)

[[1.  0.5]]
[[2.1]]

Answer. h = [1.0, 0.5] and the output is 2.1.

Your Turn · Shapes Through A Network

Problem. A network maps 784 inputs to 128 hidden units to 10 outputs. For a batch of 32, what are the shapes of the hidden activations and the outputs?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Hidden: (32, 784) @ (784, 128) → (32, 128).
2. Output: (32, 128) @ (128, 10) → (32, 10).
3. Check with zeros.

import numpy as np
X = np.zeros((32, 784)); h = X @ np.zeros((784, 128)); out = h @ np.zeros((128, 10))
print(h.shape, out.shape)

(32, 128) (32, 10)

Answer. (32, 128) and (32, 10).

5. Loss Function

We need a single number for “how wrong is the network?”

5.1 Binary Cross-Entropy (BCE) for 0/1 classification

Loss = −[ y·log(ŷ) + (1−y)·log(1−ŷ) ] (averaged over samples)

Intuition:

• If true y = 1: loss = −log(ŷ). Predict 0.99 → loss 0.01 (great). Predict 0.01 → loss 4.6 (terrible!)
• If true y = 0: loss = −log(1−ŷ). Similarly punishes confident wrong answers.

So BCE hugely punishes being confidently wrong.

Worked: our neuron (ŷ = 0.731, y = 1): 0.313

Other losses: MSE (regression), Categorical cross-entropy (multi-class).

Worked Problem · Binary Cross-Entropy

Problem. Compute the loss −[y log p + (1−y) log(1−p)] for (a) y = 1, p = 0.9, (b) y = 1, p = 0.1, (c) y = 0, p = 0.2.

1. (a) The label is 1, so the loss is −log(0.9) ≈ 0.105. The model is confident and right: small loss.
2. (b) Label 1 but p = 0.1: −log(0.1) ≈ 2.303. Confident and wrong: big loss.
3. (c) Label 0, so the loss is −log(1 − 0.2) = −log(0.8) ≈ 0.223.
4. Check in code.

import numpy as np
bce = lambda y, p: -(y * np.log(p) + (1 - y) * np.log(1 - p))
print(round(bce(1, .9), 3), round(bce(1, .1), 3), round(bce(0, .2), 3))

0.105 2.303 0.223

Answer. 0.105, 2.303, 0.223. The loss explodes when the model is confidently wrong.

Your Turn · Cross-Entropy With 3 Classes

Problem. The model outputs scores [2, 1, 0] and the true class is class 0. Apply softmax, then compute −log of the true class probability.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. e² = 7.389, e¹ = 2.718, e⁰ = 1. The total is 11.107.
2. Probabilities: [0.665, 0.245, 0.090].
3. The true class has probability 0.665, so the loss is −log(0.665) ≈ 0.408.

import numpy as np
s = np.array([2., 1., 0.]); p = np.exp(s) / np.exp(s).sum()
print(np.round(p, 3), round(-np.log(p[0]), 3))

[0.665 0.245 0.09 ] 0.408

Answer. Probabilities [0.665 0.245 0.09] and loss 0.408.

6. Backpropagation ★★★

6.1 The goal

Gradient descent needs: “how does the loss change if I nudge each weight?” i.e., ∂Loss/∂W for every weight in every layer. Backpropagation computes all of these efficiently.

6.2 The chain rule (the only math you need)

If A affects B and B affects C, then: dC/dA = dC/dB × dB/dA “Multiply the rates of change along the chain.”

Example: f = (2x + 1)², at x = 1.

• Let u = 2x + 1 = 3, f = u² = 9
• df/du = 2u = 6, du/dx = 2
• df/dx = 6 × 2 = 12

6.3 Intuition

Analogy: A team project fails. Trace back: how much did each person contribute to the failure? The final person (output layer) is blamed first; then blame is passed backward to earlier people, weighted by how much they influenced the result.

Loss depends on ŷ → ŷ depends on Z2 → Z2 depends on W2 and A1 → A1 depends on Z1 → Z1 depends on W1. We walk this chain backward, multiplying local derivatives.

6.4 Hand-calculated example (one neuron, sigmoid + BCE)

Using earlier numbers: x = [1, 2], y = 1, ŷ = a = 0.731.

A beautiful simplification for sigmoid + BCE:

∂Loss/∂z = a − y = 0.731 − 1 = −0.269
∂Loss/∂w = (∂Loss/∂z) × x = −0.269 × [1, 2] = [−0.269, −0.538]
∂Loss/∂b = −0.269

Update with learning rate 0.1:

• w = [0.5, 0.2] − 0.1×[−0.269, −0.538] = [0.527, 0.254]
• b = 0.1 − 0.1×(−0.269) = 0.127

New z = 0.527 + 0.508 + 0.127 = 1.162 → a = 0.761 (moved closer to 1 )

6.5 The full formulas for our 2-layer network (n = number of samples)

── Output layer ──
dZ2 = A2 − y                        shape (n, 1)
dW2 = A1ᵀ @ dZ2 / n                 shape (n_hidden, 1)
db2 = sum(dZ2) / n                  shape (1, 1)

── Hidden layer ──
dA1 = dZ2 @ W2ᵀ                     shape (n, n_hidden)    ← blame passed back
dZ1 = dA1 * ReLU'(Z1)               ReLU'(Z1) = (Z1 > 0)   ← through the activation
dW1 = Xᵀ @ dZ1 / n                  shape (n_in, n_hidden)
db1 = sum(dZ1, axis=0) / n          shape (1, n_hidden)

Pattern to memorize:

• gradient of weights = (input to that layer)ᵀ @ (error at that layer’s output)
• error is passed back: dA_prev = dZ @ Wᵀ
• multiply by the activation’s derivative

Shape check trick: the gradient of a parameter must have the *same shape as the parameter*. If dW1 isn’t (n_in, n_hidden), you have a bug.

6.6 Weight update

W1 −= lr × dW1     b1 −= lr × db1
W2 −= lr × dW2     b2 −= lr × db2

7. Weight Initialization

Don’t start all weights at 0 — every neuron would compute the same thing and get the same gradient (they’d never become different: symmetry problem). Start with *small random* numbers:

W1 = np.random.randn(n_in, n_hidden) * np.sqrt(2 / n_in)
# "He initialization" (good for ReLU)
b1 = np.zeros((1, n_hidden))                                 # biases can start at 0

Days 9–10

Worked Problem · Why All-Zero Weights Fail

Problem. A hidden layer has 2 neurons. Show what happens to their outputs when every weight starts at 0, and when both neurons start with identical non-zero weights.

1. With the same weights, both neurons compute exactly the same thing for every input.
2. They also receive the same gradient during training, so they stay identical forever. The layer behaves like a single neuron.
3. Different random starting weights break this symmetry.

import numpy as np
x = np.array([[1., 2.]])
print((x @ np.zeros((2, 2))).tolist())
print((x @ np.array([[0.3, 0.3], [0.2, 0.2]])).tolist())
print((x @ np.array([[0.3, -0.1], [0.2, 0.5]])).tolist())

[[0.0, 0.0]]
[[0.7, 0.7]]
[[0.7, 0.9]]

Answer. The first two layers give identical outputs for both neurons. Only the random-looking weights give different values: symmetry is broken.

Your Turn · Pick A Starting Scale

Problem. Compute the Xavier standard deviation √(2 ÷ (fanin + fanout)) and the He standard deviation √(2 ÷ fan_in) for a 784 → 128 layer.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Xavier: √(2 ÷ (784 + 128)) = √(2 ÷ 912) ≈ 0.0468.
2. He: √(2 ÷ 784) ≈ 0.0505.
3. He is slightly larger because ReLU zeroes half of the signal.

import numpy as np
print(round(np.sqrt(2 / (784 + 128)), 4), round(np.sqrt(2 / 784), 4))

0.0468 0.0505

Answer. Xavier 0.0468, He 0.0505.

Move on when you can… ☐ compute a neuron's output by hand ☐ compare sigmoid, tanh and ReLU for the same input ☐ track shapes through a forward pass ☐ compute a cross-entropy loss for one example ☐ explain why all-zero starting weights fail If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.