Day 13 · Attention & Transformers

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ Day 1 and Day 9: dot product and softmax ☐ Day 2: matrix shapes and @ ☐ Day 7: embeddings (a word as a vector) ☐ Day 11: nn.Linear and tensor shapes If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Understand the architecture behind Llama, Mistral, Qwen and every modern LLM. This is the most important day for your GenAI path.

Big picture — read this first Before attention, understand the input: A token is a piece of text. Tokenization turns text into pieces that a model can map to integer IDs. An embedding maps a token ID to a learned vector, so a sequence becomes a matrix of vectors. Attention lets each token assign different weights to other tokens instead of relying only on a fixed sequential memory.

(1) Text → (2) Tokens → (3) Token IDs → (4) Embeddings → (5) Positional information

Key idea Read slowly. Work the numeric example by hand once. If you understand soft-max(QKᵀ/√d)V deeply, you understand the heart of every LLM.

1. Background: how do we handle text?

1.1 Text → numbers

1. Tokenization: split text into pieces called tokens (words or sub-words): "I love AI" → ["I", " love", " AI"]
2. Token IDs: each token → integer ID from a vocabulary
3. Embedding: each ID → a vector (Days 7–8): (seq_len, d_model)

So a sentence with 6 tokens and d_model = 512 becomes a 6 × 512 matrix. This matrix is the input to the Transformer.

1.2 The old way: RNN / LSTM

An RNN reads one word at a time, keeping a “memory” (hidden state) that it updates.

The → cat → sat → on → the → mat
 h1 →  h2 →  h3 → h4 →  h5 →  h6      (each step needs the previous one)

Problems: 1. Slow: must process sequentially, can’t use GPU parallelism 2. Forgetting: information from early words fades after many steps (vanishing gradients) 3. Bottleneck: the whole past is squeezed into one fixed-size vector LSTM/GRU improved memory but did not fully fix these.

1.3 The Transformer idea (2017, “Attention Is All You Need”)

Process all tokens at once and let *every token directly look at every other token* to decide what is relevant. No sequential dependency → fast, parallel, great at long-range relations.

Worked Problem · Text To Numbers

Problem. Turn the sentence “I love AI” into token ids using the vocabulary I → 0, love → 1, AI → 2, and turn the ids back into text.

1. Split the sentence into tokens.
2. Look each token up in the vocabulary to get an id.
3. Reverse the dictionary to decode ids back to words.

vocab = {"I": 0, "love": 1, "AI": 2}
ids = [vocab[t] for t in "I love AI".split()]
inv = {v: k for k, v in vocab.items()}
print(ids, " ".join(inv[i] for i in ids))

[0, 1, 2] I love AI

Answer. Ids [0, 1, 2] and back to I love AI. Real tokenizers split into sub-words, but the idea is the same.

2. Attention — the core intuition

*Question a word asks: “To understand me in this sentence, which other words should I pay attention to?”*

Key idea “The animal didn’t cross the street because it was too tired.”

What does “it” mean? A human links it → “animal”. Attention lets the vector for “it” pull in information from “animal” (high weight) and mostly ignore “street” (low weight).

If the sentence were “…because it was too wide,” “it” would attend to “street”. Attention makes meaning context-dependent.

A library analogy You walk into a library with a question (Query). Every book has a label on its spine (Key) and content inside (Value). 1. Compare your question with every spine label → match scores 2. Convert scores to percentages (softmax) → attention weights 3. Read a blended summary of the books’ contents, weighted by those percentages

That’s exactly self-attention.

Worked Problem · Attention Is A Weighted Average

Problem. Three values are [10, 20, 30] and the attention weights are [0.7, 0.2, 0.1]. What does the token receive?

1. The weights say how much to listen to each value, and they sum to 1.
2. Multiply each value by its weight: 7, 4, 3.
3. Add them up: 14.

import numpy as np
print(np.dot([0.7, 0.2, 0.1], [10, 20, 30]))

14.0

Answer. 14, mostly the first value, because it has the biggest weight.

3. Self-Attention: Q, K, V explained

Input matrix X of shape (T, d_model) (T tokens). Create three versions of it with *three learned weight matrices*:

Q = X · W_Q       "what am I looking for?"
K = X · W_K       "what do I contain/advertise?"
V = X · W_V       "what information do I give if you pay attention to me?"

Each has shape (T, d_k). W’s are learned by backprop (like any weights).

The 4 steps

Step 1 — Scores: S = Q · Kᵀ → shape (T, T). Entry S[i,j] = how much token i’s query matches token j’s key (dot product = similarity).

Step 2 — Scale: S / √d_k. Why? Dot products of long vectors get large; large numbers make softmax extremely “peaky” (almost one-hot) with tiny gradients. Dividing by √d_k keeps values in a healthy range.

Step 3 — Softmax (row-wise): turns each row into probabilities that sum to 1 → attention weights A (T, T). Row i = “how token i distributes its attention across all tokens”.

Step 4 — Weighted sum of values: Output = A · V → (T, d_k). Each token’s new vector = a blend of all tokens’ values, weighted by attention.

THE formula (memorize!)

Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) V

Shape flow

X: (T, d_model)
Q,K,V: (T, d_k)
QKᵀ: (T, d_k)@(d_k, T) = (T, T)      ← every token vs every token
softmax → (T, T)
·V: (T,T)@(T,d_k) = (T, d_k)          ← new representation per token

Softmax refresher softmax(zᵢ) = e^zᵢ / Σⱼ e^zⱼ — bigger scores get exponentially bigger weights, all positive, sum to 1. Example: [2, 1, 0] → [0.665, 0.245, 0.090].

How these problems were checked. The numbers in the worked problems below were calculated and verified with NumPy when this book was built. The PyTorch calls named in the steps do the same math. The PyTorch code in the lessons was not run in the build environment, so run it in your own notebook.

Worked Problem · Attention For Two Tokens

Problem. Two tokens have Q = K = [[1, 0], [0, 1]] and V = [[1, 2], [3, 4]]. Compute softmax(Q·Kᵀ ÷ √dk) · V with dk = 2.

1. Q·Kᵀ = [[1, 0], [0, 1]] (each token matches itself).
2. Divide by √2 ≈ 1.414: [[0.707, 0], [0, 0.707]].
3. Row-wise softmax: e^0.707 = 2.028 and e^0 = 1, so each row is [2.028, 1] ÷ 3.028 ≈ [0.670, 0.330] (reversed for row 2).
4. Output row 1 = 0.670×[1, 2] + 0.330×[3, 4] = [1.66, 2.66]. Output row 2 = 0.330×[1, 2] + 0.670×[3, 4] = [2.34, 3.34].
5. Check the numbers.

import numpy as np
Q = K = np.eye(2); V = np.array([[1., 2.], [3., 4.]])
s = Q @ K.T / np.sqrt(2)
a = np.exp(s) / np.exp(s).sum(axis=1, keepdims=True)
print(np.round(a, 3))
print(np.round(a @ V, 2))

[[0.67 0.33]
 [0.33 0.67]]
[[1.66 2.66]
 [2.34 3.34]]

Answer. Weights [[0.67 0.33] [0.33 0.67]] and output [[1.66 2.66] [2.34 3.34]]: each token becomes a weighted mix of the values, mostly from itself.

Your Turn · Why Divide By √D_K

Problem. Dot products of random vectors grow with the dimension. Estimate the spread (standard deviation) of q·k for d = 4 and d = 64 with random normal vectors.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Each dot product adds d random terms, so its variance is about d and its standard deviation about √d.
2. That is about 2 for d = 4 and about 8 for d = 64.
3. Large scores make the softmax almost one-hot, so gradients vanish. Dividing by √d_k brings the spread back near 1.
4. Check with random samples.

import numpy as np
rng = np.random.default_rng(0)
for d in (4, 64):
    q = rng.normal(size=(20000, d)); k = rng.normal(size=(20000, d))
    print(d, round((q * k).sum(axis=1).std(), 1))

4 2.0
64 8.0

Answer. About 2 for d = 4 and 8 for d = 64, as √d predicts.

Code walkthrough: attention for three tokens, line by line

import numpy as np
X = np.array([[1., 0.], [0., 1.], [1., 1.]])
Wq = Wk = Wv = np.eye(2)
Q, K, V = X @ Wq, X @ Wk, X @ Wv
scores = Q @ K.T / np.sqrt(2)
w = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
out = w @ V
print(np.round(w, 2))
print(np.round(out, 2))
print(w.sum(axis=1))

[[0.4  0.2  0.4 ]
 [0.2  0.4  0.4 ]
 [0.25 0.25 0.5 ]]
[[0.8  0.6 ]
 [0.6  0.8 ]
 [0.75 0.75]]
[1. 1. 1.]

Why that output? Token 1 = [1, 0] matches itself and token 3 = [1, 1] equally well (weights 0.4 and 0.4) and token 2 least (0.2). Token 2 = [0, 1] is the mirror image. Token 3 = [1, 1] matches itself best (0.5) and the other two equally (0.25 each). Every row sums to 1 because of the softmax.

A common error and its message

import numpy as np
Q = K = np.ones((3, 2))
try:
    Q @ K
except ValueError as e:
    print("ValueError:", str(e)[:70])

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with g

Fix: the keys must be transposed so the inner sizes match: write Q @ K.T, which turns (3, 2) @ (2, 3) into a (3, 3) score matrix. The exact wording of the message depends on your NumPy version.

4. ★ Worked numeric example (3 tokens, d_k = 2)

Suppose (to keep numbers simple) after projection:

Q = K = [[1, 0],     ← token 1
         [0, 1],     ← token 2
         [1, 1]]     ← token 3

V =     [[1, 0],
         [0, 1],
         [1, 1]]

Step 1: QKᵀ

        k1 k2 k3
q1 [ 1  0  1 ]     (q1·k1 = 1, q1·k2 = 0, q1·k3 = 1)
q2 [ 0  1  1 ]
q3 [ 1  1  2 ]     (q3·k3 = 1·1+1·1 = 2)

Step 2: divide by √2 ≈ 1.414

[0.707  0     0.707]
[0      0.707 0.707]
[0.707  0.707 1.414]

Step 3: softmax each row

• Row 1: e^0.707=2.03, e^0=1, e^0.707=2.03; sum=5.06 → [0.40, 0.20, 0.40]
• Row 2: → [0.20, 0.40, 0.40]
• Row 3: e^0.707=2.03, 2.03, e^1.414=4.11; sum=8.17 → [0.25, 0.25, 0.50]

Step 4: multiply by V (row 1): 0.40·[1,0] + 0.20·[0,1] + 0.40·[1,1] = [0.80, 0.60]

Token 1’s new vector = [0.80, 0.60] — a mix of all three tokens, mostly itself and token 3.

5. Implement self-attention in PyTorch ★ (ASSIGNMENT)

import math, torch, torch.nn as nn, torch.nn.functional as F
class SelfAttention(nn.Module):
    def __init__ (self, d_model, d_k):
        super(). __init__ ()
        self.W_q = nn.Linear(d_model, d_k, bias = False)
        self.W_k = nn.Linear(d_model, d_k, bias = False)
        self.W_v = nn.Linear(d_model, d_k, bias = False)
        self.d_k = d_k
    def forward(self, x, mask = None):               # x: (B, T, d_model)
        Q, K, V = self.W_q(x), self.W_k(x), self.W_v(x)          # each (B, T, d_k)
        scores = Q @ K.transpose( - 2, - 1) / math.sqrt(self.d_k) # (B, T, T)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = F.softmax(scores, dim =- 1)                     # rows sum to 1
        out = weights @ V                                         # (B, T, d_k)
        return out, weights

Test it:

torch.manual_seed(0)
x = torch.randn(1, 6, 16)              # 1 sentence, 6 tokens, d_model=16
attn = SelfAttention(d_model = 16, d_k = 16)
out, w = attn(x)
print(out.shape, w.shape)              # (1,6,16) (1,6,6)
print(w[0].sum(dim =- 1))              # tensor of 1.0s → each row sums to 1 ✅
import matplotlib.pyplot as plt
tokens = ["The", "cat", "sat", "on", "the", "mat"]
plt.figure(figsize = (5,4))
plt.imshow(w[0].detach().numpy(), cmap = "viridis")
plt.xticks(range(6), tokens, rotation = 45) ; plt.yticks(range(6), tokens)
plt.xlabel("Key (attended TO)") ; plt.ylabel("Query (attending)")
plt.colorbar(label = "attention weight") ; plt.title("Self-attention weights") ; \
    plt.tight_layout() ; \
   plt.show()

Plot attention weights How to read it: row = a word; bright cells = words it attends to most; each row sums to 1.

Analyzing one sentence — do this properly With random untrained weights, patterns are meaningless. Two good options:

Option A — Use meaningful embeddings/pretrained model (recommended):

# pip install transformers
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
bert = AutoModel.from_pretrained("bert-base-uncased", output_attentions = True)
sent = "The animal didn't cross the street because it was too tired"
inp = tok(sent, return_tensors = "pt")
att = bert( ** inp).attentions          # tuple: 12 layers, each (1, 12 heads, T, T)
layer, head = 5, 3
A = att[layer][0, head].detach().numpy()
labels = tok.convert_ids_to_tokens(inp["input_ids"][0])
# plot A like above; look at the row for "it"

Option B: Train your own small attention on a toy task, or hand-set Q/K like the numeric example.

Write 4–5 lines of analysis: “In head 3 of layer 5, the token ‘it’ attends most strongly to ‘animal’ (0.xx), showing the model resolves the pronoun. Other tokens mostly attend to neighbors and [SEP].”

6. Masking

T = 6
causal = torch.tril(torch.ones(T, T))
# 1 0 0 0 0 0
# 1 1 0 0 0 0
# 1 1 1 0 0 0   ← token 3 can see tokens 1,2,3 only
...
out, w = attn(x, mask = causal)

−∞ → softmax gives exactly 0 weight. GPT-style models use causal masks; BERT-style don’t.

Worked Problem · A Causal Mask

Problem. Three tokens all have attention scores [1, 2, 3]. Apply a causal mask (a token may not see later tokens) and compute each row's softmax.

1. Row 1 may only see token 1, row 2 tokens 1–2, row 3 tokens 1–3.
2. Set forbidden scores to −∞. After softmax they get exactly 0 weight.
3. Row 1: only [1], so [1, 0, 0].
4. Row 2: softmax of [1, 2] = [0.269, 0.731], then 0.
5. Row 3: softmax of [1, 2, 3] = [0.090, 0.245, 0.665].

import numpy as np
s = np.array([[1., 2., 3.]] * 3)
s = np.where(np.tril(np.ones((3, 3))) == 1, s, -np.inf)
a = np.exp(s) / np.exp(s).sum(axis=1, keepdims=True)
print(np.round(a, 3))

[[1.    0.    0.   ]
 [0.269 0.731 0.   ]
 [0.09  0.245 0.665]]

Answer. Each row sums to 1, and everything above the diagonal is 0. This is how GPT-style models avoid peeking at future words.

Your Turn · A Padding Mask

Problem. A sentence has 2 real tokens and 1 padding token. Its scores are [0.5, 1.0, 2.0] and the last one is padding. Compute the attention weights.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Mask the padding position by setting its score to −∞.
2. Softmax over [0.5, 1.0]: e^0.5 = 1.649 and e^1 = 2.718, total 4.367.
3. Weights = [0.378, 0.622, 0].

import numpy as np
s = np.array([0.5, 1.0, -np.inf]); a = np.exp(s) / np.exp(s).sum()
print(np.round(a, 3))

[0.378 0.622 0.   ]

Answer. [0.378 0.622 0. ]: the padding gets no attention.

7. Multi-Head Attention

Why? One attention can only focus one way. But a token needs many kinds of relationships at once: grammar (subject↔verb), reference (it↔animal), nearby words, meaning, etc.

Idea Run h attention “heads” in parallel, each with its own smaller WQ, WK, W_V. Concatenate the results and mix with a final linear layer.

X ──► Head 1 (Q1,K1,V1) ──┐
X ──► Head 2 (Q2,K2,V2) ──┼─► concat ─► Linear W_O ─► output (T, d_model)
X ──► Head 3 ...        ──┤
X ──► Head h ...        ──┘

• Per-head size: d_k = d_model / h (e.g., 512 / 8 = 64) → same total compute as one big head
• Different heads learn different patterns

class MultiHeadAttention(nn.Module):
    def __init__ (self, d_model, n_heads):
        super(). __init__ ()
        assert d_model % n_heads == 0
        self.h, self.d_k = n_heads, d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)
        # computes Q,K,V for all heads at once
        self.out = nn.Linear(d_model, d_model)          # W_O
    def forward(self, x, mask = None):
        B, T, D = x.shape
        q, k, v = self.qkv(x).chunk(3, dim =- 1)
        # (B,T,D) → (B,h,T,d_k)
        q, k, v = [t.view(B, T, self.h, self.d_k).transpose(1, 2) for t in (q, k,
            v)]
        scores = q @ k.transpose( - 2, - 1) / math.sqrt(self.d_k)   # (B,h,T,T)
        if mask is not None: scores = scores.masked_fill(mask == 0, float("-inf"))
        w = F.softmax(scores, dim =- 1)
        ctx = (w @ v).transpose(1, 2).contiguous().view(B, T, D)    # concat heads
        return self.out(ctx)

Worked Problem · Head Size And Parameters

Problem. A Transformer has d_model = 512 and 8 heads. What is each head's size, and how many weights are in the Q, K, V and output projections (no biases)?

1. Each head works on d_model ÷ heads = 512 ÷ 8 = 64 dimensions.
2. Each of the four projection matrices is 512 × 512 = 262,144 weights.
3. Four matrices: 4 × 262,144 = 1,048,576.
4. Splitting into heads does not add parameters; it only reshapes the same matrices.

d, h = 512, 8
print(d // h, 4 * d * d)

64 1048576

Answer. 64 dimensions per head and 1,048,576 weights.

Your Turn · Bert-Base Sizes

Problem. BERT-base has d_model = 768 and 12 heads. What is the head size?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. 768 ÷ 12 = 64.
2. The head size must divide dmodel exactly, which is why the code asserts `dmodel % n_heads == 0`.

print(768 // 12, 768 % 12)

64 0

Answer. 64, with remainder 0.

8. Positional Encoding

The problem Self-attention treats input as a set: shuffle the words and, without extra info, it can’t tell. “Dog bites man” and “Man bites dog” would look the same! Word order matters.

Solution Add a position-dependent vector to each token embedding:

input = token_embedding + positional_encoding

Sinusoidal (original paper):

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Each position gets a unique pattern of waves; nearby positions have similar patterns; the model can learn relative distances.

def positional_encoding(T, d_model):
    pos = torch.arange(T).unsqueeze(1)                                   # (T,1)
    i = torch.arange(0, d_model, 2)                                      # (d/2,)
    angle = pos / (10000 ** (i / d_model))                               # (T, d/2)
    pe = torch.zeros(T, d_model)
    pe[:, 0::2] = torch.sin(angle) ; pe[:, 1::2] = torch.cos(angle)
    return pe

Modern alternatives: learned position embeddings (GPT-2), RoPE rotary embeddings (Llama and most modern LLMs), ALiBi.

Worked Problem · Sinusoidal Positions

Problem. For d_model = 4 compute the positional encoding for position 0 and position 1. PE(pos, 2i) = sin(pos ÷ 10000^(2i/d)) and PE(pos, 2i+1) = cos(same angle).

1. For i = 0 the denominator is 10000⁰ = 1. For i = 1 it is 10000^(2/4) = 100.
2. Position 0: angles are 0 and 0, so [sin 0, cos 0, sin 0, cos 0] = [0, 1, 0, 1].
3. Position 1: angle 1 for the first pair: [sin 1, cos 1] = [0.841, 0.540]. Angle 0.01 for the second pair: [0.010, 1.000].
4. Check in code.

import numpy as np
def pe(pos, d=4):
    out = []
    for i in range(d // 2):
        a = pos / 10000 ** (2 * i / d)
        out += [np.sin(a), np.cos(a)]
    return np.round(out, 3)
print(pe(0), pe(1))

[0. 1. 0. 1.] [0.841 0.54  0.01  1.   ]

Answer. Position 0 → [0, 1, 0, 1], position 1 → [0.841, 0.540, 0.010, 1.0]. Each position gets its own pattern, which is added to the word embedding.

Your Turn · Position 2

Problem. What are the first two values of the encoding for position 2 when d_model = 4?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. The angle is 2 ÷ 1 = 2 radians.
2. sin 2 ≈ 0.909 and cos 2 ≈ −0.416.

import numpy as np
print(round(np.sin(2), 3), round(np.cos(2), 3))

0.909 -0.416

Answer. 0.909 and −0.416.

9. Feed-Forward Network (FFN)

After attention mixes info between tokens, the FFN processes *each token independently* (same network applied to every position):

FFN(x) = Linear(d_model → 4·d_model) → ReLU/GELU → Linear(4·d_model → d_model)

• Expands to a bigger space (4×), applies non-linearity, then compresses back
• Roughly: attention = “gather information”, FFN = “think about it / store knowledge”
• Most of a Transformer’s parameters live in FFNs

Worked Problem · Ffn Parameter Count

Problem. The feed-forward network in a Transformer block expands dmodel = 512 to dff = 2048 and back. Count its parameters including biases.

1. Expand: 512 × 2048 weights + 2048 biases = 1,048,576 + 2,048 = 1,050,624.
2. Project back: 2048 × 512 weights + 512 biases = 1,048,576 + 512 = 1,049,088.
3. Total = 1,050,624 + 1,049,088 = 2,099,712.

print(512 * 2048 + 2048 + 2048 * 512 + 512)

2099712

Answer. 2,099,712 parameters, about twice the attention projections' 1,048,576 weights.

Your Turn · A Bigger Model

Problem. Repeat the count for dmodel = 768 and dff = 3072.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Expand: 768 × 3072 + 3072 = 2,359,296 + 3,072 = 2,362,368.
2. Project back: 3072 × 768 + 768 = 2,359,296 + 768 = 2,360,064.
3. Total = 4,722,432.

print(768 * 3072 + 3072 + 3072 * 768 + 768)

4722432

Answer. 4,722,432.

10. Residual Connections (skip connections)

output = x + Sublayer(x)

• The input bypasses the sublayer and is added to its output
• Gradients flow straight through the shortcut → very deep networks train well
• The sublayer only needs to learn the change (residual), not the whole thing

Worked Problem · Why A Skip Connection Helps Gradients

Problem. A sub-layer computes f(x) = 0.1·x. Compare the slope of f(x) alone with the slope of x + f(x).

1. Slope of f alone: 0.1. After many layers 0.1 × 0.1 × … shrinks toward zero.
2. Slope of x + f(x): 1 + 0.1 = 1.1. The “1” is the direct path that skips the sub-layer.
3. Because the direct path stays near 1, gradients reach early layers.

f = lambda x: 0.1 * x
g = lambda x: x + f(x)
e = 1e-6
print(round((f(2 + e) - f(2 - e)) / (2 * e), 3), round((g(2 + e) - g(2 - e)) / (2 *
    e), 3))

0.1 1.1

Answer. Slopes 0.1 and 1.1.

11. Layer Normalization

Normalize each token’s vector across its features to mean 0, variance 1 (then learnable scale/shift).

• Stabilizes and speeds up training
• BatchNorm (used in CNNs) normalizes across the batch; Layer-Norm across features per sample → better for sequences of variable length

Worked Problem · Normalize One Token

Problem. Layer-normalize the vector [1, 2, 3, 4] (ignore the learned scale and shift).

1. Mean = 10 ÷ 4 = 2.5.
2. Gaps: −1.5, −0.5, 0.5, 1.5. Squared: 2.25, 0.25, 0.25, 2.25. Variance = 5 ÷ 4 = 1.25, std ≈ 1.118.
3. Divide each gap by the std: [−1.342, −0.447, 0.447, 1.342].
4. The result has mean 0 and variance 1.

import numpy as np
x = np.array([1., 2., 3., 4.])
y = (x - x.mean()) / np.sqrt(x.var() + 1e-5)
print(np.round(y, 3), round(y.mean(), 3))

[-1.342 -0.447  0.447  1.342] 0.0

Answer. [-1.342 -0.447 0.447 1.342], mean 0.

Your Turn · Why The Tiny Epsilon

Problem. What does layer norm do to [10, 10, 10, 10] and why is a small ε needed?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. The mean is 10, so every gap is 0 and the variance is 0.
2. Without ε we would divide 0 by 0. With ε = 1e-5 we divide 0 by a small positive number and get 0.

import numpy as np
x = np.array([10., 10., 10., 10.])
print((x - x.mean()) / np.sqrt(x.var() + 1e-5))

[0. 0. 0. 0.]

Answer. [0. 0. 0. 0.], and no division by zero.

12. The Transformer Block (DRAW THIS FROM MEMORY)

     Input  x  (T, d_model)   [embeddings + positional encoding]
       │
       ├────────────────────┐
       ▼                    │  (residual)
Multi-Head Attention        │
       │                    │
       ▼                    │
     Add  ◄─────────────────┘
       │
   LayerNorm
       │
       ├────────────────────┐
       ▼                    │  (residual)
Feed-Forward Network        │
       │                    │
       ▼                    │
     Add  ◄─────────────────┘
       │
   LayerNorm
       │
    Output (T, d_model)  → feed into next block (stack N times, e.g. 12 or 96)

Input and output have the same shape, so blocks stack easily.

Post-LN vs Pre-LN: the drawing above is the original (Post-LN). Modern LLMs use Pre-LN: Layer-Norm before each sublayer (x + Attn(LN(x))) — trains more stably.

class TransformerBlock(nn.Module):
    def __init__ (self, d_model = 64, n_heads = 4):
        super(). __init__ ()
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ln1, self.ln2 = nn.LayerNorm(d_model), nn.LayerNorm(d_model)
       self.ffn = nn.Sequential(nn.Linear(d_model, 4 * d_model), nn.GELU(),
           nn.Linear(4 * d_model, d_model))
    def forward(self, x, mask = None):
        x = self.ln1(x + self.attn(x, mask))      # attention + residual + norm
        x = self.ln2(x + self.ffn(x))             # FFN + residual + norm
        return x

Block in code (reference — you’re not required to build a full Transformer)

Worked Problem · Shape Tracking Through A Block

Problem. A batch of 2 sentences with 5 tokens each uses dmodel = 16 and dff = 64. Track the tensor shape through attention, the residual add, and the feed-forward network.

1. Input: (2, 5, 16) = batch, tokens, features.
2. Self-attention returns the same shape: (2, 5, 16).
3. The residual connection adds the input to the output, so shapes must match: (2, 5, 16).
4. FFN expands to (2, 5, 64) and projects back to (2, 5, 16).
5. Every block keeps the shape, which is why blocks can be stacked.

import numpy as np
x = np.zeros((2, 5, 16))
h = x @ np.zeros((16, 64))
print(h.shape, (h @ np.zeros((64, 16))).shape)

(2, 5, 64) (2, 5, 16)

Answer. (2, 5, 64) inside the FFN and (2, 5, 16) at the end.

Your Turn · A Residual Connection

Problem. A block's input is [1, 2] and its sub-layer output is [0.1, -0.2]. What does the residual connection give?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. A residual connection adds the input back: output = x + f(x).
2. [1 + 0.1, 2 − 0.2] = [1.1, 1.8].
3. Even if f(x) is useless, the original signal still passes through unchanged, which makes deep networks trainable.

import numpy as np
print(np.array([1, 2]) + np.array([0.1, -0.2]))

[1.1 1.8]

Answer. [1.1 1.8].

13. Three Transformer families

How an LLM generates text (decoder-only): 1. Input tokens → embeddings + positions 2. Pass through N Transformer blocks (causal attention) 3. Final layer outputs a probability distribution over the vocabulary for the *next token 4. Pick/sample a token, append it, repeat Cross-attention:* queries come from the decoder; keys and values come from the encoder output.

Worked Problem · Pick The Family

Problem. Choose encoder-only (BERT), decoder-only (GPT) or encoder-decoder (T5) for: (a) classify customer reviews, (b) write the next sentence of a story, (c) translate English to French.

1. Encoder-only models read the whole text and produce a representation: good for understanding tasks.
2. Decoder-only models predict the next token from the left context: good for generating text.
3. Encoder-decoder models read one sequence and write another.
4. (a) classification → encoder-only. (b) generation → decoder-only. (c) translation → encoder-decoder.

Answer. (a) BERT-style, (b) GPT-style, (c) T5-style.

14. Why this matters for your GenAI path

15. Common Confusions Cleared

1. Explain self-attention in simple words. Each token creates a query, compares it to all tokens’ keys to get weights, and takes a weighted average of their values.
2. Why scale by √d_k? Large dot products push softmax into saturation → vanishing gradients.
3. Time complexity? O(T² · d) — quadratic in sequence length.
4. Why multi-head? Different heads capture different relationships in different subspaces.
5. Role of residual connections? Enable gradient flow, allow deep stacking.
6. BERT vs GPT? BERT: bidirectional encoder for understanding. GPT: causal decoder for generation.
7. Why positional encoding? Attention is permutation-invariant; order info must be injected.
8. What does the FFN do? Per-token non-linear transformation; holds much of the model’s capacity.

17. Practice Tasks

1. Redo the numeric example with token 2’s row and verify [0.20, 0.40, 0.40] then compute its output. (Answer: 0.20·[1,0]+0.40·[0,1]+0.40·[1,1] = [0.60, 0.80])
2. Print shapes at every step in SelfAttention.forward.
3. Apply a causal mask and confirm the weight matrix is lower-triangular.
4. Draw the Transformer block on paper without looking. Label every shape.
5. Explain to a friend why “bank” in “river bank” vs “bank loan” gets different vectors after attention.

18. Day 13 Deliverables (repo folder)

• self_attention.py (or notebook) with the module
• Attention heatmap image
• 5-line sentence analysis
• Hand-drawn (or digital) Transformer block diagram image
• README explaining Q/K/V in your own words

19. Quick Summary

• Attention: softmax(QKᵀ/√d_k)V — weighted average of values based on query–key similarity
• Q = what I seek, K = what I offer, V = what I give
• Multi-head = several attentions in parallel
• Positional encoding gives order; FFN processes tokens; residual + LayerNorm stabilize
• Block = MHA → Add&Norm → FFN → Add&Norm; stack many
• Decoder-only + causal mask = LLMs like Llama

Next: Days 14–15 — Capstone + Recap

⚡ Quick Recap — Day 13

9.1 Why Not RNN/LSTM?

RNN reads words one by one, passing a hidden state. Problems: ① Sequential → can’t parallelize → slow ② Vanishing gradients → forgets long-range info *③ Fixed-size memory* bottleneck LSTM/GRU help but don’t fully fix it.

Transformers: process all tokens in parallel and let every word *look at every other word directly*.

9.2 Attention — Core Idea

“When understanding a word, which other words should I focus on?”

*Key idea “The animal didn’t cross the street because it was too tired.”* What does “it” refer to? Attention lets “it” pay high attention to “animal”.

9.3 Self-Attention & Q / K / V

Each token creates 3 vectors (via learned weight matrices):

Q = X·Wq K = X·Wk V = X·Wv

Steps: 1. Scores = Q·Kᵀ → how well each query matches each key 2. Scale by √dₖ → prevents huge values that make softmax too sharp 3. Softmax → scores become weights that sum to 1 (attention weights) 4. Weighted sum of V → new context-aware representation

Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V

Attention(Q, K, V): score every Q against every K, scale, softmax into percentages, then blend V.

Shapes (seq length T, dim d): Q,K,V = (T, d) → scores (T, T) → output (T, d).

Softmax: softmax(zᵢ) = e^zᵢ / Σ e^zⱼ — turns scores into probabilities.

9.4 ★ Implementation in PyTorch

import torch, math
import torch.nn as nn
import torch.nn.functional as F
class SelfAttention(nn.Module):
    def __init__ (self, d_model, d_k):
        super(). __init__ ()
        self.Wq = nn.Linear(d_model, d_k, bias = False)
        self.Wk = nn.Linear(d_model, d_k, bias = False)
        self.Wv = nn.Linear(d_model, d_k, bias = False)
        self.d_k = d_k
    def forward(self, x, mask = None):          # x: (B, T, d_model)
        Q, K, V = self.Wq(x), self.Wk(x), self.Wv(x)
        scores = Q @ K.transpose( - 2, - 1) / math.sqrt(self.d_k) # (B,T,T)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = F.softmax(scores, dim =- 1)   # each row sums to 1
        return weights @ V, weights

import matplotlib.pyplot as plt
tokens = ["The","cat","sat","on","the","mat"]
plt.imshow(weights[0].detach(), cmap = "viridis")
plt.xticks(range(len(tokens)), tokens) ; plt.yticks(range(len(tokens)), tokens)
plt.colorbar() ; plt.xlabel("Key (attended to)") ; plt.ylabel("Query")

Plot attention weights Analyzing a sentence: Row = a word; bright cells = words it attends to. (Note: with random untrained weights patterns are arbitrary — say so in your analysis, or use a pretrained model’s attention to show meaningful links.)

9.5 Masking

• Padding mask: ignore pad tokens.
• Causal mask: in GPT-style decoders, a token can’t see the future (lower-triangular mask). torch.tril(torch.ones(T,T)).

9.6 Multi-Head Attention

Run several attentions in parallel (heads), each with its own Q/K/V, then concatenate.

• Each head learns a different relationship (grammar, coreference, position…).
• d_k = d_model / num_heads.
• Concatenate heads → final linear layer Wo.

9.7 Positional Encoding

Attention has no sense of order (it sees a bag of tokens). Add position info to embeddings:

• Original: sinusoidal — PE(pos,2i)=sin(pos/10000^(2i/d)), cos for odd dims
• Modern: learned or RoPE (rotary).

input = token_embedding + positional_encoding

9.8 Feed-Forward Network (FFN)

Applied to each token independently: FFN(x) = Linear(d → 4d) → ReLU/GELU → Linear(4d → d) Attention mixes information between tokens; FFN processes each token deeper.

9.9 Residual Connections

output = x + Sublayer(x)

• Gradient flows easily through the shortcut → trains deep networks.
• Keeps original info.

9.10 Layer Normalization

Normalizes each token’s vector (mean 0, var 1) → stable, faster training.

• BatchNorm normalizes across batch (CNNs). LayerNorm across features (Transformers).

9.11 The Transformer Block

(Draw this from memory!)

Input embeddings + Positional Encoding
        │
        ├───────────────┐
        ▼               │
 Multi-Head Attention   │
        │               │
        ▼               │
   Add (residual) ◄─────┘
        ▼
    LayerNorm
        │
        ├───────────────┐
        ▼               │
Feed-Forward Network  │
      │               │
      ▼               │
 Add (residual) ◄─────┘
      ▼
  LayerNorm
      ▼
   Output  → (stack N times)

(Original paper = Post-LN as drawn; modern LLMs use Pre-LN: LayerNorm before each sublayer.)

9.12 Architecture Types

Cross-attention: decoder’s Q attends to encoder’s K,V.

(Interview Ready)

• Why divide by √dₖ? Large dot products → softmax saturates → tiny gradients.
• Complexity of attention? O(T²) in sequence length → why long context is expensive.
• Why Q, K, V separate? Lets model learn different roles for “asking”, “matching”, “giving”.
• Why multi-head? Capture different types of relationships at once.

Beginner checkpoint — Day 13 ■ Explain the day’s main idea without using the textbook wording. ■ Write one tiny example from memory. ■ Identify one common failure mode. ■ Explain how today’s topic connects to the next day. ■ Write one question you still cannot answer.

Move on when you can… ☐ compute attention for two or three tokens by hand ☐ say what Q, K and V are for ☐ explain what a causal mask does ☐ compute the head size from d_model and the number of heads ☐ list the parts of a Transformer block and say why residuals help If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.