Chapter 27 of 33

Solutions Appendix and References

Stop. Have you tried Level 1 and the hints? If yes, go ahead. Read one solution, understand each line, close it, and type your own version from memory. Notes on the solutions - TESTED : run on the exact generated data f

Stop. Have you tried Level 1 and the hints? If yes, go ahead. Read one solution, understand each line, close it, and type your own version from memory.

Notes on the solutions

TESTED: run on the exact generated data from the book (no download needed).
VERIFY LOCALLY: needs an external dataset or model download; run it yourself and expect small differences.
ADDED STARTER: a short snippet this guide added for a required step the source solution did not show. Comments marked # added or # fix show where this guide changed the source code. Check shapes first if something errors.
Your exact numbers will differ slightly from the book's (e.g. Day 4: about w=2.77, b=4.21). That is normal.

Status TESTED for the cosine math on the book shapes. Titanic part: VERIFY LOCALLY on the real titanic.csv.

Assignment A: cosine similarity (no loops)

import numpy as np

python

def cosine_sim_matrix(A, B):
    A_norm = A / np.linalg.norm(A, axis=1, keepdims=True)
    B_norm = B / np.linalg.norm(B, axis=1, keepdims=True)
    return A_norm @ B_norm.T

python

# scalar check: should print about 0.707
a, b = np.array([1, 0]), np.array([1, 1])
print(round(float(cosine_sim_matrix(a[None, :], b[None, :])[0, 0]), 3))

python

A = np.random.rand(4, 8)
B = np.random.rand(5, 8)
S = cosine_sim_matrix(A, B)
print(S.shape)                                    # (4, 5)

python

# added: loop version, kept separate, ONLY to prove the fast one is correct
S_slow = np.zeros((4, 5))
for i in range(4):
    for j in range(5):
        S_slow[i, j] = A[i] @ B[j] / (np.linalg.norm(A[i]) * np.linalg.norm(B[j]))
print(np.allclose(S, S_slow))                     # True
print(np.allclose(np.diag(cosine_sim_matrix(A, A)), 1))   # True

Assignment B: Titanic cleaning

import pandas as pd

python

df = pd.read_csv("titanic.csv")
print(df.shape); df.info(); print(df.isna().sum())

python

df["Age"] = df["Age"].fillna(df["Age"].median())
df["Embarked"] = df["Embarked"].fillna(df["Embarked"].mode()[0])
df = df.drop(columns=["Cabin", "Ticket", "Name", "PassengerId"])
df["Sex"] = df["Sex"].map({"male": 0, "female": 1})
df = pd.get_dummies(df, columns=["Embarked"], drop_first=True, dtype=int)
# added dtype=int
assert df.isna().sum().sum() == 0                 # safety net
df.to_csv("titanic_clean.csv", index=False)

python

print(df.groupby("Sex")["Survived"].mean())
print(df.groupby("Pclass")["Survived"].mean())

Reading it

Dividing each row by its length makes every vector length 1, so the bottom of the cosine formula is 1 and only the dot product remains; A_norm @ B_norm.T computes every pair in one shot. Reasons to drop columns: Cabin is mostly empty; Name, Ticket and PassengerId are unique per person, so they cannot help predict.

Status TESTED on the exact generated 100-point setup; no external data. The plotting of the fitted line is an added starter.

Gradient descent from scratch vs scikit-learn

python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

python

w, b = 0.0, 0.0
lr, epochs, n = 0.1, 1000, len(X)
losses = []
for _ in range(epochs):
    y_pred = w * X + b                    # 1. predict
    error = y_pred - y                    # 2. how wrong
    loss = (error ** 2).mean()            # 3. MSE
    losses.append(loss)
    dw = (2 / n) * (X * error).sum()      # 4. gradients
    db = (2 / n) * error.sum()
    w -= lr * dw                          # 5. step downhill
    b -= lr * db
print(w, b)

plt.plot(losses); plt.xlabel("Epoch"); plt.ylabel("MSE"); plt.title("Loss curve"); plt.show()

python

model = LinearRegression().fit(X, y)
print(model.coef_, model.intercept_)

python

# added: data and fitted line
plt.scatter(X, y, s=10)
xs = np.array([[0.0], [2.0]])
plt.plot(xs, w * xs + b, "r", label="my line"); plt.legend(); plt.title("Fit"); \
    plt.show()

Reading it

Expect w near 2.77 and b near 4.21, not exactly 3 and 4, because the random noise nudges the best-fit line. Your numbers and scikit-learn's should match to about 2 decimals.

Status TESTED on the exact 60-point generated data; the required CV result had all four models. The degree plot is an added starter.

Degree sweep, degree-15 comparison and 5-fold CV

python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error

python

np.random.seed(0)
X = np.sort(np.random.rand(60, 1) * 6 - 3, axis=0)
y = (0.5 * X**3 - X**2 + 2 * X + np.random.randn(60, 1) * 3).ravel()
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3,
    random_state=1)

python

train_err, val_err = [], []
for d in range(1, 16):
    m = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression())
    m.fit(X_train, y_train)
    train_err.append(mean_squared_error(y_train, m.predict(X_train)))
    val_err.append(mean_squared_error(y_val, m.predict(X_val)))
best = int(np.argmin(val_err)) + 1
print("best degree", best)

python

# added: the U-shaped plot
plt.plot(range(1, 16), train_err, "o-", label="train")
plt.plot(range(1, 16), val_err, "s-", label="validation")
plt.axvline(best, ls="--", c="gray"); plt.yscale("log")
plt.xlabel("Polynomial degree"); plt.ylabel("MSE"); plt.legend(); plt.show()

python

# degree-15 validation comparison (required)
for name, reg in [("Linear", LinearRegression()), ("Ridge", Ridge(alpha=1)),
                  ("Lasso", Lasso(alpha=0.1, max_iter=50000))]:
    m = make_pipeline(PolynomialFeatures(15), StandardScaler(), reg).fit(X_train,
        y_train)
    print(name, mean_squared_error(y_val, m.predict(X_val)))
# 5-fold CV for all four models (required)
models = [
    ("Deg3 Linear",  make_pipeline(PolynomialFeatures(3), StandardScaler(),
        LinearRegression())),
    ("Deg15 Linear", make_pipeline(PolynomialFeatures(15), StandardScaler(),
        LinearRegression())),
    ("Deg15 Ridge",  make_pipeline(PolynomialFeatures(15), StandardScaler(),
        Ridge(alpha=1))),
    ("Deg15 Lasso",  make_pipeline(PolynomialFeatures(15), StandardScaler(),
                                   Lasso(alpha=0.1, max_iter=50000))),
]
for name, model in models:
    scores = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_squared_error")
    print(name, scores.mean(), scores.std())

Reading it

Plain degree-15 Linear can be wildly unstable on some folds: overfitting at its worst. Ridge and Lasso tame it. A large std tells you one split can mislead, which is why we use CV. Keeping the scaler inside the pipeline stops leakage across folds.

Status REFERENCE IMPLEMENTATION - VERIFY LOCALLY on creditcard.csv (external dataset).

Three-model comparison

python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import (confusion_matrix, precision_score, recall_score,
                             f1_score, roc_auc_score)

python

df = pd.read_csv("creditcard.csv")
print(df["Class"].value_counts(normalize=True))   # about 0.17% fraud
X = df.drop(columns=["Class"]); y = df["Class"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
X_train, X_test = X_train.copy(), X_test.copy()
# added: avoids SettingWithCopy warnings
scaler = StandardScaler().fit(X_train[["Amount", "Time"]])
# learn from TRAIN only
X_train[["Amount", "Time"]] = scaler.transform(X_train[["Amount", "Time"]])
X_test[["Amount", "Time"]] = scaler.transform(X_test[["Amount", "Time"]])

python

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000,
        class_weight="balanced"),
    "Random Forest": RandomForestClassifier(n_estimators=100,
        class_weight="balanced",
                                            n_jobs=-1, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(random_state=42),
}
rows = []
for name, model in models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    prob = model.predict_proba(X_test)[:, 1]      # probabilities for AUC
    rows.append({"Model": name,
                 "Precision": precision_score(y_test, pred),
                 "Recall": recall_score(y_test, pred),
                 "F1": f1_score(y_test, pred),
                 "ROC-AUC": roc_auc_score(y_test, prob)})
    print(name); print(confusion_matrix(y_test, pred))    # [[TN, FP], [FN, TP]]
print(pd.DataFrame(rows).round(3))

Reading it

If boosting is too slow, sample rows (keep every fraud row) or swap in HistGradientBoostingClassifier. Accuracy hides the story: a model that always says “not fraud” is about 99.8% accurate with 0% recall.

Status Telco: REFERENCE IMPLEMENTATION - VERIFY LOCALLY (Telco CSV and the sentence-transformers download are external). Parts A-E are ADDED STARTERS. One fix to the source code: the numeric columns now go through an imputer, because TotalCharges has blanks and LogisticRegression cannot accept NaN.

Telco churn pipeline (Part F)

python

import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold, \
    cross_validate
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
# fix: fill blanks inside the pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

python

df = pd.read_csv("telco.csv")
# renamed download (see Start Here)
print(df.shape, df["Churn"].value_counts(normalize=True))
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")
df = df.drop(columns=["customerID"])
df["Churn"] = df["Churn"].map({"Yes": 1, "No": 0})
X = df.drop(columns=["Churn"]); y = df["Churn"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

python

num_cols = ["tenure", "MonthlyCharges", "TotalCharges"]
cat_cols = [c for c in X.columns if c not in num_cols]
pre = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
])
models = {
    "LogReg": LogisticRegression(max_iter=1000, class_weight="balanced"),
    "RandomForest": RandomForestClassifier(n_estimators=300,
        class_weight="balanced",
                                           random_state=42),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for name, model in models.items():
    pipe = Pipeline([("pre", pre), ("clf", model)])
r = cross_validate(pipe, X_train, y_train, cv=cv,
                   scoring=["precision", "recall", "f1", "roc_auc"])
print(name, {k: round(v.mean(), 3) for k, v in r.items() if
    k.startswith("test_")})

Then: choose by CV, test once, and show top features (added starter)

python

from sklearn.metrics import classification_report, roc_auc_score
import pandas as pd

python

best = Pipeline([("pre", pre), ("clf", models["LogReg"])])
# replace with the CV winner
best.fit(X_train, y_train)
pred = best.predict(X_test); prob = best.predict_proba(X_test)[:, 1]
print(classification_report(y_test, pred)); print("ROC-AUC", roc_auc_score(y_test,
    prob))

python

rf = Pipeline([("pre", pre), ("clf", models["RandomForest"])]).fit(X_train, y_train)
names = rf.named_steps["pre"].get_feature_names_out()
imp = pd.Series(rf.named_steps["clf"].feature_importances_, index=names)
imp.sort_values().tail(10).plot.barh(title="Top 10 features")

Parts A-E: checkpoint starters (ADDED STARTER)

python

import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

python

# A. distances for P1=(25,6), P2=(30,8)
p1, p2 = np.array([25, 6]), np.array([30, 8])
print("euclidean", np.linalg.norm(p1 - p2), "manhattan", np.abs(p1 - p2).sum(),
      "cosine", p1 @ p2 / (np.linalg.norm(p1) * np.linalg.norm(p2)))
# B. k-NN with and without scaling on a tiny made-up matrix (age, income)
rng = np.random.RandomState(0)
X_small = np.c_[rng.randint(20, 60, 40), rng.randint(20000, 100000, 40)]
y_small = (X_small[:, 0] > 40).astype(int)
for name, Xm in [("raw", X_small), ("scaled",
    StandardScaler().fit_transform(X_small))]:
    knn = KNeighborsClassifier(n_neighbors=5).fit(Xm, y_small)
    print(name, "accuracy", knn.score(Xm, y_small))

python

# C + D. k-Means elbow and PCA on the scaled Telco numeric columns
num = df[["tenure", "MonthlyCharges", "TotalCharges"]]
Xs = StandardScaler().fit_transform(num.fillna(num.median()))
inertia = [KMeans(k, n_init=10, random_state=42).fit(Xs).inertia_ for k in range(1,
    11)]
plt.plot(range(1, 11), inertia, "o-"); plt.xlabel("k"); plt.ylabel("inertia"); \
    plt.show()
pca = PCA(n_components=2); X2 = pca.fit_transform(Xs)
print(X2.shape, pca.explained_variance_ratio_)

python

# E. semantic search (paste your Days 2–3 cosine_sim_matrix above this line)
from sentence_transformers import SentenceTransformer
st = SentenceTransformer("all-MiniLM-L6-v2")
docs = ["How to reset my password", "Best pizza recipes", "Change account login "
   "credentials"]
D_emb = st.encode(docs); q_emb = st.encode(["I forgot my password"])
print(D_emb.shape, cosine_sim_matrix(q_emb, D_emb))
# top score = winning sentence

Reading it

Part B: with raw numbers the income column dominates every distance, so scaling changes who the neighbors are. Real-data expectation for Telco: ROC-AUC roughly 0.80 to 0.85.

Status TESTED on the exact 1,000-point Moons setup and re-run while building this guide: about 97.5% test accuracy, gradient check agreeing to about 8 decimals, no-hidden-layer baseline about 0.86. The gradient check and baseline are ADDED STARTERS.

Two-layer network in NumPy

python

import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

python

X, y = make_moons(1000, noise=0.2, random_state=42)
y = y.reshape(-1, 1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
    random_state=42)

python

np.random.seed(0)
n_in, n_hidden, n_out = 2, 16, 1
W1 = np.random.randn(n_in, n_hidden) * np.sqrt(2 / n_in);     b1 = np.zeros((1,
    n_hidden))
W2 = np.random.randn(n_hidden, n_out) * np.sqrt(2 / n_hidden); b2 = np.zeros((1,
    n_out))
lr, epochs, eps = 0.1, 3000, 1e-8

python

def relu(z): return np.maximum(0, z)
def relu_grad(z): return (z > 0).astype(float)
def sigmoid(z): return 1 / (1 + np.exp(-z))
def forward(X):
    Z1 = X @ W1 + b1; A1 = relu(Z1); Z2 = A1 @ W2 + b2; A2 = sigmoid(Z2)
    return Z1, A1, Z2, A2
def bce(y, a): return -np.mean(y * np.log(a + eps) + (1 - y) * np.log(1 - a + eps))

python

losses = []
for epoch in range(epochs):
    Z1, A1, Z2, A2 = forward(X_train)
    loss = bce(y_train, A2); losses.append(loss); n = len(X_train)
    dZ2 = A2 - y_train                                  # backward: start at the end
    dW2 = A1.T @ dZ2 / n; db2 = dZ2.sum(axis=0, keepdims=True) / n
    dA1 = dZ2 @ W2.T; dZ1 = dA1 * relu_grad(Z1)
    # ReLU passes gradient where Z1 > 0
    dW1 = X_train.T @ dZ1 / n; db1 = dZ1.sum(axis=0, keepdims=True) / n
if epoch == 0:
# added: gradient check on W1[0,0]
    h, old = 1e-5, W1[0, 0]
    W1[0, 0] = old + h; lp = bce(y_train, forward(X_train)[3])
    W1[0, 0] = old - h; lm = bce(y_train, forward(X_train)[3]); W1[0, 0] = old
    print("analytic", dW1[0, 0], "numeric", (lp - lm) / (2 * h))

W1 -= lr * dW1; b1 -= lr * db1; W2 -= lr * dW2; b2 -= lr * db2

python

print("loss", losses[0], "->", losses[-1])
print("test acc", ((forward(X_test)[3] > 0.5) == y_test).mean())

No-hidden-layer baseline (added)

python

from sklearn.linear_model import LogisticRegression
base = LogisticRegression().fit(X_train, y_train.ravel())
print("no hidden layer:", base.score(X_test, y_test.ravel()))
# about 0.85: straight line

Reading it

For the experiment table, wrap the training code in a function train(n_hidden, lr, activation) that returns test accuracy, then loop over hidden size 2, 4, 16, 64 and learning rate 0.001, 0.01, 0.1, 1.0, and compare tanh with ReLU (tanh derivative: 1 - tanh(z)**2). Debug cheat sheet: nan loss → lr too high or log(0); accuracy 50% → y has shape (n,) not (n,1); shape mismatch → gradient shape must equal weight shape.

Status The source guide marks the 6-run grid TESTED on the 2,000-sample Moons setup (expected pattern: SGD is more learning-rate-sensitive, Adam more robust at these rates). PyTorch was not available while restyling, so this copy was not re-run; the validation-accuracy and combined-plot lines are ADDED STARTER (not run).

Six-run optimizer grid + saving weights

python

import numpy as np, torch
import torch.nn as nn
import matplotlib.pyplot as plt
from torch.utils.data import TensorDataset, DataLoader
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

python

torch.manual_seed(42); np.random.seed(42)
X, y = make_moons(2000, noise=0.25, random_state=42)
Xtr, Xv, ytr, yv = train_test_split(X, y, test_size=0.2, random_state=42)
tx = lambda a: torch.tensor(a, dtype=torch.float32)
train_loader = DataLoader(TensorDataset(tx(Xtr), tx(ytr).view(-1, 1)),
                          batch_size=32, shuffle=True)
val_loader = DataLoader(TensorDataset(tx(Xv), tx(yv).view(-1, 1)),
                        batch_size=256, shuffle=False)

python

def run(opt_name, lr, epochs=60):
    torch.manual_seed(42)
    # fresh model, fair comparison
    model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
    criterion = nn.BCEWithLogitsLoss()
    optimizer = (torch.optim.SGD(model.parameters(), lr=lr) if opt_name == "SGD"
                 else torch.optim.Adam(model.parameters(), lr=lr))
    train_hist, val_hist = [], []
    for _ in range(epochs):
        model.train(); total = 0.0
        for xb, yb in train_loader:
            optimizer.zero_grad()                       # 1
            loss = criterion(model(xb), yb)             # 2 + 3
            loss.backward()                             # 4
            optimizer.step()                            # 5
            total += loss.item() * len(xb)
        train_hist.append(total / len(train_loader.dataset))
        model.eval(); val = correct = 0.0
        with torch.no_grad():
            for xb, yb in val_loader:
                out = model(xb)
                val += criterion(out, yb).item() * len(xb)
                correct += ((torch.sigmoid(out) > 0.5).float() == yb).sum().item()
                    # added
        val_hist.append(val / len(val_loader.dataset))
    return model, train_hist, val_hist, correct / len(val_loader.dataset)
for opt_name in ["SGD", "Adam"]:
    for lr in [0.1, 0.01, 0.001]:
        model, tr, va, acc = run(opt_name, lr)
        print(opt_name, lr, round(tr[-1], 3), round(va[-1], 3), round(acc, 3))
        plt.plot(va, label=opt_name + " " + str(lr))    # added: all six on one plot
plt.yscale("log"); plt.xlabel("epoch"); plt.ylabel("validation loss"); \
    plt.legend(); plt.show()

torch.save(model.state_dict(), "model.pt") # saves the LAST run; save your chosen one

Reading it

zero_grad() clears old gradients (PyTorch adds them up otherwise); backward() computes gradients; step() updates the weights; eval() + no_grad() switch off dropout-style behavior and save memory. Because the data has noise 0.25, even a perfect model cannot reach 100%.

Status Models and data: TESTED for architectures and training-loop behavior on shape-matched tensors; full training is VERIFY LOCALLY (MNIST download). The fit / evaluate helpers are an ADDED STARTER written for this guide.

Data and the two models

python

import torch, copy
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, random_split
tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,),
    (0.3081,))])
full = datasets.MNIST("data", train=True, download=True, transform=tf)
test = datasets.MNIST("data", train=False, download=True, transform=tf)
train, val = random_split(full, [55000, 5000],
    generator=torch.Generator().manual_seed(42))
train_loader = DataLoader(train, batch_size=64, shuffle=True)
val_loader = DataLoader(val, batch_size=256)
test_loader = DataLoader(test, batch_size=256)

python

class MLP(nn.Module):
    def __init__(self, p_drop=0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Flatten(),
            nn.Linear(784, 256), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(256, 128), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(128, 10))
    def forward(self, x): return self.net(x)

python

class CNN(nn.Module):
    def __init__(self, p_drop=0.3):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            # 16 x 14 x 14
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2))
            # 32 x 7 x 7
        self.classifier = nn.Sequential(
            nn.Flatten(), nn.Linear(32 * 7 * 7, 128), nn.ReLU(),
            nn.Dropout(p_drop), nn.Linear(128, 10))
    def forward(self, x): return self.classifier(self.features(x))

Early-stopping training helpers (added starter)

python

device = "cuda" if torch.cuda.is_available() else "cpu"
criterion = nn.CrossEntropyLoss()
# raw scores + integer labels, no softmax
def run_epoch(model, loader, optimizer=None):
    training = optimizer is not None
    model.train() if training else model.eval()
    total_loss = correct = n = 0
    with torch.set_grad_enabled(training):
        for xb, yb in loader:
            xb, yb = xb.to(device), yb.to(device)
            out = model(xb); loss = criterion(out, yb)
            if training:
                optimizer.zero_grad(); loss.backward(); optimizer.step()
            total_loss += loss.item() * len(xb)
            correct += (out.argmax(1) == yb).sum().item(); n += len(xb)
    return total_loss / n, correct / n

python

def fit(model, epochs=20, patience=3):
    model.to(device)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    best_val, best_state, bad, stopped = 1e9, None, 0, epochs
    history = {"train": [], "val": []}
    for ep in range(epochs):
        tl, _ = run_epoch(model, train_loader, opt)
        vl, va = run_epoch(model, val_loader)
        history["train"].append(tl); history["val"].append(vl)
        print(f"epoch {ep+1}: train {tl:.3f} val {vl:.3f} val acc {va:.4f}")
        if vl < best_val:
            best_val, best_state, bad = vl, copy.deepcopy(model.state_dict()), 0
        else:
            bad += 1
            if bad >= patience: stopped = ep + 1; break
    model.load_state_dict(best_state)
    # restore best weights BEFORE testing
    return history, stopped

python

for name, model in [("MLP", MLP()), ("CNN", CNN())]:
    history, stopped = fit(model)
    _, test_acc = run_epoch(model, test_loader)   # test ONCE, at the very end
    params = sum(p.numel() for p in model.parameters())
    print(name, "params", params, "epochs", stopped, "test acc", round(test_acc, 4))

Reading it

Expect the MLP at about 97-98% and the CNN at about 99%+. Why the CNN wins: it looks at small local patches and reuses the same filter everywhere (shared weights), so it understands shapes with fewer parameters. For the confusion matrix and the 5+ mistakes, collect argmax predictions over test_loader and use sklearn.metrics.confusion_matrix and plt.imshow.

Status The source guide marks the SelfAttention module, shape checks, row sums and causal-mask zeroing TESTED on random tensors; not re-run here (PyTorch unavailable). Pretrained BERT is optional and needs an external download. The heatmap is ADDED STARTER (not run).

SelfAttention + causal mask

python

import math, torch
import torch.nn as nn
import torch.nn.functional as F

python

class SelfAttention(nn.Module):
    def __init__(self, d_model, d_k):
        super().__init__()
        self.W_q = nn.Linear(d_model, d_k, bias=False)
        self.W_k = nn.Linear(d_model, d_k, bias=False)
        self.W_v = nn.Linear(d_model, d_k, bias=False)
        self.d_k = d_k
    def forward(self, x, mask=None):                     # x: (B, T, d_model)
        Q, K, V = self.W_q(x), self.W_k(x), self.W_v(x)  # each (B, T, d_k)
        scores = Q @ K.transpose(-2, -1) / math.sqrt(self.d_k)   # (B, T, T)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = F.softmax(scores, dim=-1)              # rows sum to 1
        return weights @ V, weights

python

torch.manual_seed(0)
x = torch.randn(1, 6, 16)
attn = SelfAttention(16, 16)
out, w = attn(x)
print(out.shape, w.shape)                 # (1, 6, 16) (1, 6, 6)
print(w[0].sum(dim=-1))                   # all ones

python

T = 6
mask = torch.tril(torch.ones(T, T)).unsqueeze(0)
_, w_masked = attn(x, mask=mask)
print("max future weight", w_masked[0].triu(1).max().item())   # 0.0

Heatmap (added)

python

import matplotlib.pyplot as plt
tokens = ["The", "cat", "sat", "on", "the", "mat"]
plt.imshow(w[0].detach().numpy(), cmap="viridis")
plt.xticks(range(6), tokens, rotation=45); plt.yticks(range(6), tokens)
plt.xlabel("Key (attended TO)"); plt.ylabel("Query (attending)")
plt.colorbar(label="attention weight"); plt.tight_layout(); plt.show()

Reading it

A random-looking heatmap is expected with untrained weights. We divide by sqrt(d_k) so big dot products do not push softmax into extreme values. The mask is applied to the scores before softmax: -inf becomes exactly 0 probability.

The capstone must be your own rebuild, so there is no answer key. Status: REFERENCE IMPLEMENTATION - VERIFY LOCALLY. Use the matching day's solution as your starting reference: Path A → Day 4, Path B → Days 9–10, Path C → Day 12, Path D → Day 13.

Blank-page skeleton (from the book)

python

# 1. imports, seed, device
# 2. data (train / val / test)
# 3. model
# 4. loss + optimizer
# 5. train loop + validation loop
# 6. test evaluation
# 7. plots
# 8. save model

Self-checks for Path C (MNIST CNN)

Checkpoint	Correct answer
Input shape of one MNIST batch of 64	(64, 1, 28, 28)
Model output shape	(64, 10): raw scores, one per digit
Predicted digit	output.argmax(1)
Loss function	CrossEntropyLoss (contains softmax; labels are integer class numbers)
Order inside the loop	zero_grad, forward, loss, backward, step
Expected test accuracy	about 99%+ for a small CNN

Hand check from Day 13 (practice)

Weights [0.20, 0.40, 0.40] times values [1,0], [0,1], [1,1]: 0.20·[1,0] + 0.40·[0,1] + 0.40·[1,1] = [0.60, 0.80].

Optional: command-line options

python

import argparse
p = argparse.ArgumentParser()
p.add_argument("--lr", type=float, default=1e-3)
p.add_argument("--epochs", type=int, default=10)
args = p.parse_args()
# run with: python train.py --lr 0.001 --epochs 10

Official documentation, foundational papers, and datasets used while preparing this book

Official documentation

NumPy Developers. NumPy Documentation. https://numpy.org/doc/

pandas Development Team. pandas Documentation. https://pandas.pydata.org/docs/

scikit-learn Developers. scikit-learn User Guide. https://scikit-learn.org/stable/user_guide.html

PyTorch Contributors. PyTorch Documentation and Tutorials. https://pytorch.org/docs/

LangChain. LangChain and LangGraph Documentation. https://docs.langchain.com/

Model Context Protocol. Model Context Protocol Documentation. https://modelcontextprotocol.io/

Ragas. Ragas Documentation. https://docs.ragas.io/

Foundational papers

Vaswani, A. et al. (2017). Attention Is All You Need. https://arxiv.org/abs/1706.03762

Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805

Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI.

Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980

Srivastava, N. et al. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR, 15, 1929–1958.

Ioffe, S. & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training. https://arxiv.org/abs/1502.03167

He, K. et al. (2016). Deep Residual Learning for Image Recognition (ResNet). https://arxiv.org/abs/1512.03385 Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. https://arxiv.org/abs/2005.11401

Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685

Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. https://arxiv.org/abs/2305.14314

Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. https://arxiv.org/abs/2203.02155

Rafailov, R. et al. (2023). Direct Preference Optimization. https://arxiv.org/abs/2305.18290

Es, S. et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. https://arxiv.org/abs/2309.15217

Johnson, J., Douze, M. & Jégou, H. (2017). Billion-scale Similarity Search with GPUs (FAISS). https://arxiv.org/abs/1702.08734

Datasets

LeCun, Y., Cortes, C. & Burges, C. The MNIST Database of Handwritten Digits. https://yann.lecun.com/exdb/mnist/

Kaggle. Telco Customer Churn (IBM sample data). https://www.kaggle.com/

Kaggle. Credit Card Fraud Detection (ULB Machine Learning Group). https://www.kaggle.com/

Added for Part V (project walkthroughs)

Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. https://arxiv.org/abs/1908.10084 Sentence Transformers documentation. https://www.sbert.net/ pypdf documentation. https://pypdf.readthedocs.io/ FastAPI documentation. https://fastapi.tiangolo.com/ Docker documentation. https://docs.docker.com/ Model Context Protocol. MCP Python SDK. https://py.sdk.modelcontextprotocol.io/ Anthropic. Claude API documentation (current model names). https://docs.claude.com/ Kaggle. Titanic: Machine Learning from Disaster. https://www.kaggle.com/c/titanic scikit-learn. The 20 newsgroups text dataset. https://scikit-learn.org/stable/datasets/real_world.html

Chapter 27 of 33

Solutions Appendix and References

Stop. Have you tried Level 1 and the hints? If yes, go ahead. Read one solution, understand each line, close it, and type your own version from memory.

Notes on the solutions

TESTED: run on the exact generated data from the book (no download needed).
VERIFY LOCALLY: needs an external dataset or model download; run it yourself and expect small differences.
ADDED STARTER: a short snippet this guide added for a required step the source solution did not show. Comments marked # added or # fix show where this guide changed the source code. Check shapes first if something errors.
Your exact numbers will differ slightly from the book's (e.g. Day 4: about w=2.77, b=4.21). That is normal.

Status TESTED for the cosine math on the book shapes. Titanic part: VERIFY LOCALLY on the real titanic.csv.

Assignment A: cosine similarity (no loops)

import numpy as np

python

def cosine_sim_matrix(A, B):
    A_norm = A / np.linalg.norm(A, axis=1, keepdims=True)
    B_norm = B / np.linalg.norm(B, axis=1, keepdims=True)
    return A_norm @ B_norm.T

python

# scalar check: should print about 0.707
a, b = np.array([1, 0]), np.array([1, 1])
print(round(float(cosine_sim_matrix(a[None, :], b[None, :])[0, 0]), 3))

python

A = np.random.rand(4, 8)
B = np.random.rand(5, 8)
S = cosine_sim_matrix(A, B)
print(S.shape)                                    # (4, 5)

python

# added: loop version, kept separate, ONLY to prove the fast one is correct
S_slow = np.zeros((4, 5))
for i in range(4):
    for j in range(5):
        S_slow[i, j] = A[i] @ B[j] / (np.linalg.norm(A[i]) * np.linalg.norm(B[j]))
print(np.allclose(S, S_slow))                     # True
print(np.allclose(np.diag(cosine_sim_matrix(A, A)), 1))   # True

Assignment B: Titanic cleaning

import pandas as pd

python

df = pd.read_csv("titanic.csv")
print(df.shape); df.info(); print(df.isna().sum())

python

df["Age"] = df["Age"].fillna(df["Age"].median())
df["Embarked"] = df["Embarked"].fillna(df["Embarked"].mode()[0])
df = df.drop(columns=["Cabin", "Ticket", "Name", "PassengerId"])
df["Sex"] = df["Sex"].map({"male": 0, "female": 1})
df = pd.get_dummies(df, columns=["Embarked"], drop_first=True, dtype=int)
# added dtype=int
assert df.isna().sum().sum() == 0                 # safety net
df.to_csv("titanic_clean.csv", index=False)

python

print(df.groupby("Sex")["Survived"].mean())
print(df.groupby("Pclass")["Survived"].mean())

Reading it

Status TESTED on the exact generated 100-point setup; no external data. The plotting of the fitted line is an added starter.

Gradient descent from scratch vs scikit-learn

python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

python

w, b = 0.0, 0.0
lr, epochs, n = 0.1, 1000, len(X)
losses = []
for _ in range(epochs):
    y_pred = w * X + b                    # 1. predict
    error = y_pred - y                    # 2. how wrong
    loss = (error ** 2).mean()            # 3. MSE
    losses.append(loss)
    dw = (2 / n) * (X * error).sum()      # 4. gradients
    db = (2 / n) * error.sum()
    w -= lr * dw                          # 5. step downhill
    b -= lr * db
print(w, b)

plt.plot(losses); plt.xlabel("Epoch"); plt.ylabel("MSE"); plt.title("Loss curve"); plt.show()

python

model = LinearRegression().fit(X, y)
print(model.coef_, model.intercept_)

python

# added: data and fitted line
plt.scatter(X, y, s=10)
xs = np.array([[0.0], [2.0]])
plt.plot(xs, w * xs + b, "r", label="my line"); plt.legend(); plt.title("Fit"); \
    plt.show()

Reading it

Expect w near 2.77 and b near 4.21, not exactly 3 and 4, because the random noise nudges the best-fit line. Your numbers and scikit-learn's should match to about 2 decimals.

Status TESTED on the exact 60-point generated data; the required CV result had all four models. The degree plot is an added starter.

Degree sweep, degree-15 comparison and 5-fold CV

python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error

python

np.random.seed(0)
X = np.sort(np.random.rand(60, 1) * 6 - 3, axis=0)
y = (0.5 * X**3 - X**2 + 2 * X + np.random.randn(60, 1) * 3).ravel()
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3,
    random_state=1)

python

train_err, val_err = [], []
for d in range(1, 16):
    m = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression())
    m.fit(X_train, y_train)
    train_err.append(mean_squared_error(y_train, m.predict(X_train)))
    val_err.append(mean_squared_error(y_val, m.predict(X_val)))
best = int(np.argmin(val_err)) + 1
print("best degree", best)

python

# added: the U-shaped plot
plt.plot(range(1, 16), train_err, "o-", label="train")
plt.plot(range(1, 16), val_err, "s-", label="validation")
plt.axvline(best, ls="--", c="gray"); plt.yscale("log")
plt.xlabel("Polynomial degree"); plt.ylabel("MSE"); plt.legend(); plt.show()

python

# degree-15 validation comparison (required)
for name, reg in [("Linear", LinearRegression()), ("Ridge", Ridge(alpha=1)),
                  ("Lasso", Lasso(alpha=0.1, max_iter=50000))]:
    m = make_pipeline(PolynomialFeatures(15), StandardScaler(), reg).fit(X_train,
        y_train)
    print(name, mean_squared_error(y_val, m.predict(X_val)))
# 5-fold CV for all four models (required)
models = [
    ("Deg3 Linear",  make_pipeline(PolynomialFeatures(3), StandardScaler(),
        LinearRegression())),
    ("Deg15 Linear", make_pipeline(PolynomialFeatures(15), StandardScaler(),
        LinearRegression())),
    ("Deg15 Ridge",  make_pipeline(PolynomialFeatures(15), StandardScaler(),
        Ridge(alpha=1))),
    ("Deg15 Lasso",  make_pipeline(PolynomialFeatures(15), StandardScaler(),
                                   Lasso(alpha=0.1, max_iter=50000))),
]
for name, model in models:
    scores = -cross_val_score(model, X, y, cv=5, scoring="neg_mean_squared_error")
    print(name, scores.mean(), scores.std())

Reading it

Status REFERENCE IMPLEMENTATION - VERIFY LOCALLY on creditcard.csv (external dataset).

Three-model comparison

python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import (confusion_matrix, precision_score, recall_score,
                             f1_score, roc_auc_score)

python

df = pd.read_csv("creditcard.csv")
print(df["Class"].value_counts(normalize=True))   # about 0.17% fraud
X = df.drop(columns=["Class"]); y = df["Class"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
X_train, X_test = X_train.copy(), X_test.copy()
# added: avoids SettingWithCopy warnings
scaler = StandardScaler().fit(X_train[["Amount", "Time"]])
# learn from TRAIN only
X_train[["Amount", "Time"]] = scaler.transform(X_train[["Amount", "Time"]])
X_test[["Amount", "Time"]] = scaler.transform(X_test[["Amount", "Time"]])

python

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000,
        class_weight="balanced"),
    "Random Forest": RandomForestClassifier(n_estimators=100,
        class_weight="balanced",
                                            n_jobs=-1, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(random_state=42),
}
rows = []
for name, model in models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    prob = model.predict_proba(X_test)[:, 1]      # probabilities for AUC
    rows.append({"Model": name,
                 "Precision": precision_score(y_test, pred),
                 "Recall": recall_score(y_test, pred),
                 "F1": f1_score(y_test, pred),
                 "ROC-AUC": roc_auc_score(y_test, prob)})
    print(name); print(confusion_matrix(y_test, pred))    # [[TN, FP], [FN, TP]]
print(pd.DataFrame(rows).round(3))

Reading it

Telco churn pipeline (Part F)

python

import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold, \
    cross_validate
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
# fix: fill blanks inside the pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

python

df = pd.read_csv("telco.csv")
# renamed download (see Start Here)
print(df.shape, df["Churn"].value_counts(normalize=True))
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")
df = df.drop(columns=["customerID"])
df["Churn"] = df["Churn"].map({"Yes": 1, "No": 0})
X = df.drop(columns=["Churn"]); y = df["Churn"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

python

num_cols = ["tenure", "MonthlyCharges", "TotalCharges"]
cat_cols = [c for c in X.columns if c not in num_cols]
pre = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc", StandardScaler())]), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
])
models = {
    "LogReg": LogisticRegression(max_iter=1000, class_weight="balanced"),
    "RandomForest": RandomForestClassifier(n_estimators=300,
        class_weight="balanced",
                                           random_state=42),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for name, model in models.items():
    pipe = Pipeline([("pre", pre), ("clf", model)])
r = cross_validate(pipe, X_train, y_train, cv=cv,
                   scoring=["precision", "recall", "f1", "roc_auc"])
print(name, {k: round(v.mean(), 3) for k, v in r.items() if
    k.startswith("test_")})

Then: choose by CV, test once, and show top features (added starter)

python

from sklearn.metrics import classification_report, roc_auc_score
import pandas as pd

python

best = Pipeline([("pre", pre), ("clf", models["LogReg"])])
# replace with the CV winner
best.fit(X_train, y_train)
pred = best.predict(X_test); prob = best.predict_proba(X_test)[:, 1]
print(classification_report(y_test, pred)); print("ROC-AUC", roc_auc_score(y_test,
    prob))

python

rf = Pipeline([("pre", pre), ("clf", models["RandomForest"])]).fit(X_train, y_train)
names = rf.named_steps["pre"].get_feature_names_out()
imp = pd.Series(rf.named_steps["clf"].feature_importances_, index=names)
imp.sort_values().tail(10).plot.barh(title="Top 10 features")

Parts A-E: checkpoint starters (ADDED STARTER)

python

import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

python

# A. distances for P1=(25,6), P2=(30,8)
p1, p2 = np.array([25, 6]), np.array([30, 8])
print("euclidean", np.linalg.norm(p1 - p2), "manhattan", np.abs(p1 - p2).sum(),
      "cosine", p1 @ p2 / (np.linalg.norm(p1) * np.linalg.norm(p2)))
# B. k-NN with and without scaling on a tiny made-up matrix (age, income)
rng = np.random.RandomState(0)
X_small = np.c_[rng.randint(20, 60, 40), rng.randint(20000, 100000, 40)]
y_small = (X_small[:, 0] > 40).astype(int)
for name, Xm in [("raw", X_small), ("scaled",
    StandardScaler().fit_transform(X_small))]:
    knn = KNeighborsClassifier(n_neighbors=5).fit(Xm, y_small)
    print(name, "accuracy", knn.score(Xm, y_small))

python

# C + D. k-Means elbow and PCA on the scaled Telco numeric columns
num = df[["tenure", "MonthlyCharges", "TotalCharges"]]
Xs = StandardScaler().fit_transform(num.fillna(num.median()))
inertia = [KMeans(k, n_init=10, random_state=42).fit(Xs).inertia_ for k in range(1,
    11)]
plt.plot(range(1, 11), inertia, "o-"); plt.xlabel("k"); plt.ylabel("inertia"); \
    plt.show()
pca = PCA(n_components=2); X2 = pca.fit_transform(Xs)
print(X2.shape, pca.explained_variance_ratio_)

python

# E. semantic search (paste your Days 2–3 cosine_sim_matrix above this line)
from sentence_transformers import SentenceTransformer
st = SentenceTransformer("all-MiniLM-L6-v2")
docs = ["How to reset my password", "Best pizza recipes", "Change account login "
   "credentials"]
D_emb = st.encode(docs); q_emb = st.encode(["I forgot my password"])
print(D_emb.shape, cosine_sim_matrix(q_emb, D_emb))
# top score = winning sentence

Reading it

Part B: with raw numbers the income column dominates every distance, so scaling changes who the neighbors are. Real-data expectation for Telco: ROC-AUC roughly 0.80 to 0.85.

Two-layer network in NumPy

python

import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

python

X, y = make_moons(1000, noise=0.2, random_state=42)
y = y.reshape(-1, 1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
    random_state=42)

python

np.random.seed(0)
n_in, n_hidden, n_out = 2, 16, 1
W1 = np.random.randn(n_in, n_hidden) * np.sqrt(2 / n_in);     b1 = np.zeros((1,
    n_hidden))
W2 = np.random.randn(n_hidden, n_out) * np.sqrt(2 / n_hidden); b2 = np.zeros((1,
    n_out))
lr, epochs, eps = 0.1, 3000, 1e-8

python

def relu(z): return np.maximum(0, z)
def relu_grad(z): return (z > 0).astype(float)
def sigmoid(z): return 1 / (1 + np.exp(-z))
def forward(X):
    Z1 = X @ W1 + b1; A1 = relu(Z1); Z2 = A1 @ W2 + b2; A2 = sigmoid(Z2)
    return Z1, A1, Z2, A2
def bce(y, a): return -np.mean(y * np.log(a + eps) + (1 - y) * np.log(1 - a + eps))

python

losses = []
for epoch in range(epochs):
    Z1, A1, Z2, A2 = forward(X_train)
    loss = bce(y_train, A2); losses.append(loss); n = len(X_train)
    dZ2 = A2 - y_train                                  # backward: start at the end
    dW2 = A1.T @ dZ2 / n; db2 = dZ2.sum(axis=0, keepdims=True) / n
    dA1 = dZ2 @ W2.T; dZ1 = dA1 * relu_grad(Z1)
    # ReLU passes gradient where Z1 > 0
    dW1 = X_train.T @ dZ1 / n; db1 = dZ1.sum(axis=0, keepdims=True) / n
if epoch == 0:
# added: gradient check on W1[0,0]
    h, old = 1e-5, W1[0, 0]
    W1[0, 0] = old + h; lp = bce(y_train, forward(X_train)[3])
    W1[0, 0] = old - h; lm = bce(y_train, forward(X_train)[3]); W1[0, 0] = old
    print("analytic", dW1[0, 0], "numeric", (lp - lm) / (2 * h))

W1 -= lr * dW1; b1 -= lr * db1; W2 -= lr * dW2; b2 -= lr * db2

python

print("loss", losses[0], "->", losses[-1])
print("test acc", ((forward(X_test)[3] > 0.5) == y_test).mean())

No-hidden-layer baseline (added)

python

from sklearn.linear_model import LogisticRegression
base = LogisticRegression().fit(X_train, y_train.ravel())
print("no hidden layer:", base.score(X_test, y_test.ravel()))
# about 0.85: straight line

Reading it

Six-run optimizer grid + saving weights

python

import numpy as np, torch
import torch.nn as nn
import matplotlib.pyplot as plt
from torch.utils.data import TensorDataset, DataLoader
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

python

torch.manual_seed(42); np.random.seed(42)
X, y = make_moons(2000, noise=0.25, random_state=42)
Xtr, Xv, ytr, yv = train_test_split(X, y, test_size=0.2, random_state=42)
tx = lambda a: torch.tensor(a, dtype=torch.float32)
train_loader = DataLoader(TensorDataset(tx(Xtr), tx(ytr).view(-1, 1)),
                          batch_size=32, shuffle=True)
val_loader = DataLoader(TensorDataset(tx(Xv), tx(yv).view(-1, 1)),
                        batch_size=256, shuffle=False)

python

def run(opt_name, lr, epochs=60):
    torch.manual_seed(42)
    # fresh model, fair comparison
    model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
    criterion = nn.BCEWithLogitsLoss()
    optimizer = (torch.optim.SGD(model.parameters(), lr=lr) if opt_name == "SGD"
                 else torch.optim.Adam(model.parameters(), lr=lr))
    train_hist, val_hist = [], []
    for _ in range(epochs):
        model.train(); total = 0.0
        for xb, yb in train_loader:
            optimizer.zero_grad()                       # 1
            loss = criterion(model(xb), yb)             # 2 + 3
            loss.backward()                             # 4
            optimizer.step()                            # 5
            total += loss.item() * len(xb)
        train_hist.append(total / len(train_loader.dataset))
        model.eval(); val = correct = 0.0
        with torch.no_grad():
            for xb, yb in val_loader:
                out = model(xb)
                val += criterion(out, yb).item() * len(xb)
                correct += ((torch.sigmoid(out) > 0.5).float() == yb).sum().item()
                    # added
        val_hist.append(val / len(val_loader.dataset))
    return model, train_hist, val_hist, correct / len(val_loader.dataset)
for opt_name in ["SGD", "Adam"]:
    for lr in [0.1, 0.01, 0.001]:
        model, tr, va, acc = run(opt_name, lr)
        print(opt_name, lr, round(tr[-1], 3), round(va[-1], 3), round(acc, 3))
        plt.plot(va, label=opt_name + " " + str(lr))    # added: all six on one plot
plt.yscale("log"); plt.xlabel("epoch"); plt.ylabel("validation loss"); \
    plt.legend(); plt.show()

torch.save(model.state_dict(), "model.pt") # saves the LAST run; save your chosen one

Reading it

Data and the two models

python

import torch, copy
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, random_split
tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,),
    (0.3081,))])
full = datasets.MNIST("data", train=True, download=True, transform=tf)
test = datasets.MNIST("data", train=False, download=True, transform=tf)
train, val = random_split(full, [55000, 5000],
    generator=torch.Generator().manual_seed(42))
train_loader = DataLoader(train, batch_size=64, shuffle=True)
val_loader = DataLoader(val, batch_size=256)
test_loader = DataLoader(test, batch_size=256)

python

class MLP(nn.Module):
    def __init__(self, p_drop=0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Flatten(),
            nn.Linear(784, 256), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(256, 128), nn.ReLU(), nn.Dropout(p_drop),
            nn.Linear(128, 10))
    def forward(self, x): return self.net(x)

python

class CNN(nn.Module):
    def __init__(self, p_drop=0.3):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            # 16 x 14 x 14
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2))
            # 32 x 7 x 7
        self.classifier = nn.Sequential(
            nn.Flatten(), nn.Linear(32 * 7 * 7, 128), nn.ReLU(),
            nn.Dropout(p_drop), nn.Linear(128, 10))
    def forward(self, x): return self.classifier(self.features(x))

Early-stopping training helpers (added starter)

python

device = "cuda" if torch.cuda.is_available() else "cpu"
criterion = nn.CrossEntropyLoss()
# raw scores + integer labels, no softmax
def run_epoch(model, loader, optimizer=None):
    training = optimizer is not None
    model.train() if training else model.eval()
    total_loss = correct = n = 0
    with torch.set_grad_enabled(training):
        for xb, yb in loader:
            xb, yb = xb.to(device), yb.to(device)
            out = model(xb); loss = criterion(out, yb)
            if training:
                optimizer.zero_grad(); loss.backward(); optimizer.step()
            total_loss += loss.item() * len(xb)
            correct += (out.argmax(1) == yb).sum().item(); n += len(xb)
    return total_loss / n, correct / n

python

def fit(model, epochs=20, patience=3):
    model.to(device)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    best_val, best_state, bad, stopped = 1e9, None, 0, epochs
    history = {"train": [], "val": []}
    for ep in range(epochs):
        tl, _ = run_epoch(model, train_loader, opt)
        vl, va = run_epoch(model, val_loader)
        history["train"].append(tl); history["val"].append(vl)
        print(f"epoch {ep+1}: train {tl:.3f} val {vl:.3f} val acc {va:.4f}")
        if vl < best_val:
            best_val, best_state, bad = vl, copy.deepcopy(model.state_dict()), 0
        else:
            bad += 1
            if bad >= patience: stopped = ep + 1; break
    model.load_state_dict(best_state)
    # restore best weights BEFORE testing
    return history, stopped

python

for name, model in [("MLP", MLP()), ("CNN", CNN())]:
    history, stopped = fit(model)
    _, test_acc = run_epoch(model, test_loader)   # test ONCE, at the very end
    params = sum(p.numel() for p in model.parameters())
    print(name, "params", params, "epochs", stopped, "test acc", round(test_acc, 4))

Reading it

SelfAttention + causal mask

python

import math, torch
import torch.nn as nn
import torch.nn.functional as F

python

class SelfAttention(nn.Module):
    def __init__(self, d_model, d_k):
        super().__init__()
        self.W_q = nn.Linear(d_model, d_k, bias=False)
        self.W_k = nn.Linear(d_model, d_k, bias=False)
        self.W_v = nn.Linear(d_model, d_k, bias=False)
        self.d_k = d_k
    def forward(self, x, mask=None):                     # x: (B, T, d_model)
        Q, K, V = self.W_q(x), self.W_k(x), self.W_v(x)  # each (B, T, d_k)
        scores = Q @ K.transpose(-2, -1) / math.sqrt(self.d_k)   # (B, T, T)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = F.softmax(scores, dim=-1)              # rows sum to 1
        return weights @ V, weights

python

torch.manual_seed(0)
x = torch.randn(1, 6, 16)
attn = SelfAttention(16, 16)
out, w = attn(x)
print(out.shape, w.shape)                 # (1, 6, 16) (1, 6, 6)
print(w[0].sum(dim=-1))                   # all ones

python

T = 6
mask = torch.tril(torch.ones(T, T)).unsqueeze(0)
_, w_masked = attn(x, mask=mask)
print("max future weight", w_masked[0].triu(1).max().item())   # 0.0

Heatmap (added)

python

import matplotlib.pyplot as plt
tokens = ["The", "cat", "sat", "on", "the", "mat"]
plt.imshow(w[0].detach().numpy(), cmap="viridis")
plt.xticks(range(6), tokens, rotation=45); plt.yticks(range(6), tokens)
plt.xlabel("Key (attended TO)"); plt.ylabel("Query (attending)")
plt.colorbar(label="attention weight"); plt.tight_layout(); plt.show()

Reading it

Blank-page skeleton (from the book)

python

# 1. imports, seed, device
# 2. data (train / val / test)
# 3. model
# 4. loss + optimizer
# 5. train loop + validation loop
# 6. test evaluation
# 7. plots
# 8. save model

Self-checks for Path C (MNIST CNN)

Checkpoint	Correct answer
Input shape of one MNIST batch of 64	(64, 1, 28, 28)
Model output shape	(64, 10): raw scores, one per digit
Predicted digit	output.argmax(1)
Loss function	CrossEntropyLoss (contains softmax; labels are integer class numbers)
Order inside the loop	zero_grad, forward, loss, backward, step
Expected test accuracy	about 99%+ for a small CNN

Hand check from Day 13 (practice)

Weights [0.20, 0.40, 0.40] times values [1,0], [0,1], [1,1]: 0.20·[1,0] + 0.40·[0,1] + 0.40·[1,1] = [0.60, 0.80].

Optional: command-line options

python

import argparse
p = argparse.ArgumentParser()
p.add_argument("--lr", type=float, default=1e-3)
p.add_argument("--epochs", type=int, default=10)
args = p.parse_args()
# run with: python train.py --lr 0.001 --epochs 10