Day 4 · Linear Regression + Gradient Descent

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ The straight line y = w·x + b (Day 1, Math 2) ☐ Averages and squared gaps (Day 1, Math 1 and 5) ☐ The slope idea from Day 1, Math 7 (derivative intuition) ☐ Day 2: arrays and @ If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Understand how a machine “learns”. By the end you will build a learning algorithm from scratch and watch it improve.

Big picture — read this first A model is not “intelligent” during training. It starts with parameters, makes predictions, measures mistakes, and adjusts the parameters. Gradient descent is repeated small correction. The learning rate controls the size of each correction. Use this story: *data → prediction → loss → gradient → parameter update → better prediction*. Do not memorize the gradient formula before understanding what the gradient is telling you: the direction in which the loss changes.

Key idea Gradient descent is the core idea behind training most deep learning models, LLMs included. Master it today and everything later gets easier.

1. What is Machine Learning? (from zero)

Normal programming: You write rules → computer applies rules to data → answer. Machine learning: You give data + answers → computer finds the rules by itself.

Example: predict house price.

• Normal programming: you write price = 5000 * size + 2 lakh (you must know the rule).
• ML: you show 1000 houses with their sizes and prices. The computer figures out the formula.

1.1 Types of ML

Today = supervised learning.

1.2 Key words

• Sample / row / example: one item (one house)
• Features (X): the input columns (size, bedrooms, location) — what we know
• Target / label (y): the answer we want to predict (price)
• Model: the formula/machine that maps features → prediction
• Training: adjusting the model so its predictions match the real answers
• Parameters: numbers inside the model that training adjusts (weights, bias)

1.3 Two kinds of supervised tasks

• Regression: predict a number (price, temperature, age)
• Classification: predict a category (spam/not spam, cat/dog)

Worked Problem · Learn A Rule From Examples

Problem. Study hours [1, 2, 3, 4] led to scores [50, 60, 70, 80]. Let the computer learn the rule, then predict the score for 5 hours.

1. Classic programming writes the rule by hand. Machine learning finds the rule from examples.
2. Here the scores rise by 10 for every hour, so the rule is score = 10 × hours + 40.
3. A fitting tool finds this automatically: np.polyfit.
4. Use the learned rule on new input: 10 × 5 + 40.

import numpy as np
w, b = np.polyfit([1, 2, 3, 4], [50, 60, 70, 80], 1)
print(round(w, 2), round(b, 2), round(w * 5 + b, 2))

10.0 40.0 90.0

Answer. Learned w = 10, b = 40, so 5 hours predicts 90.

Worked Problem · Regression Or Classification?

Problem. Label each task: (a) predict a house price, (b) decide if an email is spam, (c) predict tomorrow's temperature, (d) tell cat from dog in a photo.

1. Ask: is the answer a number on a scale or a category?
2. (a) price is a number → regression.
3. (b) spam / not spam is a category → classification.
4. (c) temperature is a number → regression.
5. (d) cat / dog is a category → classification.

Answer. (a) regression, (b) classification, (c) regression, (d) classification.

2. Linear Regression — the simplest model

2.1 The idea

Draw the best straight line through your data points.

price
  |            •
  |        • /
  |      •/ •
  |    /•
  |  /•
  |/________ size

2.2 The equation (remember school: y = mx + c)

ŷ = w · x + b

Example: ŷ = 3x + 4

• x = 2 → ŷ = 3×2 + 4 = 10
• x = 5 → ŷ = 19

Learning = finding the best w and b.

2.3 Many features

ŷ = w₁x₁ + w₂x₂ + w₃x₃ + b

House: size (x₁), bedrooms (x₂), age (x₃). Each weight says how important that feature is.

In matrix form for ALL samples at once:

y_pred = X @ w + b       # X shape (n_samples, n_features), w shape (n_features,)

Worked Problem · Fit A Line And Predict

Problem. House sizes are x = [1, 2, 3] and prices are y = [3, 5, 7]. Find the line y = w·x + b that fits and predict the price for size 4.

1. Look at the pattern: each time x grows by 1, y grows by 2. So the slope is w = 2.
2. Find b: when x = 1, y = 3, so 3 = 2×1 + b, which gives b = 1.
3. Check another point: x = 3 gives 2×3 + 1 = 7 ✓.
4. Predict for x = 4: 2×4 + 1 = 9.
5. np.polyfit(x, y, 1) finds the best line automatically. Use it to confirm.

import numpy as np
x = np.array([1, 2, 3]); y = np.array([3, 5, 7])
w, b = np.polyfit(x, y, 1)
print(round(w, 3), round(b, 3))
print(round(w * 4 + b, 3))

2.0 1.0
9.0

Answer. w = 2, b = 1, so the predicted price for size 4 is 9.

Your Turn · Read A Fitted Line

Problem. A model learned price = 0.5 × size + 10. What does the weight mean, and what price does it predict for size 20?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. The weight 0.5 is the slope: each extra unit of size adds 0.5 to the price.
2. The bias 10 is the price when size is 0.
3. For size 20: 0.5 × 20 + 10 = 10 + 10 = 20.

Answer. Each extra unit of size adds 0.5 to the price, and the prediction for size 20 is 20.

3. Loss Function — “How wrong are we?”

To improve, we must measure how bad the current line is. That measurement is the loss (also called cost or error). Lower loss = better model.

3.1 Error for one point

error = ŷ − y (prediction − truth)

Worked Problem · Sign Of The Error

Problem. The true value is 5. The model predicts 7, and for another point predicts 3. Find error = prediction − truth for each.

1. First point: 7 − 5 = +2.
2. Second point: 3 − 5 = −2.
3. A positive error means the model guessed too high; negative means too low.

Answer. Errors +2 (too high) and −2 (too low).

3.2 Why not just add up errors?

Errors of +5 and −5 would cancel to 0 (looks perfect but isn’t!). So we square them:

• Always positive
• Big mistakes are punished much more (error 10 → 100, error 2 → 4)

Worked Problem · Errors That Cancel

Problem. A model's errors on two points are +5 and −5. Compare the sum of errors with the sum of squared errors.

1. Plain sum: 5 + (−5) = 0, which falsely looks perfect.
2. Squared errors: 25 + 25 = 50.
3. Squaring makes every error positive, so errors cannot cancel.

e = [5, -5]
print(sum(e), sum(v ** 2 for v in e))

0 50

Answer. Sum 0, sum of squares 50.

3.3 MSE — Mean Squared Error

MSE = (1/n) · Σ (ŷ − y)²

“Square each error, then take the average.”

3.4 Worked example (do this by hand!)

True values y = [3, 5, 7]. Model predicts ŷ = [2, 6, 10].

MSE = (1 + 1 + 9) / 3 = 3.67

3.5 Other losses

mse = np.mean((y_pred - y) ** 2)

Worked Problem · Compute Mae, Mse And Rmse

Problem. True values are [10, 20, 30] and the model predicts [12, 18, 33]. Compute MAE, MSE and RMSE.

1. Errors (prediction − truth): +2, −2, +3.
2. MAE averages the absolute errors: (2 + 2 + 3) ÷ 3 = 7 ÷ 3 ≈ 2.333.
3. MSE averages the squared errors: (4 + 4 + 9) ÷ 3 = 17 ÷ 3 ≈ 5.667.
4. RMSE is the square root of MSE: √5.667 ≈ 2.380, in the same units as y.
5. Check with code.

import numpy as np
y = np.array([10, 20, 30]); p = np.array([12, 18, 33])
e = p - y
print(round(np.abs(e).mean(), 3))
print(round((e ** 2).mean(), 3))
print(round(np.sqrt((e ** 2).mean()), 3))

2.333
5.667
2.38

Answer. MAE 2.333, MSE 5.667, RMSE 2.38.

Your Turn · Why Squaring Punishes Big Mistakes

Problem. Model A makes four errors of size 1. Model B makes three errors of 0 and one error of 4. Compare their MAE and MSE.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Model A errors: [1, 1, 1, 1]. MAE = 4 ÷ 4 = 1. MSE = 4 ÷ 4 = 1.
2. Model B errors: [0, 0, 0, 4]. MAE = 4 ÷ 4 = 1. MSE = 16 ÷ 4 = 4.
3. MAE treats both equally, but MSE says B is four times worse because it has one big miss.

import numpy as np
for e in ([1, 1, 1, 1], [0, 0, 0, 4]):
    e = np.array(e)
    print(np.abs(e).mean(), (e ** 2).mean())

1.0 1.0
1.0 4.0

Answer. A: MAE 1.0, MSE 1.0. B: MAE 1.0, MSE 4.0. MSE is the loss to pick when large errors are especially bad.

4. Gradient Descent — How the Machine Learns ★★★

4.1 The mountain analogy

You’re on a foggy mountain, blindfolded. You want to reach the valley (lowest point). What do you do?

1. Feel the ground to find which direction slopes downward 2. Take a small step that way 3. Repeat until the ground is flat

• The height = loss
• Your position = the values of w and b
• The slope = gradient
• The step size = learning rate

4.2 The loss curve

Plot the loss for different values of w (one parameter):

loss
 |\               /
 | \             /
 |  \           /
 |   \_________/
 |        ^
 |     minimum (best w)
 +------------------- w

It’s a bowl shape (for linear regression MSE, there’s only one bottom — no traps!).

4.3 What’s a gradient? (no scary math)

The gradient (derivative) answers: “If I increase w slightly, does the loss go up or down, and how fast?”

• Gradient positive → loss goes up as w goes up → so decrease w
• Gradient negative → loss goes down as w goes up → so increase w
• Gradient ≈ 0 → you’re at the bottom

So we always move opposite to the gradient.

4.4 The update rule (MEMORIZE)

w = w − α · (∂Loss/∂w)
b = b − α · (∂Loss/∂b)

• α (alpha) = learning rate — step size
• ∂Loss/∂w = gradient of loss with respect to w

4.5 The gradients for linear regression with MSE

Let error = ŷ − y and n = number of samples:

∂Loss/∂w = (2/n) · Σ (error · x)
∂Loss/∂b = (2/n) · Σ (error)

(You can skip the derivation for now; these formulas come from calculus/chain rule.)

4.6 ★ A full worked example — ONE step by hand

Data: x = [1, 2, 3], y = [2, 4, 6] (true relation: y = 2x). Start: w = 0, b = 0. Learning rate α = 0.1.

Step 1 — predict: ŷ = 0·x + 0 = [0, 0, 0]

Step 2 — errors: ŷ − y = [−2, −4, −6]

Step 3 — loss: MSE = (4 + 16 + 36)/3 = 18.67

Step 4 — gradients:

• dw = (2/3) · [(−2·1) + (−4·2) + (−6·3)] = (2/3)(−28) = −18.67
• db = (2/3) · (−2 −4 −6) = (2/3)(−12) = −8

Step 5 — update:

• w = 0 − 0.1 × (−18.67) = 1.867
• b = 0 − 0.1 × (−8) = 0.8

Check the new predictions: ŷ = 1.867x + 0.8 = [2.67, 4.53, 6.4] — much closer to [2, 4, 6]! Repeat many times → w → 2, b → 0. That’s learning!

4.7 Vocabulary

4.8 Learning rate — the most important setting

Too small (0.00001)      Just right (0.1)       Too large (5)
 loss                     loss                   loss
 |\                       |\                     |  /\  /\
 | \                      | \                    | /  \/  \  (explodes)
 |  \  (super slow)       |  \___ (smooth)       |/
 +------ epochs           +------ epochs        +------ epochs

Try: 0.001, 0.01, 0.1, 1.

4.9 Feature scaling

If feature 1 ranges 0–1 and feature 2 ranges 0–100000, gradient descent zigzags and is slow. Standardize features first:

X = (X - X.mean(axis = 0)) / X.std(axis = 0)

4.10 Three flavors of gradient descent

Worked Problem · Three Gradient-Descent Steps

Problem. Fit y = w·x + b to x = [1, 2, 3], y = [2, 4, 6]. Start at w = 0, b = 0 with learning rate 0.1. Do step 1 by hand, then run three steps in code.

1. Predictions with w = 0, b = 0 are all 0, so the errors (ŷ − y) are −2, −4, −6.
2. Gradient for w = (2/n) × Σ(error × x) = (2/3) × (−2×1 − 4×2 − 6×3) = (2/3) × (−28) ≈ −18.667.
3. Gradient for b = (2/n) × Σ(error) = (2/3) × (−12) = −8.
4. Update (move against the gradient): w = 0 − 0.1 × (−18.667) ≈ 1.867 and b = 0 − 0.1 × (−8) = 0.8.
5. Run the loop to see all three steps and the loss shrinking.

import numpy as np
x = np.array([1., 2., 3.]); y = np.array([2., 4., 6.])
w, b, lr, n = 0.0, 0.0, 0.1, 3
for step in range(1, 4):
    err = (w * x + b) - y
    loss = (err ** 2).mean()
    dw = (2 / n) * (err * x).sum()
    db = (2 / n) * err.sum()
    w, b = w - lr * dw, b - lr * db
    print(step, round(loss, 3), round(w, 3), round(b, 3))

1 18.667 1.867 0.8
2 0.296 1.671 0.693
3 0.073 1.701 0.686

Answer. After step 1: w ≈ 1.867, b = 0.8. The printed loss falls at every step, so the line is moving toward y = 2x.

Your Turn · A Learning Rate That Is Too Big

Problem. Minimize f(w) = w² starting at w = 3. The slope is 2w. Take three steps with learning rate 0.1 and then with 1.1. What happens?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Update rule: w_new = w − lr × 2w.
2. lr = 0.1: w = 3 − 0.1×6 = 2.4, then 1.92, then 1.536. It shrinks toward 0 (the minimum).
3. lr = 1.1: w = 3 − 1.1×6 = −3.6, then −3.6 − 1.1×(−7.2) = 4.32, then −5.184. It jumps back and forth and gets farther from 0 each time.
4. Run both.

for lr in (0.1, 1.1):
    w = 3.0
    out = []
    for _ in range(3):
        w = w - lr * 2 * w
        out.append(round(w, 3))
    print(lr, out)

0.1 [2.4, 1.92, 1.536]
1.1 [-3.6, 4.32, -5.184]

Answer. lr 0.1 gives [2.4, 1.92, 1.536] (converging); lr 1.1 gives *[-3.6, 4.32, -5.184]* (diverging). Too large a step overshoots the minimum.

5. ★ ASSIGNMENT — Linear Regression From Scratch

5.1 Plan

1. Create fake data (true line: y = 3x + 4 + noise)
2. Initialize w=0, b=0
3. Loop for many epochs: predict → loss → gradients → update
4. Record loss each epoch
5. Plot loss curve and final line
6. Repeat with scikit-learn, compare

import numpy as np
import matplotlib.pyplot as plt
# ----- 1. Make data -----
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
# 100 numbers between 0 and 2, shape (100,1)
y = 4 + 3 * X + np.random.randn(100, 1)     # true line + random noise
# ----- 2. Initialize parameters -----
w = 0.0

b = 0.0
lr = 0.1            # learning rate
epochs = 200
n = len(X)
losses = []         # to store loss history
# ----- 3. Training loop -----
for epoch in range(epochs):
    # forward: predictions
    y_pred = w * X + b                       # shape (100,1)
    # error and loss
    error = y_pred - y
    loss = np.mean(error ** 2)
    losses.append(loss)
    # gradients
    dw = (2 / n) * np.sum(error * X)
    db = (2 / n) * np.sum(error)
    # update (move opposite to gradient)
    w = w - lr * dw
    b = b - lr * db
    if epoch % 20 == 0:
        print(f"epoch { epoch :3d} loss { loss :.4f} w { w :.3f} b { b :.3f} ")
print("Learned:", w, b)     # should be near 3 and 4

5.2 Code (every line explained)

plt.plot(losses)
plt.xlabel("Epoch")
plt.ylabel("MSE loss")
plt.title("Loss curve")
plt.show()

5.3 Plot loss curve

What you should see: steep drop at the start, then flattening. If it doesn’t drop → check learning rate.

plt.scatter(X, y, alpha = 0.6, label = "data")
plt.plot(X, w * X + b, color = "red", label = "fitted line")
plt.legend() ; plt.show()

5.4 Plot the fitted line

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
model = LinearRegression()
model.fit(X, y)                       # training in ONE line
print("sklearn w:", model.coef_[0][0], "b:", model.intercept_[0])
print("sklearn MSE:", mean_squared_error(y, model.predict(X)))

5.5 Scikit-learn version

5.6 Compare

From scratch scikit-learn

w ≈ 2.77 b ≈ 4.21 Method Gradient ≈ 2.77 ≈ 4.21 Closed-form formula descent (iterative) (Normal Equation)

They match closely. sklearn gets the exact answer directly; gradient descent approaches it step by step (and works for models with no closed form, like neural networks).

Put a table like this in your README.

# X: (n, d), y: (n,1), w: (d,1)
y_pred = X @ w + b
error = y_pred - y
dw = (2 / n) * X.T @ error      # (d,n)@(n,1) -> (d,1)
db = (2 / n) * error.sum()

5.7 Extra: multiple features (vectorized)

6. Evaluating a Regression Model

from sklearn.metrics import r2_score
r2_score(y, model.predict(X))

Worked Problem · Compute R²

Problem. True values [3, 5, 7], predictions [2.5, 5.5, 6.5]. Compute R² = 1 − SSres ÷ SStot.

1. Residuals: 0.5, −0.5, 0.5. Squared: 0.25 each, so SSres = 0.75.
2. The mean of y is 5. Distances from the mean: −2, 0, 2. Squared: 4, 0, 4, so SStot = 8.
3. R² = 1 − 0.75 ÷ 8 = 1 − 0.09375 = 0.90625.
4. Confirm with scikit-learn.

from sklearn.metrics import r2_score
print(r2_score([3, 5, 7], [2.5, 5.5, 6.5]))

0.90625

Answer. R² = 0.90625: the model explains about 91% of the variation in y.

Your Turn · The Do-Nothing Baseline

Problem. A model always predicts the mean of y. For y = [2, 4, 6] it predicts [4, 4, 4]. What is its R²?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. SSres = (2−4)² + (4−4)² + (6−4)² = 4 + 0 + 4 = 8.
2. SStot is the same calculation around the mean: 8.
3. R² = 1 − 8 ÷ 8 = 0.

from sklearn.metrics import r2_score
print(r2_score([2, 4, 6], [4, 4, 4]))

0.0

Answer. 0.0. R² = 0 means no better than always guessing the mean; negative means worse.

7. Common Problems & Fixes

Problem Cause Fix

Loss = nan Learning rate too high Lower by 10× Loss flat from the start lr too tiny / bug in gradient Increase lr; check formula signs

8. Practice Questions

1. In ŷ = 2x + 1, what is ŷ when x = 4? → 9
2. Compute MSE for y=[1,2], ŷ=[2,4]. → (1+4)/2 = 2.5
3. Why do we square errors?
4. What happens if the learning rate is too large?
5. What is an epoch?
6. Explain gradient descent using the mountain analogy in 3 sentences.
7. Why is mini-batch preferred over batch GD in deep learning?

9. Quick Summary

• ML = learn rules from data
• Linear regression: ŷ = wx + b
• Loss (MSE) = average squared error
• Gradient descent: w ← w − α·gradient repeated for many epochs
• Learning rate too small → slow, too big → explodes
• sklearn gives closed-form result; scratch version should match

Next: Day 5 — Overfitting, Regularization, Cross-Validation

⚡ Quick Recap — Day 4

2.1 Supervised Learning

Learn from labeled examples (input → correct answer).

• Features (X) = inputs (house size, bedrooms)
• Target (y) = what we predict (price)
• Regression → predict a number
• Classification → predict a category

Unsupervised = no labels (clustering). Reinforcement = learn by reward.

2.2 Linear Regression

Fit a straight line through data.

ŷ = w·x + b

• w (weight) = slope: how much y changes per unit x
• b (bias) = intercept: y when x = 0

Multiple features: ŷ = w₁x₁ + w₂x₂ + ⋯ + b → in matrix form y_pred = X @ w + b.

2.3 Loss Function & MSE

Loss = a single number saying “how wrong is the model?” Lower = better.

MSE (Mean Squared Error):

MSE = (1/n) · Σ (y − ŷ)²

• Squaring: makes errors positive + punishes big errors more.
• Other: MAE (average absolute error, robust to outliers), RMSE (√MSE, same units as y).

2.4 Gradient Descent ★

Analogy: You’re blindfolded on a hill and want to reach the valley. Feel the slope under your feet, take a small step downhill, repeat.

• Gradient = slope of the loss w.r.t. each parameter.
• Move opposite to the gradient.

Update rule:

w = w − α · ∂L/∂w and b = b − α · ∂L/∂b

Gradients for MSE:

∂L/∂w = −(2/n) · Σ x(y − ŷ) and ∂L/∂b = −(2/n) · Σ (y − ŷ)

Key terms

Learning-rate guide

• Loss goes up / NaN → α too high
• Loss decreases very slowly → α too low
• Smooth downward curve →

2.5 ★ Linear Regression From Scratch (NumPy)

import numpy as np
import matplotlib.pyplot as plt
# data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)   # true w=3, b=4
w, b = 0.0, 0.0
lr, epochs, n = 0.1, 1000, len(X)
losses = []
for _ in range(epochs):
    y_pred = w * X + b
    error = y_pred - y
    loss = (error ** 2).mean()
    losses.append(loss)
    dw = (2 / n) * (X * error).sum()
    db = (2 / n) * error.sum()
    w -= lr * dw
    b -= lr * db
print(w, b)                # ~3, ~4
plt.plot(losses) ; plt.xlabel("Epoch") ; plt.ylabel("MSE") ; plt.show()

from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X, y)
print(model.coef_, model.intercept_)

Scikit-learn version Compare: both should give nearly the same w and b. sklearn uses a closedform solution (Normal Equation), not gradient descent — that’s why it’s exact.

Tips

• Scale features (StandardScaler) before gradient descent → converges faster.
• Vectorize: X.T @ error instead of loops.

Variants of Gradient Descent

Type Data per step Notes

Batch All Stable, slow on big data

Beginner checkpoint — Day 4 ■ Explain the day’s main idea without using the textbook wording. ■ Write one tiny example from memory. ■ Identify one common failure mode. ■ Explain how today’s topic connects to the next day. ■ Write one question you still cannot answer.

Move on when you can… ☐ compute MSE for three points by hand ☐ do one gradient-descent update by hand ☐ explain what a learning rate that is too big or too small does ☐ fit LinearRegression and read its coefficients ☐ explain what R² = 0 means If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.