Day 6 · Metrics, Decision Trees, Random Forest, Boosting

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ Day 5: train / validation / test split and cross-validation ☐ Day 1: probability as a number from 0 to 1 ☐ Day 3: Pandas basics for the fraud example If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Learn how to measure classification models correctly, and learn the most powerful models for table-style (tabular) data.

Big picture — read this first Accuracy answers only one question: how many predictions were correct overall? It can be misleading when classes are imbalanced. Precision asks how reliable positive predictions are. Recall asks how many actual positives were found. F1 balances the two. Use a baseline before complicated models. A complicated model should earn its complexity through better validation performance or a useful operational tradeoff. Error analysis should ask which examples the model gets wrong, and why.

PART A — CLASSIFICATION METRICS

1. Classification recap

Predict a category. Simplest case = binary: two classes, usually called Positive (1) and Negative (0).

• Fraud (1) vs Not fraud (0)
• Spam (1) vs Not spam (0)
• Disease (1) vs Healthy (0)

“Positive” means the thing you are trying to find, not “good”.

Worked Problem · From Probability To Class

Problem. A model says an email is spam with probability 0.7. What class does it predict at thresholds 0.5 and 0.8?

1. A classifier first gives a probability, then compares it with a threshold.
2. At 0.5: 0.7 ≥ 0.5 → class 1 (spam).
3. At 0.8: 0.7 < 0.8 → class 0 (not spam).
4. Moving the threshold trades precision against recall.

p = 0.7
print(int(p >= 0.5), int(p >= 0.8))

1 0

Answer. 1 at threshold 0.5 and 0 at 0.8.

2. Logistic Regression (quick intro — used in today’s

assignment)

Despite the name, it’s a classification model. It computes a score like linear regression, then squashes it to a probability between 0 and 1 using the sigmoid function:

probability = 1 / (1 + e^(−(w·x + b)))

Then: probability ≥ 0.5 → predict 1, else 0 (threshold = 0.5, adjustable).

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(max_iter = 1000).fit(X_train, y_train)
clf.predict(X_test)          # 0/1 labels
clf.predict_proba(X_test)    # probabilities: [[P(0), P(1)], ...]

Worked Problem · Logistic Regression By Hand

Problem. A logistic model computes the score z = 1.5. Turn it into a probability with the sigmoid and predict the class.

1. Logistic regression is a linear model followed by a sigmoid: p = 1 ÷ (1 + e^−z).
2. e^−1.5 ≈ 0.223, so p = 1 ÷ 1.223 ≈ 0.818.
3. Since 0.818 ≥ 0.5, the predicted class is 1.

import numpy as np
z = 1.5; p = 1 / (1 + np.exp(-z))
print(round(p, 3), int(p >= 0.5))

0.818 1

Answer. Probability 0.818, class 1.

3. The Confusion Matrix — foundation of everything

A 2×2 table comparing predictions with reality.

Predicted: Positive Predicted: Negative

Actually Positive TP True Positive ✓ FN False Negative ✗ (missed it!) TN (caught it) Actually Negative FP False True Negative ✓ Positive ✗ (false alarm)

Memory trick: the second word is what the model predicted; the first word says if it was right.

• True Positive = predicted positive, correct
• False Positive = predicted positive, wrong

Example: fraud detection, 1000 transactions

• 10 are actually fraud, 990 are legit.
• Model flags 12 as fraud: 8 are truly fraud, 4 are legit.

Pred Fraud Pred Legit

Actual Fraud (10) TP = 8 Actual FN = 2 TN Legit (990) FP = 4 = 986

from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, y_pred)     # [[TN, FP],[FN, TP]]  ← sklearn layout!

sklearn’s layout has negatives first: [[TN, FP],[FN, TP]].

Worked Problem · Build A Confusion Matrix

Problem. True labels [1,0,1,1,0,0,1,0], predictions [1,0,0,1,0,1,1,0] (1 = fraud). Count TP, FP, FN, TN.

1. Compare position by position (true, predicted): (1,1) (0,0) (1,0) (1,1) (0,0) (0,1) (1,1) (0,0).
2. TP (true 1, predicted 1): positions 0, 3, 6 → 3.
3. FN (true 1, predicted 0): position 2 → 1.
4. FP (true 0, predicted 1): position 5 → 1.
5. TN (true 0, predicted 0): positions 1, 4, 7 → 3.
6. scikit-learn lays the matrix out as [[TN, FP], [FN, TP]].

from sklearn.metrics import confusion_matrix
yt = [1, 0, 1, 1, 0, 0, 1, 0]; yp = [1, 0, 0, 1, 0, 1, 1, 0]
print(confusion_matrix(yt, yp))

[[3 1]
 [1 3]]

Answer. TP = 3, FP = 1, FN = 1, TN = 3, so the matrix is [[3, 1], [1, 3]].

Your Turn · Metrics From Counts

Problem. A fraud model gives TP = 40, FP = 10, FN = 20, TN = 930. Find accuracy, precision, recall and F1.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Accuracy = (TP + TN) ÷ all = (40 + 930) ÷ 1000 = 0.97.
2. Precision = TP ÷ (TP + FP) = 40 ÷ 50 = 0.80.
3. Recall = TP ÷ (TP + FN) = 40 ÷ 60 ≈ 0.667.
4. F1 = 2PR ÷ (P + R) = 2 × 0.8 × 0.667 ÷ 1.467 ≈ 0.727.
5. Notice accuracy looks great while recall shows 1 in 3 frauds is missed.

TP, FP, FN, TN = 40, 10, 20, 930
acc = (TP + TN) / (TP + FP + FN + TN)
p = TP / (TP + FP); r = TP / (TP + FN)
print(round(acc, 3), round(p, 3), round(r, 3), round(2 * p * r / (p + r), 3))

0.97 0.8 0.667 0.727

Answer. Accuracy 0.97, precision 0.8, recall 0.667, F1 0.727.

4. Metrics (using the fraud example)

4.1 Accuracy

Accuracy = (TP + TN) / total = (8 + 986) / 1000 = 99.4%

“What % of all predictions were right?”

The accuracy trap: A useless model that ALWAYS says “legit” gets 990/1000 = 99% accuracy — yet catches zero fraud. On imbalanced data accuracy is misleading!

4.2 Precision

Precision = TP / (TP + FP) = 8 / (8 + 4) = 0.67

“Of everything I flagged as positive, how many were actually positive?” = How trustworthy are my alarms? High precision = few false alarms.

4.3 Recall (Sensitivity, True Positive Rate)

Recall = TP / (TP + FN) = 8 / (8 + 2) = 0.80

“Of all the actual positives, how many did I find?” High recall = few misses.

4.4 F1-score

F1 = 2 × (Precision × Recall) / (Precision + Recall) = 2×(0.67×0.8)/(0.67+0.8) = 0.73

The harmonic mean — it’s high only if BOTH precision and recall are high. Good single number for imbalanced data.

4.5 Which metric matters? (think about the COST of errors)

Precision-recall tradeoff: Lower the threshold (flag more things) → recall ↑, precision ↓. Raise threshold → precision ↑, recall ↓.

4.6 ROC Curve and AUC

• Model outputs a probability. Different thresholds give different (TPR, FPR) pairs.
• TPR = Recall = TP/(TP+FN). FPR = FP/(FP+TN) (fraction of negatives wrongly flagged).
• ROC curve plots TPR (y) vs FPR (x) for all thresholds.
• AUC = area under it: – 1.0 = perfect ranking – 0.5 = random guessing (diagonal line) – Interpretation: probability that the model ranks a random positive above a random negative.

TPR
1 |      ______ ← good model (curve hugs top-left)
  |    /
  |  /    ← diagonal = random (AUC 0.5)
  |/______ FPR 1

For very imbalanced data, also consider PR-AUC (average_precision_score) — it’s more honest than ROC-AUC there.

from sklearn.metrics import (classification_report, confusion_matrix,
                             roc_auc_score, precision_score, recall_score, f1_score)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]     # probability of class 1
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits = 3))
print("ROC-AUC:", roc_auc_score(y_test, y_prob))     # use probabilities, NOT labels

4.7 All in code

PART B — DECISION TREES

5. Decision Tree — 20 Questions game

A tree of yes/no questions that ends in a prediction.

           Is income > 50k?
            /            \
         Yes              No
  Age > 30?           Has debt?
   /     \             /     \
Approve  Reject     Reject  Approve

• Root = first question; Node = a question; Leaf = final answer; Depth = number of levels.

How does it choose questions? It tries every feature and threshold and picks the split that makes the resulting groups most pure (each group mostly one class).

Gini impurity measures mixing: Gini = 1 − Σ (pᵢ)²

• Group all one class → Gini = 0 (pure)
• 50/50 mix → Gini = 0.5 (most mixed) Another measure: entropy. The tree greedily picks the best split at each step.

Pros / Cons

from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth = 5, min_samples_leaf = 20, random_state =
    42)

Controlling overfitting

• max_depth — limit levels
• min_samples_leaf — each leaf needs at least N samples
• Full depth tree → train accuracy 100%, test poor.

Worked Problem · Pick A Split With Gini

Problem. A node has 6 positives and 2 negatives. A split sends [4 pos, 0 neg] left and [2 pos, 2 neg] right. Compute the Gini impurity before and after.

1. Gini = 1 − (share of class 1)² − (share of class 2)². Lower means purer.
2. Before: shares 0.75 and 0.25, so Gini = 1 − 0.5625 − 0.0625 = 0.375.
3. Left node is pure: Gini = 0. Right node is 50/50: Gini = 1 − 0.25 − 0.25 = 0.5.
4. Weighted average after the split: (4/8) × 0 + (4/8) × 0.5 = 0.25.
5. Impurity fell from 0.375 to 0.25, so this is a good split.

def gini(pos, neg):
    n = pos + neg
    return 1 - (pos / n) ** 2 - (neg / n) ** 2
before = gini(6, 2)
after = 4 / 8 * gini(4, 0) + 4 / 8 * gini(2, 2)
print(round(before, 3), round(after, 3))

0.375 0.25

Answer. Before 0.375, after 0.25, a drop of 0.125. The tree chooses the split with the biggest drop.

Your Turn · Pure And Mixed Nodes

Problem. What is the Gini impurity of a node with 5 positives and 0 negatives? And with 5 positives and 5 negatives?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. All one class: 1 − 1² − 0² = 0.
2. Half and half: 1 − 0.5² − 0.5² = 0.5, the worst possible for two classes.

g = lambda p, n: 1 - (p / (p + n)) ** 2 - (n / (p + n)) ** 2
print(g(5, 0), g(5, 5))

0.0 0.5

Answer. 0.0 (pure) and 0.5 (maximally mixed).

6. Random Forest — “Wisdom of the crowd”

The idea One tree is unreliable. Build many different trees and let them vote. Errors of individual trees cancel out.

How each tree is made different

1. Bootstrap sampling: each tree trains on a random sample of rows (with replacement)
2. Random features: at each split, only a random subset of features is considered

Prediction Classification: majority vote. Regression: average.

This technique = Bagging (Bootstrap Aggregating). It reduces variance (overfitting).

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators = 200, max_depth = None,
                            class_weight = "balanced", n_jobs =- 1, random_state =
                                42)
rf.fit(X_train, y_train)
rf.feature_importances_     # which features matter most

3 Strong default model, little tuning, feature importance ✗ Slower, less interpretable than one tree, large in memory

Worked Problem · Majority Vote

Problem. Five trees predict [1, 0, 1, 1, 0] for one customer. What does the forest predict, and with what probability?

1. Count the votes for class 1: three trees say 1.
2. Count the votes for class 0: two trees say 0.
3. The majority wins: class 1.
4. Probability of class 1 = 3 ÷ 5 = 0.6.

votes = [1, 0, 1, 1, 0]
print(1 if sum(votes) > len(votes) / 2 else 0, sum(votes) / len(votes))

1 0.6

Answer. The forest predicts class 1 with probability 0.6.

Your Turn · Why Many Weak Trees Help

Problem. Three independent trees are each right 70% of the time. How often is the majority vote right?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. The vote is right if at least 2 of the 3 trees are right.
2. Exactly 2 right: 3 ways × 0.7 × 0.7 × 0.3 = 3 × 0.147 = 0.441.
3. All 3 right: 0.7³ = 0.343.
4. Total = 0.441 + 0.343 = 0.784.
5. This assumes the trees make independent mistakes, which is why a forest gives each tree different random data and features.

p = 0.7
print(round(3 * p**2 * (1 - p) + p**3, 3))

0.784

Answer. 0.784, better than any single tree's 0.7.

7. Gradient Boosting — “Learn from mistakes”

The idea Instead of trees voting independently, build trees one after another, each one focusing on fixing the errors of the previous ones.

Analogy: A team where person 1 makes a rough guess, person 2 corrects person 1’s mistakes, person 3 corrects what remains, and so on. Final answer = sum of all corrections.

How it works (simple version)

1. Start with a simple prediction (e.g. the average)
2. Compute errors (residuals)
3. Train a small tree to predict those errors
4. Add it (scaled by learning rate) to the prediction 5. Repeat many times

This is gradient descent, but in “function space” — each tree is a step downhill.

This technique = Boosting. It reduces bias.

from sklearn.ensemble import GradientBoostingClassifier
gb = GradientBoostingClassifier(n_estimators = 200, learning_rate = 0.1, max_depth
    = 3, random_state = 42)
gb.fit(X_train, y_train)

Popular fast libraries: for tabular data.

Random Forest vs Gradient Boosting

Random Forest Gradient Boosting

Tree building Parallel, independent Trees Sequential, each fixes previous Shallow Bias Deep Reduces Variance Overfitting risk Higher — tune carefully More Slower Lower Tuning needed Little Speed Faster

Worked Problem · Learn From Mistakes

Problem. True values [10, 20, 30]. Start with the average as the first prediction. A second small model learns the leftover errors and is added with learning rate 0.5. Find the new predictions.

1. First prediction for everyone: the mean, 20.
2. Residuals (truth − prediction): [−10, 0, 10].
3. Suppose the second model predicts these residuals perfectly. Add half of them: 20 + 0.5×(−10) = 15, 20 + 0 = 20, 20 + 0.5×10 = 25.
4. New predictions [15, 20, 25] are closer to the truth. Repeat to keep shrinking the errors.

import numpy as np
y = np.array([10., 20., 30.]); p = np.full(3, y.mean())
res = y - p
p = p + 0.5 * res
print(res.tolist(), p.tolist(), np.abs(y - p).tolist())

[-10.0, 0.0, 10.0] [15.0, 20.0, 25.0] [5.0, 0.0, 5.0]

Answer. Residuals [-10, 0, 10], new predictions [15, 20, 25], and the errors are halved.

PART C — CLASS IMBALANCE

8. The problem

Fraud = 0.17% of data. A model can ignore fraud and still get 99.8% accuracy.

Worked Problem · Why Imbalance Fools Accuracy

Problem. In 1,000 transactions only 10 are fraud. What accuracy does “always predict not fraud” get?

1. The model is right on all 990 normal transactions.
2. Accuracy = 990 ÷ 1000 = 0.99.
3. Yet it catches 0 of the 10 frauds, so it is useless.

print(990 / 1000)

0.99

Answer. 0.99 accuracy with 0 frauds caught.

9. Solutions

# SMOTE (pip install imbalanced-learn)
from imblearn.over_sampling import SMOTE
X_res, y_res = SMOTE(random_state = 42).fit_resample(X_train, y_train)

*Watch out Apply resampling ONLY to training data.* Never to validation/test — they must reflect real-world imbalance.

Tune threshold:

import numpy as np
from sklearn.metrics import precision_recall_curve
p, r, t = precision_recall_curve(y_test, y_prob)
f1 = 2 * p * r / (p + r + 1e-9)
best_t = t[np.argmax(f1[: - 1])]
y_pred_custom = (y_prob >= best_t).astype(int)

Worked Problem · Class Weights For Imbalance

Problem. A dataset has 95 normal and 5 fraud rows. Compute balanced class weights = n ÷ (2 × count).

1. Total rows n = 100 and there are 2 classes.
2. Normal weight = 100 ÷ (2 × 95) ≈ 0.526.
3. Fraud weight = 100 ÷ (2 × 5) = 10.
4. Each fraud row now counts about 19 times as much as a normal row, so the model cannot ignore it.

n = 100
for name, c in (("normal", 95), ("fraud", 5)):
    print(name, round(n / (2 * c), 3))

normal 0.526
fraud 10.0

Answer. Normal 0.526, fraud 10. In scikit-learn, class_weight="balanced" does this for you.

Your Turn · The Accuracy Trap

Problem. A dataset has 990 normal and 10 fraud rows. A model always predicts “normal”. What are its accuracy and recall?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. It is right on all 990 normal rows: accuracy = 990 ÷ 1000 = 0.99.
2. It finds none of the 10 frauds: recall = 0 ÷ 10 = 0.
3. High accuracy hides a useless model.

acc = 990 / 1000
rec = 0 / 10
print(acc, rec)

0.99 0.0

Answer. Accuracy 0.99 but recall 0.0. On imbalanced data, always check recall and precision, not accuracy alone.

10. ★ ASSIGNMENT — Credit Card Fraud Comparison

Dataset: Kaggle “Credit Card Fraud Detection” (creditcard.csv, 284,807 rows, only 492 fraud). Columns V1–V28 (already anonymized), Amount, Time, Class (target).

import pandas as pd, numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, \
    precision_score, \
   recall_score, f1_score
df = pd.read_csv("creditcard.csv")
print(df["Class"].value_counts(normalize = True))   # ~0.17% fraud
X = df.drop(columns = ["Class"]) ; y = df["Class"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, stratify
    = y, random_state = 42)
scaler = StandardScaler().fit(X_train[["Amount","Time"]])
for d in (X_train, X_test):
    d[["Amount","Time"]] = scaler.transform(d[["Amount","Time"]])
models = {
  "Logistic Regression": LogisticRegression(max_iter = 1000, class_weight =
      "balanced"),
  "Random Forest": RandomForestClassifier(n_estimators = 100, class_weight =
      "balanced", n_jobs =- 1,
     random_state = 42),
  "Gradient Boosting": GradientBoostingClassifier(random_state = 42),
}
rows = []
for name, m in models.items():
    m.fit(X_train, y_train)
    pred = m.predict(X_test) ; prob = m.predict_proba(X_test)[:,1]
    rows.append({"Model": name,
                 "Precision": precision_score(y_test, pred),
                 "Recall": recall_score(y_test, pred),
                 "F1": f1_score(y_test, pred),
                 "ROC-AUC": roc_auc_score(y_test, prob)})
    print(name) ; print(confusion_matrix(y_test, pred))
print(pd.DataFrame(rows).round(3))

(Gradient Boosting on 230k rows can take a few minutes — use a sample or HistGradientBoostingClassifier to speed up.)

Typical findings:

• Logistic Regression with balanced weights: high recall, low precision (many false alarms)
• Random Forest: high precision, good recall
• Gradient Boosting: strong overall

Write your conclusion by answering: Which model has best recall? Best precision? Which would you choose for a bank and why (missed fraud vs annoyed customers)?

11. Common Mistakes

12. Practice Questions

1. TP=40, FP=10, FN=20, TN=930. Compute precision, recall, F1. → P=0.8, R=0.667, F1=0.727
2. Why can 99% accuracy be a bad model?
3. Cancer screening: precision or recall first? Why?
4. What is bagging vs boosting?
5. Why does a Random Forest overfit less than a single tree?
6. What does AUC = 0.5 mean?
7. Name three ways to handle class imbalance.

13. Quick Summary

• Confusion matrix → TP/FP/FN/TN → all metrics
• Precision: “when I say yes, am I right?” Recall: “did I find all the yeses?”
• Imbalanced data → never trust accuracy
• Decision tree = flowchart; overfits alone
• Random Forest = many parallel trees voting (↓ variance)
• Gradient Boosting = sequential error-fixing trees (↓ bias)

Next: Days 7–8 — k-NN, k-Means, PCA, Embeddings + Telco Churn Project

⚡ Quick Recap — Day 6

4.1 Confusion Matrix

For binary classification (Positive = the thing you’re hunting, e.g. fraud):

4.2 Metrics

Remember:

• Precision = “When I say yes, am I right?”
• Recall = “Did I find all the yeses?”

Accuracy trap: 99% of transactions are legit → model always saying “not fraud” gets 99% accuracy but is useless.

ROC curve: plots TPR (recall) vs FPR at every threshold. Threshold (default 0.5) can be moved to trade precision ↔ recall.

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
print(confusion_matrix(y_te, pred))
print(classification_report(y_te, pred))
print(roc_auc_score(y_te, model.predict_proba(X_te)[:, 1]))

4.3 Decision Trees

A flowchart of yes/no questions.

Age > 30?
 ├─ Yes → Income > 50k? → ...
 └─ No  → ...

• Picks the split that makes groups purest.
• Gini impurity: 1 − Σ p² (0 = pure). Entropy is an alternative.
• ✓ Easy to interpret, no scaling needed, handles non-linear data

⊠ Overfits easily → control with max_depth, min_samples_leaf

4.4 Random Forest

Many trees, each trained on a *random sample of rows + random subset of features; predictions are voted/averaged*.

• Technique = Bagging (Bootstrap Aggregating) → reduces variance.
• ✓ Robust, less overfitting, gives feature importance
• ✗ Slower, less interpretable
• Key params: n_estimators, max_depth, class_weight="balanced"

4.5 Gradient Boosting

Trees built one after another; each new tree fixes the errors of the previous ones. .

• Technique = Boosting → reduces bias.
• Tools: GradientBoostingClassifier, XGBoost, LightGBM, CatBoost (best for tabular data).
• Key params: n_estimators, learning_rate (small = better but needs more trees), max_depth (keep shallow, 3–6).

Random Forest Gradient Boosting

Trees built Parallel (independent) Sequential (each fixes the last) Reduces Variance Bias Overfit risk Lower Higher (tune carefully) Speed Faster to train Slower

4.6 Class Imbalance

Fraud is ~0.2% of data. Fixes: 1. Right metrics (Recall, F1, PR-AUC) 2. Class weights: class_weight="balanced" 3. Resampling: oversample minority (SMOTE) or undersample majority 4. Threshold tuning 5. Stratified splits to keep ratio Apply SMOTE only on training data, never on test/validation.

Day 6 Assignment Recipe

1. Load credit-card fraud data → train_test_split(..., stratify=y)
2. Train Logistic Regression, Random Forest, Gradient Boosting
3. Compare Precision, Recall, F1, ROC-AUC, confusion matrix in a table
4. Conclude which model is best and why (e.g. recall vs false alarms)

Beginner checkpoint — Day 6 ■ Explain the day’s main idea without using the textbook wording. ■ Write one tiny example from memory. ■ Identify one common failure mode. ■ Explain how today’s topic connects to the next day. ■ Write one question you still cannot answer.

Move on when you can… ☐ fill a confusion matrix and compute accuracy, precision, recall and F1 ☐ explain the accuracy trap on imbalanced data ☐ compute Gini impurity for a node ☐ explain how a random forest votes ☐ name two ways to handle class imbalance If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.