Day 3 · Pandas: Load, Clean & Prepare Data

Suggested pace: 1–1.5 hrs learn · 2–3 hrs code · rest to debug and document

What you should know first ☐ Day 2: arrays and shapes ☐ Day 1: dictionaries and lists (a DataFrame is built from them) ☐ Averages (mean, median) If one of these feels shaky, spend 15 minutes revisiting it before starting. That is normal and saves time later.

Goal of today Learn Pandas, the tool for tables of data. You will load a real dataset (Titanic), inspect it, select and filter rows, handle missing values, group, merge and encode categories.

You will be able to:

• Load and inspect a CSV file
• Select, filter and group data
• Find and fix missing values
• Merge tables and encode categorical columns
• Clean a real dataset end to end

PART B — PANDAS

10. What is Pandas?

Pandas = Excel inside Python. Two objects:

• Series = one column
• DataFrame = a whole table (many columns)

df = pd.DataFrame({
    "name": ["Ravi", "Sita", "Kiran"],
    "age":  [22, 25, 23],
    "city": ["Hyd", "Chennai", "Hyd"]
})

    name  age     city
0   Ravi   22      Hyd
1   Sita   25  Chennai
2  Kiran   23      Hyd

The left numbers (0,1,2) are the index (row labels).

Worked Problem · Series And Dataframe

Problem. Make a Series of ages and a DataFrame with two columns. What type is each, and how do you get one column out of the table?

1. A Series is one labeled column of values. A DataFrame is a table of Series that share row labels.
2. Build a Series from a list, and a DataFrame from a dictionary of columns.
3. Selecting one column with df["Age"] returns a Series.

import pandas as pd
s = pd.Series([21, 23, 22])
df = pd.DataFrame({"Name": ["A", "B", "C"], "Age": [21, 23, 22]})
print(type(s).__name__, type(df).__name__, type(df["Age"]).__name__)

Series DataFrame Series

Answer. Series DataFrame Series.

11. Loading and Inspecting Data (CSV)

CSV = plain text file, values separated by commas. The most common data format.

df = pd.read_csv("titanic.csv")

First things to ALWAYS do with new data:

df.head()          # first 5 rows
df.tail()          # last 5 rows
df.shape           # (rows, columns)
df.info()          # column names, types, non-null counts   ← shows missing data!
df.describe()      # count, mean, std, min, max for numeric columns
df.columns         # list of column names
df.dtypes          # type of each column
df["Sex"].value_counts()   # how many of each category
df.isna().sum()    # missing values per column

Data types: int64 (whole numbers), float64 (decimals), object (text), bool, datetime64.

Worked Problem · Inspect A Small Table

Problem. A table of students has columns Name, Age, Passed. Show its shape, the mean age and how many students passed.

1. We build the table from a dictionary of columns. In real projects you would use pd.read_csv("file.csv").
2. df.shape gives (rows, columns).
3. df["Age"].mean() averages one column.
4. df["Passed"].value_counts() counts each distinct value. Use .head() to peek at the first rows.

import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ben", "Cara", "Dev"],
                   "Age": [21, 23, 22, 24],
                   "Passed": ["yes", "no", "yes", "yes"]})
print(df.shape)
print(df["Age"].mean())
print(df["Passed"].value_counts())

(4, 3)
22.5
Passed
yes    3
no     1
Name: count, dtype: int64

Answer. Shape (4, 3), mean age 22.5, and 3 students passed versus 1 who did not.

Your Turn · Summarize Numbers

Problem. Using the same table idea with Age = [21, 23, 22, 24], find the youngest and oldest ages in one call.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. describe() gives count, mean, std, min, quartiles and max for numeric columns.
2. We only need min and max, so pick them from the result.

import pandas as pd
ages = pd.Series([21, 23, 22, 24])
d = ages.describe()
print(d["min"], d["max"])

21.0 24.0

Answer. 21.0 and 24.0. describe() is the fastest first look at any numeric column.

12. Selecting Data

df["age"]                 # one column → Series
df[["name", "age"]]       # multiple columns → DataFrame (double brackets!)
df.loc[0]                 # row with label 0
df.loc[0, "age"]          # row label 0, column "age"
df.iloc[0]                # row at POSITION 0
df.iloc[0, 1]             # position row 0, position column 1
df.iloc[0:2, 0:2]         # slice by position

• loc = by l abel (names)
• iloc = by i nteger position

Worked Problem · Loc Versus Iloc

Problem. From the table below, get (a) the Age column, (b) the Name in row 2 using labels, (c) rows 1 and 2 by position.

1. A column is selected by name: df["Age"].
2. loc uses labels. The row labels here are 0, 1, 2, 3, so df.loc[2, "Name"] is the row labeled 2, column Name.
3. iloc uses positions and the stop is excluded, so df.iloc[1:3] gives positions 1 and 2.

import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ben", "Cara", "Dev"], "Age": [21, 23, 22, 24]})
print(df["Age"].tolist())
print(df.loc[2, "Name"])
print(df.iloc[1:3])

[21, 23, 22, 24]
Cara
   Name  Age
1   Ben   23
2  Cara   22

Answer. Ages [21, 23, 22, 24], name Cara, and the rows for Ben (23) and Cara (22).

Your Turn · Two Columns, One Row

Problem. From the same table, get the Name and Age of the first row only.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Use a list of column names for several columns: ["Name", "Age"].
2. df.loc[0, ["Name", "Age"]] takes row label 0 and those two columns.

import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ben", "Cara", "Dev"], "Age": [21, 23, 22, 24]})
print(df.loc[0, "Name"], df.loc[0, "Age"])

Asha 21

Answer. Asha 21.

13. Filtering Rows

df[df["age"] > 22]
df[df["city"] == "Hyd"]
df[(df["age"] > 22) & (df["city"] == "Hyd")]   # AND → &
df[(df["age"] > 24) | (df["city"] == "Hyd")]   # OR  → |
df[ ~ (df["city"] == "Hyd")]                   # NOT → ~
df[df["city"].isin(["Hyd", "Chennai"])]
df[df["name"].str.contains("Ra")]

Use &, | — NOT and, or. And wrap each condition in ().

Creating and editing columns:

df["age_next_year"] = df["age"] + 1
df["is_adult"] = df["age"] >= 18
df["name"] = df["name"].str.upper()
df = df.rename(columns = {"name": "Name"})
df = df.drop(columns = ["age_next_year"])
df = df.sort_values("age", ascending = False)

Worked Problem · Filter With Two Conditions

Problem. From a table of passengers, keep the rows where Age is over 30 and Sex is male.

1. Make one True/False condition per rule: df["Age"] > 30 and df["Sex"] == "male".
2. Combine them with &, and put each condition in parentheses. Python's and does not work on columns.
3. Put the combined mask inside df[...] to keep only the True rows.

import pandas as pd
df = pd.DataFrame({"Name": ["A", "B", "C", "D"],
                   "Age": [25, 40, 35, 50],
                   "Sex": ["male", "male", "female", "male"]})
out = df[(df["Age"] > 30) & (df["Sex"] == "male")]
print(out["Name"].tolist())

['B', 'D']

Answer. Passengers B and D. A is too young, and C is not male.

Your Turn · Either Condition

Problem. From the same table, keep rows where Age is under 30 or Sex is female.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Use | for “or”, again with parentheses around each condition.
2. A (25) is under 30, and C is female. B (40, male) and D (50, male) match neither.

import pandas as pd
df = pd.DataFrame({"Name": ["A", "B", "C", "D"],
                   "Age": [25, 40, 35, 50],
                   "Sex": ["male", "male", "female", "male"]})
print(df[(df["Age"] < 30) | (df["Sex"] == "female")]["Name"].tolist())

['A', 'C']

Answer. ['A', 'C'].

14. Missing Values (NaN)

Real data always has gaps. Pandas marks them as NaN (“Not a Number”).

df.isna().sum()                   # count per column
df.isna().mean() * 100            # percent missing per column

Step 1 — Find them

Step 2 — Decide what to do

Why median over mean? Ages: [20, 22, 25, 90]. Mean = 39 (pulled up by 90). Median = 23.5 (typical). Median resists extreme values.

fillna returns a new object; assign it back: df["Age"] = df["Age"].fillna(...).

Worked Problem · Find And Fill Missing Ages

Problem. Ages are [22, NaN, 30, NaN, 26]. Count the missing values, then fill them with the median age.

1. isna() marks missing values as True. .sum() counts them: 2.
2. The median is computed from the values that exist: 22, 26, 30, so the median is 26.
3. fillna(26) replaces every NaN with 26. Assign the result back, because fillna returns a new object.
4. Median is safer than mean when there are outliers.

import pandas as pd, numpy as np
age = pd.Series([22, np.nan, 30, np.nan, 26])
print(age.isna().sum())
print(age.median())
print(age.fillna(age.median()).tolist())

2
26.0
[22.0, 26.0, 30.0, 26.0, 26.0]

Answer. 2 missing, median 26.0, filled list [22.0, 26.0, 30.0, 26.0, 26.0].

Your Turn · Drop Instead Of Fill

Problem. A table has a Cabin column that is mostly empty and an Embarked column with one missing value out of 100 rows. Which fix suits each column? Show the code for one tiny example.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. A column that is mostly empty carries almost no information: drop the column.
2. One missing value in 100 rows is tiny: drop that row (or fill with the most frequent value).
3. Code: drop(columns=[...]) removes a column, and dropna(subset=[...]) removes rows missing a value in that column.

import pandas as pd, numpy as np
df = pd.DataFrame({"Cabin": [np.nan, np.nan, "C85", np.nan],
                   "Embarked": ["S", np.nan, "C", "S"]})
df = df.drop(columns=["Cabin"])
df = df.dropna(subset=["Embarked"])
print(df.shape)

(3, 1)

Answer. Drop the mostly-empty column and the one bad row. The result has shape (3, 1).

15. GroupBy — “Split, Apply, Combine”

Answers questions like “average age per city?” or “survival rate per class?”

df.groupby("city")["age"].mean()

What happens: 1. Split rows into groups by city 2. Apply mean to each group 3. Combine into a result

df.groupby("Pclass")["Survived"].mean()      # survival rate for each class
df.groupby("Sex")["Fare"].agg(["mean", "max", "count"])
df.groupby(["Sex", "Pclass"])["Survived"].mean()   # two levels
df.groupby("city").agg(avg_age = ("age", "mean"), people = ("name", "count"))

Tip: for a 0/1 column, the mean = percentage of 1s.

Worked Problem · Average Per Group

Problem. Rows hold a class and a fare. Find the average fare per class, and the number of rows per class.

1. Split: group the rows by Pclass.
2. Apply: take the mean of the Fare column in each group.
3. Combine: pandas returns one row per group.
4. By hand: class 1 fares 100 and 80 → mean 90. Class 3 fares 10, 20, 30 → mean 20.

import pandas as pd
df = pd.DataFrame({"Pclass": [1, 3, 1, 3, 3], "Fare": [100, 10, 80, 20, 30]})
print(df.groupby("Pclass")["Fare"].mean())
print(df.groupby("Pclass")["Fare"].count())

Pclass
1    90.0
3    20.0
Name: Fare, dtype: float64
Pclass
1    2
3    3
Name: Fare, dtype: int64

Answer. Mean fare 90.0 for class 1 and 20.0 for class 3, with 2 and 3 rows.

Your Turn · Survival Rate Per Group

Problem. Passengers have Sex and Survived (1 = survived). Find the survival rate for each sex.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. For a 0/1 column the mean equals the share of 1s.
2. Group by Sex, then take the mean of Survived.
3. By hand: female [1, 1, 0] → 2/3 ≈ 0.667. Male [0, 0, 1, 0] → 1/4 = 0.25.

import pandas as pd
df = pd.DataFrame({"Sex": ["f", "m", "f", "m", "f", "m", "m"],
                   "Survived": [1, 0, 1, 0, 0, 1, 0]})
print(df.groupby("Sex")["Survived"].mean().round(3))

Sex
f    0.667
m    0.250
Name: Survived, dtype: float64

Answer. Female 0.667, male 0.25.

16. Merge (Join) — combining two tables

Like SQL JOIN or Excel VLOOKUP.

students = pd.DataFrame({"id":[1,2,3], "name":["A","B","C"]})
marks    = pd.DataFrame({"id":[1,2,4], "score":[80,90,70]})
pd.merge(students, marks, on = "id", how = "inner")

pd.concat([df1, df2]) stacks tables on top of each other (same columns).

Worked Problem · Join Two Tables

Problem. Table students has ids 1, 2, 3 and names. Table marks has ids 1, 2, 4 and scores. Join them on id keeping only ids in both tables.

1. Both tables share the key column id.
2. how="inner" keeps only ids present in both: 1 and 2.
3. Id 3 has no score, and id 4 has no name, so both are dropped.
4. Switch to how="left" to keep all students and put NaN where the score is missing.

import pandas as pd
students = pd.DataFrame({"id": [1, 2, 3], "name": ["A", "B", "C"]})
marks = pd.DataFrame({"id": [1, 2, 4], "score": [80, 90, 70]})
print(pd.merge(students, marks, on="id", how="inner")["id"].tolist())
print(pd.merge(students, marks, on="id", how="left")["score"].tolist())

[1, 2]
[80.0, 90.0, nan]

Answer. Inner join keeps ids [1, 2]. The left join gives scores [80.0, 90.0, nan]: student 3 stays, with a missing score.

Your Turn · Pick The Join Type

Problem. You have all customers and a table of their payments. You want every customer, even those who never paid. Which join do you use, and which rows will have NaN?

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. You want every customer, so customers is the “left” table.
2. A left join keeps all left rows and fills NaN on the right when there is no match.
3. Customers with no payment get NaN in the payment columns.

import pandas as pd
cust = pd.DataFrame({"cid": [1, 2, 3], "name": ["A", "B", "C"]})
pay = pd.DataFrame({"cid": [1, 3], "paid": [50, 20]})
out = pd.merge(cust, pay, on="cid", how="left")
print(out["paid"].tolist())

[50.0, nan, 20.0]

Answer. A left join. Customer 2 never paid, so the result is [50.0, nan, 20.0].

17. Data Cleaning — Complete Checklist ✓

1. Look: head(), info(), describe()
2. Duplicates: df = df.drop_duplicates()
3. Wrong types: df["Age"] = df["Age"].astype(int), pd.to_numeric(col, errors="coerce")
4. Missing values: see section 14
5. Useless columns: IDs, names, unique tickets → drop (they don’t help prediction)
6. Outliers: check with describe(); clip: df["Fare"].clip(upper=df["Fare"].quantile(0.99))
7. Inconsistent text: df["city"].str.strip().str.lower()
8. Encode categories (next section)
9. Save

Worked Problem · Run The Checklist On A Messy Table

Problem. A table has a duplicated row, ages stored as text, one missing age and an extra space in a city name. Clean it step by step.

1. Remove duplicates: drop_duplicates().
2. Fix the type: pd.to_numeric(df["age"]).
3. Fill the missing age with the median.
4. Clean the text: str.strip().str.lower().
5. Print the result.

import pandas as pd
df = pd.DataFrame({"age": ["25", "25", "31", None], "city": [" Pune", " Pune",
    "Delhi ", "Delhi"]})
df = df.drop_duplicates()
df["age"] = pd.to_numeric(df["age"])
df["age"] = df["age"].fillna(df["age"].median())
df["city"] = df["city"].str.strip().str.lower()
print(df.values.tolist())

[[25.0, 'pune'], [31.0, 'delhi'], [28.0, 'delhi']]

Answer. [[25.0, 'pune'], [31.0, 'delhi'], [28.0, 'delhi']]: one duplicate removed, the missing age filled with the median 28, and the text tidied.

18. Encoding Categorical Features

Why? ML models only understand numbers. “male”/“female” must become numbers.

Method 1 — Label encoding (map to numbers) Use for two categories or ordered categories.

df["Sex"] = df["Sex"].map({"male": 0, "female": 1})
size = {"small": 0, "medium": 1, "large": 2}   # ordered → OK

Method 2 — One-hot encoding Use for unordered categories with 3+ values (city, color, port).

df = pd.get_dummies(df, columns = ["Embarked"], drop_first = True)

Before → After:

Embarked          Embarked_Q  Embarked_S
   S       →           0           1
   C                   0           0      (C is the "missing" baseline)
   Q                   1           0

drop_first=True removes one column (it’s redundant — if not Q and not S, it must be C).

Why not label-encode cities as 0,1,2? The model would think Chennai(2) is “greater than” Hyderabad(1), and that (Hyd+Chennai)/2 = Delhi. Meaningless. One-hot avoids this false ordering.

Worked Problem · Turn Text Into Numbers

Problem. Encode Sex (male/female) as 0/1 and Embarked (S/C/Q) with one-hot columns.

1. Two categories with no ordering can be mapped to 0 and 1: map({"male": 0, "female": 1}).
2. Three or more categories with no ordering need one-hot encoding, one 0/1 column per category. Use pd.get_dummies.
3. dtype=int makes the columns 0/1 instead of True/False.
4. Never label S, C, Q as 0, 1, 2 for a model that would read the numbers as an order.

import pandas as pd
df = pd.DataFrame({"Sex": ["male", "female", "male"], "Embarked": ["S", "C", "Q"]})
df["Sex"] = df["Sex"].map({"male": 0, "female": 1})
df = pd.get_dummies(df, columns=["Embarked"], dtype=int)
print(df.columns.tolist())
print(df.values.tolist())

['Sex', 'Embarked_C', 'Embarked_Q', 'Embarked_S']
[[0, 0, 0, 1], [1, 1, 0, 0], [0, 0, 1, 0]]

Answer. Columns ['Sex', 'Embarked_C', 'Embarked_Q', 'Embarked_S'], rows *[[0, 0, 0, 1], [1, 1, 0, 0], [0, 0, 1, 0]]*.

Your Turn · Ordered Categories

Problem. Sizes are small < medium < large. Encode ["small", "large", "medium", "small"] as numbers that keep the order.

Try it on paper or in a notebook first. Then check the solution.

Solution — Step By Step

1. Sizes have a natural order, so integer labels are fine here.
2. Make a dictionary: small → 0, medium → 1, large → 2.
3. Apply it with map.

import pandas as pd
s = pd.Series(["small", "large", "medium", "small"])
print(s.map({"small": 0, "medium": 1, "large": 2}).tolist())

[0, 2, 1, 0]

Answer. [0, 2, 1, 0]. Use label encoding for ordered categories and one-hot for unordered ones.

19. ★ ASSIGNMENT 2 — Titanic Cleaning (full solution)

Dataset: get titanic.csv from Kaggle or sns.load_dataset("titanic"). Goal: prepare data to predict Survived.

import pandas as pd
# 1. Load
df = pd.read_csv("titanic.csv")
print(df.shape)
print(df.isna().sum())       # Age, Cabin, Embarked have missing values
# 2. Drop columns that don't help
# Cabin: ~77% missing; Name/Ticket/PassengerId: unique per person
df = df.drop(columns = ["Cabin", "Name", "Ticket", "PassengerId"])
# 3. Missing values
df["Age"] = df["Age"].fillna(df["Age"].median())                  # numeric → median
df["Embarked"] = df["Embarked"].fillna(df["Embarked"].mode()[0])  # category → mode
# 4. Remove duplicates
# skip drop_duplicates() here: after IDs are dropped, different people can look
# alike
# 5. Encode categoricals
df["Sex"] = df["Sex"].map({"male": 0, "female": 1})
df = pd.get_dummies(df, columns = ["Embarked"], drop_first = True)
# 6. Final check - should be all numbers and zero missing
print(df.isna().sum().sum())     # 0
print(df.dtypes)
# 7. Save
df.to_csv("titanic_clean.csv", index = False)
# index=False avoids extra index column

Bonus insights with GroupBy:

df.groupby("Sex")["Survived"].mean()      # women survived far more
df.groupby("Pclass")["Survived"].mean()   # 1st class survived more

What to put in your repo Day-01-NumPy-Pandas/ :

• cosine_similarity.ipynb
• titanic_cleaning.ipynb
• titanic_clean.csv
• README.md (what you did + 3 insights)

20. Common Errors & Fixes

21. Practice Questions (test yourself)

1. What is the shape of np.zeros((3,4)).T? → (4,3)
2. a = np.arange(6).reshape(2,3); what does a.sum(axis=0) return? → [3 5 7]
3. Can you add shapes (4,3) and (4,)? Why not? → No: 3 vs 4 mismatch.
4. Multiply shapes (5,2) @ (2,7) → (5,7)
5. When would you use median instead of mean to fill missing values?
6. Why one-hot instead of label encoding for a “city” column?
7. What does df.groupby("Pclass")["Survived"].mean() calculate?

22. Quick Summary

• NumPy = fast math on arrays → think in shapes, avoid loops
• axis=0 down, axis=1 across
• Broadcasting stretches size-1 dimensions
• @ is matrix multiplication: (m,n)@(n,p)→(m,p)
• Pandas = tables; always inspect → clean → encode → save
• Cosine similarity = normalize rows, then A_norm @ B_norm.T

Next: Day 4 — Linear Regression & Gradient Descent

⚡ Quick Recap — Days 2–3

1.1 NumPy Arrays, Shapes, Indexing

NumPy array = a fast grid of numbers (all same type). Much faster than Python lists because it runs in C.

import numpy as np
a = np.array([1, 2, 3])            # 1D, shape (3,)
b = np.array([[1, 2, 3],
              [4, 5, 6]])          # 2D, shape (2, 3)  -> (rows, cols)
print(b.shape, b.ndim, b.dtype)    # (2,3) 2 int64

Shape = size in each dimension. (2, 3) → 2 rows, 3 columns.

Indexing & slicing

b[0, 1]        # row 0, col 1 -> 2
b[:, 0]        # all rows, col 0 -> [1, 4]
b[1, :]        # row 1 -> [4, 5, 6]
b[b > 2]       # boolean mask -> [3, 4, 5, 6]

Useful creators: np.zeros((3,3)), np.ones(...), np.arange(0,10,2), np.linspace(0,1,5), np.random.randn(3,3), np.eye(3).

1.2 Broadcasting and Vectorization

Vectorization = do operations on whole arrays at once — no Python loops.

# ❌ Slow
result = [x * 2 for x in lst]
# ✅ Fast
result = arr * 2

Broadcasting = NumPy automatically “stretches” smaller arrays to match bigger ones.

Rules (compare shapes from the right): 1. Dimensions are equal, or 2. One of them is 1 (gets stretched).

A = np.ones((3, 4))          # (3,4)
v = np.array([1, 2, 3, 4])   # (4,)  -> treated as (1,4)
A + v                        # ✅ works, v added to every row
col = np.array([[1],[2],[3]])  # (3,1)
A + col                        # ✅ col stretched across columns

7 (3,4) + (3,) fails (4 vs 3 don’t match).

1.3 Matrix Multiplication

• A * B → element-wise (same shape needed)
• A @ B or np.dot(A,B) → matrix multiplication

Rule: (m, n) @ (n, p) → (m, p). Inner dimensions must match.

A = np.random.randn(2, 3)
B = np.random.randn(3, 4)
C = A @ B      # shape (2, 4)

Why it matters: every neural network layer is output = X @ W + b.

1.4 Reshape & Basic Linear Algebra

x = np.arange(12)
x.reshape(3, 4)      # 3x4
x.reshape( - 1, 6)   # -1 = "figure it out" -> (2,6)
x.flatten()          # back to 1D
A.T                  # transpose

axis trick: axis=0 collapses rows (result per column). axis=1 collapses columns (result per row).

1.5 ★ Assignment: Cosine Similarity (No Loops)

Idea: Cosine similarity measures the angle between two vectors. 1 = same direction, 0 = unrelated, -1 = opposite.

cos(a, b) = (a · b) / (‖a‖ · ‖b‖)

For two embedding matrices A (n, d) and B (m, d):

def cosine_sim_matrix(A, B):
    A_norm = A / np.linalg.norm(A, axis = 1, keepdims = True) # normalize rows
    B_norm = B / np.linalg.norm(B, axis = 1, keepdims = True)
    return A_norm @ B_norm.T        # shape (n, m)

Why it works: after normalizing to length 1, dot product = cosine. One matrix multiplication computes all pairs.

Used in: semantic search, RAG, recommendation.

1.6 Pandas Basics

DataFrame = a table (rows + columns). Series = one column.

import pandas as pd
df = pd.read_csv("titanic.csv")
df.head() ; df.info() ; df.describe() ; df.shape
df["Age"]                 # one column (Series)
df[["Name", "Age"]]       # multiple columns
df.loc[0, "Age"]          # by label
df.iloc[0, 3]             # by position

1.7 Filtering & Missing Values

df[df["Age"] > 30]
df[(df["Sex"] == "male") & (df["Survived"] == 1)]   # use & | ~ with ()
df["Embarked"].isin(["S", "C"])
df.isna().sum()                          # count missing per column
df["Age"].fillna(df["Age"].median())     # fill with median
df.dropna(subset = ["Embarked"])         # drop rows
df.drop(columns = ["Cabin"])             # drop mostly-empty column

1.8 GroupBy & Merge

df.groupby("Pclass")["Survived"].mean()            # survival rate per class
df.groupby(["Sex","Pclass"]).agg({"Age":"mean","Fare":"max"})
pd.merge(df1, df2, on = "id", how = "left")        # SQL-style join
# how = inner | left | right | outer

1.9 Data Cleaning Checklist ✓

1. df.info() → check types
2. Handle missing values
3. Remove duplicates: df.drop_duplicates()
4. Fix types: df["Age"].astype(int)
5. Handle outliers (clip / remove)
6. Encode categoricals
7. Save: df.to_csv("clean.csv", index=False)

1.10 Encoding Categorical Features

ML models need numbers only. Never label-encode unordered categories (model thinks 2 > 1 means something).

Titanic Pipeline Summary

df = pd.read_csv("titanic.csv")
df["Age"] = df["Age"].fillna(df["Age"].median())
df["Embarked"] = df["Embarked"].fillna(df["Embarked"].mode()[0])
df = df.drop(columns = ["Cabin", "Ticket", "Name", "PassengerId"])
df["Sex"] = df["Sex"].map({"male":0, "female":1})
df = pd.get_dummies(df, columns = ["Embarked"], drop_first = True)
df.to_csv("titanic_clean.csv", index = False)

Beginner checkpoint — Days 2–3 ■ Explain the day’s main idea without using the textbook wording. ■ Write one tiny example from memory. ■ Identify one common failure mode. ■ Explain how today’s topic connects to the next day. ■ Write one question you still cannot answer.

Move on when you can… ☐ load a CSV and inspect it with head(), info() and describe()

☐ filter rows with & and |

☐ count missing values and fill them with a sensible value ☐ use groupby and merge

☐ turn text columns into numbers ☐ explain why we clean data before modeling If you cannot tick most of these, repeat the worked problems for this day. Taking an extra day is fine; it is not falling behind.