3.2 Pandas: series, dataframes, indexing

What this is and why it exists

Pandas is where you will spend more hours than in any other library, because most data arrives as a table and a dataframe is the standard vehicle for one. This topic is about loading a messy file and getting a clean, correctly-typed dataframe out of it — and about the selection habits that separate a clean session from a mysterious one where changes do not take effect.

The vocabulary

• Series — one labelled column.
• DataFrame — a table of Series sharing an index.
• Index — the row labels, which are not the same thing as row positions.
• loc — selection by label; the documentation says it "is primarily label based, but may also be used with a boolean array".
• iloc — selection by position; "primarily integer position based (from 0 to length-1 of the axis)".
• Boolean mask — a column of true and false used to select rows.
• dtype — a column's type; the thing that decides whether your numbers are numbers.
• Categorical — a type for a column with few distinct values, stored compactly.
• Parquet — a columnar file format that keeps types and reads far faster than CSV.

The mental model

A dataframe is a set of columns, each of one type, sharing a set of row labels. Two things follow immediately, and both cause beginner confusion.

First, the index is labels, not positions. After filtering, a row's label may be 4,207 while its position is 0. loc speaks labels and iloc speaks positions, and mixing them up produces either an error or, worse, the wrong row. There is one asymmetry worth memorising: with loc, a label slice includes both ends — the documentation notes that "contrary to usual Python slices, both the start and the stop are included" — while iloc follows the usual rule and excludes the stop.

Second, dtypes decide meaning. A column read from a CSV with one stray value in it becomes the general object type, and every number in it is then a string: sums concatenate, comparisons sort alphabetically, and nothing warns you. So the first thing to do after loading anything is to look at the dtypes and confront any column that should be numeric and is not. That single habit prevents a large fraction of all pandas confusion.

Then the selection rule that this topic exists to teach. Never assign through two indexing operations. Modern pandas is explicit: "chained assignment will never work", because the technique "would have to modify the view and the parent in one step", and it raises rather than silently doing nothing. The documented replacement is a single loc with both the row condition and the column: select the rows and the column in one operation, and assign to that. The deeper rule behind it is worth carrying: "any DataFrame or Series derived from another in any way always behaves as a copy" — so an edit to something you sliced out changes the slice, never the parent.

Reading files is the last piece and the choice of format matters more than people expect. CSV is universal and lossy: it has no types, so every load re-guesses them, and a large file is slow to parse every single time. Parquet is columnar and keeps its types, so it loads far faster, uses less space, and does not lose the distinction between a number and text. The practical pattern for any project: read the original CSV once, clean it, write Parquet, and work from that afterwards.

Categoricals are the memory lever. A column holding a handful of distinct strings repeated a million times is enormously wasteful as text and small as a categorical, and grouping on it is faster too. Converting the two or three obvious columns is often the difference between a dataset that fits in memory and one that does not.

In code

Checked against the pandas user guide.

import pandas as pd

df = pd.read_csv(
    "scores.csv",
    dtype={"student_id": "string", "subject": "category"},
    parse_dates=["taken_on"],
)
print(df.dtypes)            # look at this every single time

# Label-based and position-based selection are different operations.
first_ten = df.iloc[:10]                       # positions 0-9
one_student = df.loc[df["student_id"] == "S042"]
maths = df.loc[df["subject"] == "maths", ["student_id", "score"]]

# Assignment: one .loc, rows and column together.
df.loc[df["score"] > 100, "score"] = pd.NA     # correct
# df[df["score"] > 100]["score"] = pd.NA       # chained: raises, changes nothing

df.to_parquet("scores.parquet")                # types preserved, loads faster

Two details are the lesson. The dtypes are declared at read time rather than corrected afterwards, which stops the "one stray value made this column text" failure at its source. And the assignment uses one loc with both parts, because the two-step form is the one pandas now refuses outright.

What you should now be able to explain or do

Say what the index is and why a row's label is not its position. Choose between loc and iloc correctly, including the slice-endpoint difference. Read the dtypes after every load and say what an unexpected object column means. Write a conditional assignment in one operation and say why the two-step version fails. Convert a repeated-string column to a categorical and say what you saved. Explain when to move a project from CSV to Parquet.

Check yourself