2.5 Files, paths and data I/O

What this is and why it exists

Every project starts by reading files and ends by writing them, and doing that badly produces the two most common failures in student code: paths that work on one operating system and break on another, and files left open when something goes wrong. This topic fixes both, and covers the four formats that will hold nearly every dataset and configuration you meet this year.

The vocabulary

• Path object — a path as a thing with methods, rather than a string you concatenate.
• Absolute and relative — anchored at the filesystem root, or measured from where you are.
• Context manager — an object that sets something up and reliably tears it down.
• The with statement — the syntax that uses a context manager.
• CSV — rows of comma-separated fields; a spreadsheet's plainest export.
• JSON — nested objects and arrays; the format most interfaces speak.
• JSONL — one JSON object per line, so a huge file can be read one record at a time.
• YAML — an indentation-based configuration format, easier for humans to write and easier to get subtly wrong.

The mental model

Use pathlib rather than string concatenation, and the reason is not style. The module "offers classes representing filesystem paths with semantics appropriate for different operating systems" — which means the same code produces the right separators on Windows and on Linux, where a hand-built string with slashes in it does not. The slash operator joins: the documentation shows that a path divided by a name "helps create child paths", so data / "raw" / "scores.csv" reads like a path and behaves like one everywhere. Path objects also carry the operations you want — checking existence, creating parent directories, listing matches, and reading or writing text in one call.

The with statement is the other habit, and it exists because of what happens when things go wrong. Opening a file and closing it at the end works right up until an exception is raised in between, at which point the file stays open — and on a long-running program, enough of those exhausts the operating system's limit. A context manager guarantees the cleanup happens whether the block finished normally or raised. That is why every file you open should be opened in a with, without exception, and why the same pattern shows up later for database connections and locks.

Then the formats, chosen by what the data is.

CSV is for tables and it is deceptively simple: the moment a field contains a comma, a quote or a newline, hand-splitting on commas is wrong. Use the csv module, which knows the quoting rules — and if the data is really tabular and you are about to analyse it, the next module's pandas is the better tool.

JSON is for nested structure, and it is what almost every interface returns. Its limitation is that reading a large JSON file means holding all of it in memory at once, because the closing bracket is at the end.

JSONL removes that limitation by putting one object on each line, so a file of any size can be streamed a record at a time with constant memory. For logs, model outputs and datasets that grow, this is the format to reach for.

YAML is for configuration a human edits. It is pleasant to read and it has sharp edges — significant indentation, and values that look like text being interpreted as something else — so validate what you load rather than trusting it, and keep configuration files short enough to read in one screen.

And the trap the whole topic shares: name the encoding. A file written on one machine and read on another with a different default produces either an error or, worse, wrong characters. Passing encoding="utf-8" on every read and write is a two-second habit that removes an entire category of bug reports.

In code

Checked against the pathlib, csv and json references.

import csv
import json
from pathlib import Path

data = Path("data")
data.mkdir(parents=True, exist_ok=True)

with (data / "scores.csv").open(newline="", encoding="utf-8") as f:
    rows = list(csv.DictReader(f))

with (data / "scores.json").open("w", encoding="utf-8") as f:
    json.dump(rows, f, ensure_ascii=False, indent=2)

# One record per line: readable with constant memory, whatever the size.
with (data / "scores.jsonl").open("w", encoding="utf-8") as f:
    for r in rows:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")

with (data / "scores.jsonl").open(encoding="utf-8") as f:
    for line in f:
        record = json.loads(line)

Two details worth noticing. ensure_ascii=False keeps Telugu or Hindi text as itself rather than as escape sequences, which matters as soon as your data is not English. And the JSONL read loops over the file object directly, which yields one line at a time — the file is never held whole in memory, so the same three lines work on a file larger than your machine's RAM.

What you should now be able to explain or do

Build a path that works on any operating system without string concatenation. Say what a with block guarantees and what goes wrong without one. Choose between CSV, JSON, JSONL and YAML for four described jobs. Read a JSONL file of any size with constant memory. State the one argument that prevents most encoding failures, and why it matters more for Indian-language text than for English.

Check yourself