3.9 Bigger-than-memory data
You can work with a 20 GB file on a 16 GB laptop.
Before:01. Mathematics for Machine Learning02. Python — Basics to AdvancedUnlocks:05. Classical Machine Learning
Datasets outgrow memory long before they justify a cluster, and chunking, Polars and DuckDB let a laptop handle files many times its RAM — with Parquet's columnar storage as the enabling trick. It closes the data module as the scaling answer. The confusion it corrects is reflexively reaching for distributed tools: a twenty-gigabyte file is a DuckDB one-liner, and the cluster's overhead only pays for itself far beyond laptop scale.
Work through these
Chunking and streaming in Pandas
Reading a large file in pieces and processing each piece in turn. It is the simplest technique here and it works with tools you already know.
Getting Started with pandas · TutorialPolars and lazy evaluation
A newer dataframe library that plans the whole computation before running it, which lets it avoid work and use memory efficiently. The planning step is what makes the difference.
DuckDB for SQL over local files
An embedded analytical database that queries files on disk directly, with no server to install. For many local analysis tasks it is faster than loading the data at all.
SQLBolt — Interactive SQL Lessons · TutorialParquet, columnar storage and partitioning
A columnar file format that stores each column separately and compresses well, with partitioning that lets a query skip most of the file. This is the standard storage choice for analytical data.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.