A Crashcourse in using Make for Data Scientists
Mental model#
Make builds a dependency graph of targets and runs a recipe only when its target is missing or older than any prerequisite (file mtime comparison). Targets that are not files are declared .PHONY and always run.
target: prereq1 prereq2 | order-only-prereq
recipe line 1 # TAB-indented; each line runs in its own shell
@recipe line 2 # @ = don't echo the command
-recipe line 3 # - = ignore a non-zero exit
- Order-only prerequisites (after
|) must exist but don’t trigger rebuilds by being newer. Use for output directories. - The first target in the file is the default for a bare
make.
Variables and environment#
| Syntax | Meaning | Use for |
|---|---|---|
X = val |
Recursive: expanded every time it’s used | Values that reference later-defined vars |
X := val |
Simple: expanded once, at definition | Default choice; $(shell ...) results |
X ?= val |
Set only if not already set (env or CLI) | Overridable defaults: CSV ?= data/raw/x.csv |
X += val |
Append | Building flag lists |
$(X) / $$X |
Make variable / shell variable inside a recipe | $$ escapes $ for the shell |
Precedence, highest first: command line (make train EPOCHS=5) → Makefile → environment.
Load .env into every recipe:
include .env
export
- Make keeps inline
# commentsas part of the value; put.envcomments on their own line. - Alternative that leaves Make out of it:
uv run --env-file .env ....
File targets: incremental data pipelines#
Make skips steps whose outputs are newer than their inputs. That gives you cached ETL and feature steps for free.
data/processed/features.parquet: data/raw/txns.csv src/features.py | data/processed
uv run python src/features.py $< $@
models/model.pkl: data/processed/features.parquet src/train.py
uv run python src/train.py --in $< --out $@
data/processed:
mkdir -p $@
| Automatic variable | Value |
|---|---|
$@ |
The target |
$< |
First prerequisite |
$^ |
All prerequisites, deduplicated |
$* |
The stem matched by % |
Pattern rule, one rule for many files:
data/clean/%.parquet: data/raw/%.csv
uv run python -m pkg.clean $< $@
- List the script itself as a prerequisite, so editing the code reruns the step.
- Steps whose outputs aren’t single files (Delta tables, DB loads): touch a marker file such as
.done/loadas the target.
DS/AI recipes#
.DEFAULT_GOAL := help
.PHONY: help setup lint test up down train eval
help: ## List targets
@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk -F':.*?## ' '{printf "%-12s %s\n", $$1, $$2}'
setup: ## Install deps (lockfile)
uv sync --frozen
lint: ## Ruff check + format check
uv run ruff check . && uv run ruff format --check .
test: lint ## Unit tests
uv run pytest -q $(ARGS)
up: ## Start stack, wait for healthchecks
docker compose up -d --wait
train: up ## Train; override: make train EPOCHS=5
uv run python -m pkg.train --epochs $(EPOCHS)
EPOCHS ?= 10
## commentafter a target plus thehelprecipe gives a self-documentingmakewith no arguments.ARGSpass-through:make test ARGS="-k scd2 -x".- Same targets locally and in CI (
make lint test), so CI never drifts from what you run.
CLI flags#
| Flag | Does | When |
|---|---|---|
-n |
Dry run: print recipes, run nothing | Before anything destructive |
-B |
Rebuild everything, ignoring timestamps | Force a full pipeline rerun |
-j N |
Run N independent targets in parallel | Independent feature or data steps |
-k |
Keep going after a failure | Seeing every failing step at once |
-C dir |
Run in another directory | Monorepo subprojects |
--trace |
Show why each target is rebuilt | “Why did this rerun?” (GNU Make 4+) |
-p |
Dump the whole rule and variable database | Deep debugging |
Gotchas#
| Symptom | Cause | Fix |
|---|---|---|
missing separator |
Recipe indented with spaces | Use a TAB, or set .RECIPEPREFIX = > |
cd or export in a recipe has no effect on the next line |
Each line runs in a new shell | Chain with && on one line, or add .ONESHELL: |
| Target “is up to date” but never runs | A file or folder with the same name exists | Add it to .PHONY |
| A failing command in a pipe passes | sh only checks the last command’s exit status |
SHELL := bash and .SHELLFLAGS := -eu -o pipefail -c |
$VAR comes out empty |
Make expanded it before the shell saw it | $$VAR |
| A broken step left a half-written output that now looks up to date | Make treats any existing output as done | Add .DELETE_ON_ERROR: |
| A step reruns every time | One of its prerequisites is .PHONY |
Use an order-only | prerequisite, or a marker file |