Mental model#

Make builds a dependency graph of targets and runs a recipe only when its target is missing or older than any prerequisite (file mtime comparison). Targets that are not files are declared .PHONY and always run.

target: prereq1 prereq2 | order-only-prereq
	recipe line 1      # TAB-indented; each line runs in its own shell
	@recipe line 2     # @ = don't echo the command
	-recipe line 3     # - = ignore a non-zero exit
  • Order-only prerequisites (after |) must exist but don’t trigger rebuilds by being newer. Use for output directories.
  • The first target in the file is the default for a bare make.

Variables and environment#

Syntax Meaning Use for
X = val Recursive: expanded every time it’s used Values that reference later-defined vars
X := val Simple: expanded once, at definition Default choice; $(shell ...) results
X ?= val Set only if not already set (env or CLI) Overridable defaults: CSV ?= data/raw/x.csv
X += val Append Building flag lists
$(X) / $$X Make variable / shell variable inside a recipe $$ escapes $ for the shell

Precedence, highest first: command line (make train EPOCHS=5) → Makefile → environment.

Load .env into every recipe:

include .env
export
  • Make keeps inline # comments as part of the value; put .env comments on their own line.
  • Alternative that leaves Make out of it: uv run --env-file .env ....

File targets: incremental data pipelines#

Make skips steps whose outputs are newer than their inputs. That gives you cached ETL and feature steps for free.

data/processed/features.parquet: data/raw/txns.csv src/features.py | data/processed
	uv run python src/features.py $< $@

models/model.pkl: data/processed/features.parquet src/train.py
	uv run python src/train.py --in $< --out $@

data/processed:
	mkdir -p $@
Automatic variable Value
$@ The target
$< First prerequisite
$^ All prerequisites, deduplicated
$* The stem matched by %

Pattern rule, one rule for many files:

data/clean/%.parquet: data/raw/%.csv
	uv run python -m pkg.clean $< $@
  • List the script itself as a prerequisite, so editing the code reruns the step.
  • Steps whose outputs aren’t single files (Delta tables, DB loads): touch a marker file such as .done/load as the target.

DS/AI recipes#

.DEFAULT_GOAL := help
.PHONY: help setup lint test up down train eval

help:                 ## List targets
	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk -F':.*?## ' '{printf "%-12s %s\n", $$1, $$2}'

setup:                ## Install deps (lockfile)
	uv sync --frozen

lint:                 ## Ruff check + format check
	uv run ruff check . && uv run ruff format --check .

test: lint            ## Unit tests
	uv run pytest -q $(ARGS)

up:                   ## Start stack, wait for healthchecks
	docker compose up -d --wait

train: up             ## Train; override: make train EPOCHS=5
	uv run python -m pkg.train --epochs $(EPOCHS)

EPOCHS ?= 10
  • ## comment after a target plus the help recipe gives a self-documenting make with no arguments.
  • ARGS pass-through: make test ARGS="-k scd2 -x".
  • Same targets locally and in CI (make lint test), so CI never drifts from what you run.

CLI flags#

Flag Does When
-n Dry run: print recipes, run nothing Before anything destructive
-B Rebuild everything, ignoring timestamps Force a full pipeline rerun
-j N Run N independent targets in parallel Independent feature or data steps
-k Keep going after a failure Seeing every failing step at once
-C dir Run in another directory Monorepo subprojects
--trace Show why each target is rebuilt “Why did this rerun?” (GNU Make 4+)
-p Dump the whole rule and variable database Deep debugging

Gotchas#

Symptom Cause Fix
missing separator Recipe indented with spaces Use a TAB, or set .RECIPEPREFIX = >
cd or export in a recipe has no effect on the next line Each line runs in a new shell Chain with && on one line, or add .ONESHELL:
Target “is up to date” but never runs A file or folder with the same name exists Add it to .PHONY
A failing command in a pipe passes sh only checks the last command’s exit status SHELL := bash and .SHELLFLAGS := -eu -o pipefail -c
$VAR comes out empty Make expanded it before the shell saw it $$VAR
A broken step left a half-written output that now looks up to date Make treats any existing output as done Add .DELETE_ON_ERROR:
A step reruns every time One of its prerequisites is .PHONY Use an order-only | prerequisite, or a marker file