A Crashcourse in using Make for Data Scientists

Mental model

Make builds a dependency graph of targets and runs a recipe only when its target is missing or older than any prerequisite (file mtime comparison). Targets that are not files are declared .PHONY and always run.

target: prereq1 prereq2 | order-only-prereq
	recipe line 1      # TAB-indented; each line runs in its own shell
	@recipe line 2     # @ = don't echo the command
	-recipe line 3     # - = ignore a non-zero exit
  • Order-only prerequisites (after |) must exist but don’t trigger rebuilds by being newer. Use for output directories.
  • The first target in the file is the default for a bare make.

Variables and environment

Syntax Meaning Use for
X = val Recursive: expanded every time it’s used Values that reference later-defined vars
X := val Simple: expanded once, at definition Default choice; $(shell ...) results
X ?= val Set only if not already set (env or CLI) Overridable defaults: CSV ?= data/raw/x.csv
X += val Append Building flag lists
$(X) / $$X Make variable / shell variable inside a recipe $$ escapes $ for the shell

Precedence, highest first: command line (make train EPOCHS=5) → Makefile → environment.

[Read more]

Basic LLM Monitoring with Grafana and Prometheus

Note, this post assumes you have a basic understanding of containerization and kubernetes works. I highly suggest skimming my introduction to k3s first.

Repo for this project lives here: llm-serving-vllm-k8s

In this blog post, I walk through how to setup Prometheus for monitoring the pods we set up in my previous post. Additionally, we walk through setting up monitoring for the local vLLM instance using Grafana.

Some Context

Prometheus is the thing that we used to monitor vLLM. It works on a pull-based model, which means that it periodically queries some endpoint (in this case /metrics using a GET request) on each target, and stores the numbers it scrapes. This means that “deploying” Prometheus involves the following:

[Read more]

Spinning up a simple k3s to manage a local LLM Docker Container

Link to Github

In this blog post, I walk through how to take a lone Docker image and use k3s to manage it on a single-node machine (i.e. my desktop). The idea behind Docker is an isolated, self-sufficient container that can run anywhere, k3s and its bigger brother k8s (Kubernetes) is used to manage, in a declarative fashinon, these containers.

The core model of Kubernetes, and the main difference from simply calling docker run is the declarative reconciliation. Kubernetes is a control system built around a single loop:

[Read more]

Building a multi-stage Docker image for locally serving an LLM

Serving a small, open-source LLM behind vLLM, containerized on a single desktop GPU.

The repo is here: https://github.com/codecalligrapher/llm-serving-vllm-k8s

The target is to have a single-node Kubernetes, with Grafana to monitor and a throughput benchmark.

Status: containerized OpenAI-compatible endpoint running on an RTX 3060 Ti (8GB). K8s, monitoring, and benchmark are scoped below but not yet built.

Why vLLM

Mainly because of the OpenAI-compatible server, so the endpoint is v1/chat/completions and any OpenAI client works against it.

[Read more]

Writing a Trainable Attention Mechanism in Tensorflow

The primary goal of langugae models is next-word/next-sequence prediction. The transformer architecture is built on the premise of “attention”, this was developed to solve the precursor’s weakness in modeling long-length text sequences (namely recursive neural networks).

In its most basic format, an RNN is a reduce over a sequence with a carried accumulator (in machine-learning terms, the current state).

For context, reduce(function, iterable, initializer) goes through the iterable from left to right, applying the function at each call. It takes the output of function(accumulator, item) at each call, returning the new accumulator and moves to the next item. The initializer is the starting accumulator/state

[Read more]