Link to Github

In this blog post, I walk through how to take a lone Docker image and use k3s to manage it on a single-node machine (i.e. my desktop). The idea behind Docker is an isolated, self-sufficient container that can run anywhere, k3s and its bigger brother k8s (Kubernetes) is used to manage, in a declarative fashinon, these containers.

The core model of Kubernetes, and the main difference from simply calling docker run is the declarative reconciliation. Kubernetes is a control system built around a single loop:

Serving a small, open-source LLM behind vLLM, containerized on a single desktop GPU.

The repo is here: https://github.com/codecalligrapher/llm-serving-vllm-k8s

The target is to have a single-node Kubernetes, with Grafana to monitor and a throughput benchmark.

Status: containerized OpenAI-compatible endpoint running on an RTX 3060 Ti (8GB). K8s, monitoring, and benchmark are scoped below but not yet built.

Why vLLM

Mainly because of the OpenAI-compatible server, so the endpoint is v1/chat/completions and any OpenAI client works against it.