Spinning up a simple k3s to manage a local LLM Docker Container

Link to Github

In this blog post, I walk through how to take a lone Docker image and use k3s to manage it on a single-node machine (i.e. my desktop). The idea behind Docker is an isolated, self-sufficient container that can run anywhere, k3s and its bigger brother k8s (Kubernetes) is used to manage, in a declarative fashinon, these containers.

The core model of Kubernetes, and the main difference from simply calling docker run is the declarative reconciliation. Kubernetes is a control system built around a single loop:

observe current state → compare to desired state → act to close the gap → repeat forever

A desired state is stored, and a set of controllers runt he loop do drive the actual state towards it. This is the main difference between docker run, which is imperative: it performs a single action and doesn’t maintain the state; if the container crashes, it crashes. If a pod within Kubernetes dies, the controller will attempt to recreate it.

Before I get started, there’s some terminology to become familiar with:

k3s: this is the lighter version of Kubernetes with less overhead, and small enough to run on my desktop
API server: this is an HTTP server fronting a datastore, in this case etcd. It holds all the objects
kubectl: this is an HTP client for the server, the commands I’m going to use the most are kubectl apply and kubcetl get. Both of these functions don’t have any logic on their own, they just serialize some YAML declarations into API calls.
scheduler: this is a specialized contrller that assigns unscheduled “pods” to nodes based on resource requests
kubelet: this the per-node agent
Node: this is the machines used to run the pods
Pod: this is the smallest deployable unit, a gorup of one or more containers that are always scheduled together on the same node and share certain namespaces. In most setups, a pod holds exactly one container. If there are more than one containers in a pod, they share: network namespaces, storage volumes and container lifecyle.

Side note: pods are ephemeral and immutable in identity. A pod isn’t restared, a pod that dies is replaced by a new pod witha new name and identity. An issue I ran into while learning how this works is that I ended up with 4 pods all trying to use my one GPU, which caused all to run out of memory.

In this blog, I’m using a multi-stage Dockerfile from my previous-post that I used to spin up a local LLM. It’s fairly straightforward, but separates pulling down the model used and actually serving it using vllm, thereby reducing cold-start latency. It can be found in my Github codecalligrapher/llm-serving-vllm-k8s/Dockerfile

That Docker image is exported: docker save llm-serving:gpu -o /tmp/llm.tar so that I have an importable .tar file for k3s to reference.

Setting up `k3s`#

Run curl -sfL https://get.k3s.io | sh -. This sets up a single-node cluster (server and agent in one). k3s runs as a systemd service and ships with it its own containerd and kubectl.

From there, we need to ensure that k3s points to a configuration that does not exist in a root-only path:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

In order for k3s to be able to use my GPU, and in the previous post we verified that docker can actually access the GPU, we need to ensure k3s defaults to the NVIDIA runtime. In order to do this, we need to deploy the device plugin that advertises my GPU as a schedulable resource:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.19.2/deployments/static/nvidia-device-plugin.yml

I ran into an error, where I had to modify the system file for k3s. In order to do this, I had to use systemctl edit:

sudo systemctl edit k3s --full
# find the ExecStart line, change it to:
#   ExecStart=/usr/local/bin/k3s server --default-runtime nvidia
# save and exit
sudo systemctl daemon-reload
sudo systemctl restart k3s

In order to verify that step worked, running kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}{"\n"}' should return a $1$, indicating that a GPU exists and is indeed allocatable.

Serving my LLM using k3s#

Create the yaml file to describe what I want to be served:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-svc
spec:
  replicas: 1                       # one GPU → one replica, non-negotiable here
  selector:
    matchLabels: {app: vllm}
  template:
    metadata:
      labels: {app: vllm}
    spec:
      enableServiceLinks: false
      containers:
      - name: vllm
        image: llm-serving:gpu
        imagePullPolicy: Never       # use the imported image, don't pull
        args: ["--model","Qwen/Qwen2.5-0.5B-Instruct", "--gpu-memory-utilization","0.35","--max-model-len","1024", "--enforce-eager"]
        ports:
        - {containerPort: 8000}
        resources:
          limits:
            nvidia.com/gpu: 1        # the line that makes k8s assign the GPU
        volumeMounts:
        - {name: shm, mountPath: /dev/shm}
        startupProbe:                # gates liveness while the model loads
          httpGet: {path: /health, port: 8000}
          periodSeconds: 10
          failureThreshold: 30       # 30×10s = 5 min grace to become ready
        livenessProbe:
          httpGet: {path: /health, port: 8000}
          periodSeconds: 10
        readinessProbe:
          httpGet: {path: /health, port: 8000}
          periodSeconds: 5
      volumes:
      - name: shm
        emptyDir: {medium: Memory, sizeLimit: 2Gi}
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
spec:
  selector: {app: vllm}
  ports:
  - {port: 8000, targetPort: 8000, name: http}

Things to pay attention to:

The GPU request where limit is set to 1. If that’s left out it runs on CPU, since I only have one GPU. If I left the runtime as runc , I would need to set runtimeClass: nvidia
The mount for /dev/shm is the k8s transalation of --ipc-host . dev/shm is a location which lets applications on linux to pass data to each other equickly. This sets up tmpfs RAM-backed scratch space for the pods
startupProbe prevents crash loops. vLLM takes a hwile to load weights, and livenessProbe starts checking immediately, which will fail while the model is still laoading, and will kil the k8s containerfor being unhealthy, repeatedly, forever. Once startupProbe passes once, liveness takes over for ongoing health. The three probes do distinct jobs: startup = “is it up yet, don’t kill it meanwhile”; liveness = “is it wedged, restart it”; readiness = “can it take traffic, add/remove from the Service.”

Applying the Config#

This part I had to wrap my head around, since it separates the imperative manner in which docker run operates by the declarative paradign of Kubernetes. kubectl apply writes to the API server, and returns as soon as the write is persisted. Mechanically it:

Serializes each object in the YAML
For each, does a three-way merge: comparing my new manifest, the live object and the last-applied-configuration from whatever previous apply was done. It patches only a diff. It’s an upsert not a create.
Persists the merged state, and returns

Of note, apply doesn’t execute the workload, it only records the state. The return value confirms the write worked, while everything afterwards is asynchronous.

So the steps are as follows:

sudo k3s ctr -n k8s.io images import /tmp/llm.tar: this imports my exported Docker iamge into k3s
sudo k3s ctr -n k8s.io images ls | grep llm-serving: this confirms that the image was successfully imported
kubectl apply -f vllm.yaml: this is the apply mentioned above, which sets up the loop to maintain the docker container
Finally kubectl port-forward svc/vllm 8000:8000 enables port-forwarding from the managed container, so I can query it locally

Once this is set up, run kubectl get pods -w and it should return something like the following:

NAME                       READY   STATUS    RESTARTS   AGE
vllm-svc-db6f67c58-gddt7   1/1     Running   0          3h12m

Testing The Container Endpoint#

Running the following simple cURL command:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"hi"}]}'

Returns:

{
  "id": "chatcmpl-a66f98c533cca45e",
  "object": "chat.completion",
  "created": 1789165375,
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today? If you have any questions or need help with anything specific, feel free to ask and I'll do my best to provide the information or guidance you're looking for.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.28.0-a73339b9",
  "usage": {
    "prompt_tokens": 30,
    "total_tokens": 73,
    "completion_tokens": 43,
    "prompt_tokens_details": null,
    "completion_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null,
  "ec_transfer_params": null,
  "metrics": null
}

Additionally, I can verify that the pod is running and view the -tail of the logs using:

kubectl logs -f deploy/vllm-svc

# Output
(APIServer pid=1) INFO:     10.42.0.1:46978 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:46984 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:47678 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:47694 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:47710 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:43030 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:43040 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.42.0.1:43046 - "GET /health HTTP/1.1" 200 OK

And we’re done!

Spinning up a simple k3s to manage a local LLM Docker Container

Setting up k3s#

Serving my LLM using k3s#

Applying the Config#

Testing The Container Endpoint#

Setting up `k3s`#