From a container to Kubernetes: serving models for real

In AI & MLOps, from the ground up, I described my 24-week plan: a notebook, then a tested package, a tracked pipeline, an API with FastAPI, a container, and eventually a deployment in the cloud with monitoring.

A container is a great milestone. But a single container on a laptop isn't how models run in production. This post looks at the next step: what changes when you move model serving onto Kubernetes, and what's different when the model is a large language model that needs GPUs.

Where my project stands

The plan's core project is a classic machine learning model: scikit-learn, served by FastAPI, packaged with Docker. For that kind of model, Kubernetes deployment is close to any other web service. A Deployment, a Service, health checks, resource requests, and a HorizontalPodAutoscaler.

LLMs are a different world. The model files are gigabytes, they need GPUs, they load slowly, and one request can take seconds. That's where the interesting DevOps problems are.

Why Docker alone isn't enough anymore

A single container can't:

• Scale with traffic, up or down.
• Restart itself on another machine if the machine dies.
• Roll out a new model version gradually and roll back if quality drops.
• Share expensive GPU machines between workloads.

Kubernetes handles all of that. The model becomes just another workload, using the same pipelines, monitoring and security as everything else.

vLLM, KServe and the other options

The serving stack for LLMs on Kubernetes usually has two layers.

The inference engine actually runs the model:

• vLLM is the most popular open-source engine. It's fast thanks to clever memory management and batching, and exposes an OpenAI-compatible API.
• Hugging Face TGI, SGLang and NVIDIA Triton / TensorRT-LLM are alternatives with different strengths.
• Ollama is great on a laptop and for small setups, less so for heavy production traffic.

The serving platform manages the engine on Kubernetes:

• KServe gives you an InferenceService resource: you describe the model and runtime, and it handles deployment, scaling and routing, including scale to zero.
• llm-d is a newer project focused on distributed LLM inference on Kubernetes, with smart routing that sends requests to the replica most likely to have the right data cached.
• Or just a plain Deployment running vLLM, which is a perfectly fine way to start.

A minimal vLLM deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm
spec:
  replicas: 1
  selector:
    matchLabels: { app: llm }
  template:
    metadata:
      labels: { app: llm }
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest   # pin a specific version in real use
        args: ["--model", "Qwen/Qwen2.5-1.5B-Instruct"]
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
        readinessProbe:
          httpGet: { path: /health, port: 8000 }
          initialDelaySeconds: 60

Serving a model is the easy demo. Serving it reliably, cheaply and at scale is the actual MLOps job.

GPUs: scheduling and cost

GPUs change how you run Kubernetes:

• Device plugins / GPU Operator. The NVIDIA GPU Operator installs drivers and the device plugin so pods can request nvidia.com/gpu.
• Dedicated node pools with taints, so only GPU workloads land on expensive GPU nodes.
• Sharing GPUs with MIG (splitting one GPU into isolated slices) or time-slicing for smaller models.
• Dynamic Resource Allocation (DRA), now stable in Kubernetes, gives a more flexible way to request specific kinds of GPUs and devices.

And cost: an idle GPU node can cost more per day than an entire small web application per month. Monitoring GPU utilisation (the DCGM exporter for Prometheus) is not optional.

Scaling on demand

CPU-based autoscaling doesn't work well for LLMs. A GPU can be fully busy while the CPU is idle. Better signals:

• Queue length or pending requests in the inference engine.
• Tokens per second or time to first token.
• KEDA can scale on these metrics from Prometheus.

Also plan for slow starts. Downloading and loading a model can take minutes. Caching model files on a persistent volume or node-local storage, and keeping at least one warm replica, makes scaling far less painful.

What's next in the series

Next on my list: monitoring the model in production, including the quality of its answers, not just its uptime. That's where tools like Prometheus, Grafana and Evidently come in.

My take: Kubernetes doesn't make AI simple, but it makes AI workloads look like the rest of your infrastructure. And that's half the battle.

— Desly