LLM Serving on Kubernetes

Round 1 — Mental Model

Think of a restaurant kitchen with a few very expensive chefs (GPUs). Cooking one dish at a time wastes them — so orders are batched onto the same stove pass. But diners arrive continuously and their meals take different times, so you don't wait for a full table before starting; you slot new orders into the pass as plates leave (continuous batching). The maître d' (the router) seats arrivals across kitchens, and when the queue out front grows, you open another kitchen (autoscale a replica) — which takes minutes because a new chef must first read the entire recipe book (load ~100+ GB of weights).

Two clocks matter to the diner: how long until the first bite (time-to-first-token, set by prefill) and how fast the rest arrives (inter-token latency, set by decode). Batch harder and the kitchen is efficient but every diner waits longer between bites. That tension — throughput vs latency — is the whole game.

Round 2 — Internal Mechanics & Mathematical Model

The two latency metrics

Serving quality is two numbers. TTFT (time-to-first-token) is dominated by prefill: \(\text{TTFT} \approx t_{\text{queue}} + \frac{2\,P\,n_{\text{prompt}}}{\text{FLOPS}_{\text{eff}}}\) for a \(P\)-parameter model and \(n_{\text{prompt}}\)-token prompt (prefill is compute-bound). ITL (inter-token latency) is dominated by decode and is memory-bound:

\[ \text{ITL} \approx \frac{M_{\text{weights}} + M_{\text{KV}}(B,n)}{\text{BW}_{\text{HBM}}} \]

Note ITL grows with batch \(B\) through the cache term — bigger batches raise per-token latency even as they raise throughput. That coupling is the crux.

Throughput vs latency, formally (the roofline)

Per-GPU decode throughput in tokens/s is roughly \(\Lambda \approx B / \text{ITL}(B)\). Since ITL rises sub-linearly with \(B\) until the cache saturates HBM, \(\Lambda\) increases with \(B\) up to a knee, then flattens or falls when cache pressure forces evictions/preemptions. The serving problem is: choose the largest \(B\) whose ITL still meets the SLO.

\[ B^\star = \max\{\,B : \text{ITL}(B) \le \text{SLO}_{\text{ITL}} \;\wedge\; M_{\text{KV}}(B,n)\le M_{\text{HBM}}-M_{\text{weights}}\,\} \]

Little's Law sets the replica count

For a target request arrival rate \(\lambda\) (req/s) and mean in-system time \(W\), the average number of concurrently active requests is \(N = \lambda W\) (Little's Law). If one replica sustains concurrency \(c\) at the SLO, the required replicas are:

\[ R = \left\lceil \frac{\lambda W}{c} \right\rceil \]

This is exactly what a queue-depth-based autoscaler (KEDA on a custom metric) approximates online. Scaling on GPU utilization is wrong: a memory-bound decode pod can be 100% "busy" streaming the cache while compute sits idle, so utilization is a lying signal — scale on queue depth or pending tokens.

Prefill/decode disaggregation

Prefill (compute-bound, bursty) and decode (memory-bound, steady) have opposite hardware profiles, yet co-locating them lets a long prefill stall ongoing decodes (head-of-line blocking). Disaggregated serving (splitwise, 2024) runs prefill and decode on separate pod pools and ships the KV cache between them, letting each scale on its own bottleneck. The cost is a cache transfer over the network — a bandwidth term that must be budgeted.

Complexity, invariants, limiting cases

Cold start is \(O(M_{\text{weights}}/\text{BW}_{\text{load}})\) — loading 10s–100s of GB from storage into HBM takes seconds to minutes, so reactive autoscaling always lags a burst; over-provisioning headroom or fast model-streaming is mandatory. Invariant: a request's KV cache is pinned to one replica for its lifetime (decode is stateful) — you cannot mid-generation migrate it cheaply, which constrains routing and draining. Limiting cases: \(B=1\) minimizes latency and wastes the GPU; \(B\to\) cache-limit maximizes throughput and blows the SLO; \(\lambda\) spike faster than cold-start time \(\Rightarrow\) inevitable SLO violation unless pre-warmed.

Round 3 — Where It Breaks & Expert Debates

Autoscaling LLMs is not autoscaling web apps. The default Kubernetes HPA scales on CPU/memory — useless here. The correct signal (queue depth, pending tokens, or TTFT) needs KEDA or a custom metrics adapter, and even then the multi-minute cold start means a reactive scaler is always behind the burst. Predictive / scheduled scaling and warm pools are common, but they trade cost for safety, and where to sit on that curve is contested per-workload.

GPU sharing is unsolved-ish. A model that doesn't fill a GPU wastes it, but Kubernetes treats a GPU as an indivisible resource by default. MIG (hardware partitioning), MPS (process sharing), and time-slicing all exist, each with isolation/performance caveats. Fractional-GPU scheduling remains an active, messy area — no clean primitive as of 2026.

Tail latency vs utilization is fundamental, not a bug. The batch size that maximizes GPU utilization is the one that most inflates p99 ITL. Continuous batching narrows the gap but cannot erase it — you are always choosing a point on the throughput/latency frontier. Debates over "optimal" batch policy are really debates over which SLO percentile matters.

Multi-tenant fairness. One user's 100k-token request can monopolize a replica's cache and starve short requests behind it (head-of-line blocking). Fair scheduling across tenants inside a serving engine is immature; most systems fall back to separate replica pools per tier, wasting capacity.

Round 4 — AI × Networks Connection

This is the most direct AI→operations bridge in the KB. The non-RT RIC and CU run as cloud-native K8s workloads, so serving an inference model inside the RAN control plane is literally an LLM-serving-on-K8s problem with a network SLO instead of a chat SLO. The autoscaling, batching, and cold-start facts transfer verbatim — except the latency budgets are tighter and the "requests" are E2 indications, not user prompts.

The latency/throughput frontier maps onto the RIC latency gradient: near-RT inference xApps must run small-batch, latency-optimized replicas (accepting poor GPU utilization for a ~10 ms response), while non-RT model-training and policy-synthesis jobs batch aggressively for throughput. The cold-start tax is why you cannot elastically spin up a near-RT inference pod in response to a traffic surge — it must be pre-warmed, which turns capacity planning into the same queueing problem as slice admission. And the KV-cache memory wall is the term that decides how many concurrent inference streams a RIC GPU can host.

Cross-links

AI · KV cache mechanics → the memory term \(M_{\text{KV}}\) that drives batch limits, ITL, and replica capacity here.

Networks · O-RAN architecture → the RIC/CU are the K8s workloads; serving a model there is this problem under a network SLO.

Networks · Network slicing + SLA guarantees → multi-tenant serving fairness is structurally the same as slice isolation + SLA enforcement.

Pending intersection nodes this unblocks: Inference at the edge (constraints, architecture), LLM for network config generation, ML for RAN anomaly detection (as a served xApp).