For a prototype, serving an open model is surprisingly straightforward. You can bring up a model server, attach a GPU, put an API in front of it, and get a useful result in an afternoon. The harder problem starts later. Requests begin arriving from several applications. Different models need different amounts of memory. Some prompts are long and repetitive. Some workloads are latency-sensitive while others can run asynchronously. A GPU that looked comfortably sized for one service is suddenly being shared by embeddings, rerankers, batch jobs, agents, or development environments.

At that point, the question is no longer simply:

How do I run an LLM?

It becomes:

How do I operate inference and accelerator capacity as a platform?

That is the territory where llm-d and HAMi become useful. They do not solve the same problem, and understanding that distinction makes the rest of the Kubernetes AI stack much easier to reason about.

Problems with GPU cluster

A conventional Kubernetes setup already gives you most of the basic building blocks. You have Pods, Services, scheduling, device plugins, autoscaling, networking and observability. NVIDIA and other accelerator ecosystems can expose GPUs to Kubernetes as schedulable resources. The difficulty is that a GPU-heavy inference platform has two very different control loops.

The first is the inference control loop:

• Which model server should receive this request?
• Which replica currently has useful KV-cache state?
• Which endpoint has the right model variant?
• Is this request better handled by a prefill worker or a decode worker?
• How should the platform react when queue depth and latency change?

The second is the accelerator control loop:

• Which workload should run on which GPU?
• Can several small workloads share the same physical device?
• How much GPU memory and compute should each workload consume?
• How do you prevent one workload from consuming capacity reserved for another?
• How do you schedule heterogeneous accelerators without forcing every workload to understand the hardware?

llmd and HAMi can help us fix these problems with Inference routing and resource distribution, let's see them in more detail

What is llmd?

llm-d is a Kubernetes-native distributed inference stack designed to operate model serving across a cluster rather than treating every vLLM or SGLang instance as an isolated endpoint. The project originated from a collaboration between Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. llm-d was accepted into the CNCF Sandbox in March 2026. Its current architecture builds around a Kubernetes-native router, InferencePool abstractions, and model servers such as vLLM and SGLang. The simplest mental model is:

vLLM or SGLang executes the model. llm-d decides how requests should be distributed around those model servers.

That means llm-d is not a replacement for an inference engine. It is an infrastructure layer around one. Its current architecture includes capabilities for:

• inference-aware routing
• KV-cache-aware routing
• request queue and load signals
• model and endpoint selection
• prefill/decode disaggregation
• hierarchical KV-cache offloading
• predicted-latency routing
• batch inference
• Kubernetes-native autoscaling integrations
• hardware-aware deployment patterns

Start with the ordinary Kubernetes pattern

Imagine a cluster with three vLLM replicas behind a regular load balancer:

This architecture is completely reasonable for many workloads. A conventional load balancer can determine whether an endpoint is alive, route traffic, and apply ordinary networking policies. What it generally does not know is the inference state inside the model server. Two replicas can both report as healthy while being very different from an inference perspective. One may have a deep queue. Another may have useful prefix-cache state for the incoming request. A third may have lower current load but no reusable cache for that prompt. For ordinary web traffic, that distinction might not matter much. For LLM inference, it can matter a lot. This is the core reason inference-aware routing exists.

The llm-d architecture

The current llm-d architecture is centered on three concepts:

1. Router
2. InferencePool
3. Model Server

The Router itself is split into a proxy and an Endpoint Picker (EPP). The proxy accepts the request, while the EPP evaluates candidate endpoints and selects where inference should happen.

A simplified view looks like this:

The router can make decisions using signals such as model compatibility, current load and KV-cache locality. The EPP uses pluggable filters and scorers, which makes the scheduling logic extensible rather than hard-coding every policy into one monolithic component.

That distinction becomes much more useful once you have more than a few replicas.

Why KV cache changes request routing

LLM inference is stateful in a way that ordinary HTTP load balancing is not. During the prefill stage, the model processes the input context and produces the internal attention state that is held in the KV cache. When future requests reuse a prefix, some of that work can be avoided. vLLM's Automatic Prefix Caching, for example, can reuse cached KV state when a new request shares a prefix with an earlier request. That is useful for repeated system prompts, long documents, multi-turn conversations, coding workloads and other patterns with substantial shared context. A conventional load balancer does not know which backend owns the useful cache. An inference-aware router can take cache locality into account. The decision therefore starts looking more like:

Incoming request
      ↓
What model does it require?
      ↓
Which endpoints can serve it?
      ↓
Where is useful KV-cache state?
      ↓
What is the current queue/load?
      ↓
What does the routing policy prefer?
      ↓
Select endpoint

That is a qualitatively different problem from round-robin routing.

There is an important limitation here: cache-aware routing is useful only when cache reuse actually exists. If every request is unique, or the output generation dominates the latency profile, the benefit can be smaller. That is why this should be measured against the workload rather than treated as a magic optimization.

Prefill and decode are different workloads

A second reason distributed inference becomes interesting is the difference between prefill and decode.

Prefill processes the input context. It is generally more compute-intensive and has a direct effect on time to first token (TTFT). Decode generates output tokens one by one. It is typically much more sensitive to memory bandwidth and KV-cache behavior, and it directly affects the rate at which output tokens are delivered and metrics such as inter-token latency. Putting both phases on the same workers is perfectly valid. In many deployments, it is also the simplest and most practical approach. But under heavy traffic, long-context requests can make the two phases interfere with each other.

This is where prefill/decode disaggregation comes in.

Instead of treating the entire inference request as one undifferentiated operation, the system can place the stages on specialized workers:

If prefill is compute-bound while decode is memory-bandwidth-bound, the two stages can have different resource profiles. Separating them allows the platform to tune capacity and placement differently for each phase. llm-d's current router architecture supports disaggregated inference patterns and can select separate prefill and decode endpoints. The project documentation also describes more experimental encode/prefill/decode topologies for multimodal workloads.

The cache layer at scale

Once the platform starts caring about KV-cache locality, cache management itself becomes an infrastructure problem.

llm-d's current architecture includes mechanisms for KV-cache management and hierarchical offloading. The project documentation describes CPU and SSD cache tiers as well as peer-to-peer prefix-cache sharing patterns. That does not mean the architecture suddenly turns every byte of storage into equivalent GPU memory. It means the serving system can treat cache as a resource hierarchy rather than assuming all useful state must remain on the accelerator that created it.

A simplified mental model is:

GPU KV cache
     ↓
CPU cache tier
     ↓
SSD / lower-cost storage tier

The trade-offs are obvious: moving data between tiers costs time and bandwidth. The point is not to make storage behave like VRAM. The point is to avoid recomputing expensive work when the latency and transfer cost of retrieving cached state are still favorable. This becomes particularly relevant in workloads with repeated long prefixes.

llm-d is becoming more than a router

The project has grown beyond the original question of endpoint selection.

Its current architecture documentation includes:

• KV-cache management and offloading
• disaggregated inference
• predicted-latency routing
• batch inference components
• HPA/KEDA integrations
• workload-aware autoscaling
• hardware-aware deployment
• tested deployment recipes called Well-Lit Paths

llm-d's Well-Lit Paths are designed around tested inference patterns, deployment manifests, configuration guidance, benchmarking and observability.

GPU allocation using HAMi

llm-d helps answer:

Where should this inference request go?

HAMi starts from a different question:

How much accelerator capacity should this workload get?

HAMi stands for Heterogeneous AI Computing Virtualization Middleware. It is a Kubernetes-focused platform for sharing and scheduling heterogeneous AI accelerators. HAMi became a CNCF Sandbox project in 2024 and moved to CNCF Incubating in July 2026. Its core idea is straightforward. Kubernetes can expose a GPU as a schedulable device using device plugins and Operators like NVIDIA GPU Operator. That is useful when a workload really needs the whole GPU.

But consider a 24 GB GPU with three smaller workloads:

Physical GPU: 24 GB
 
Workload A: 4 GB
Workload B: 6 GB
Workload C: 5 GB
 
Potential unused capacity: 9 GB

The problem is not that the GPU is too small. The problem is granularity. If the platform allocates only whole devices, the hardware can remain underused even when several workloads could coexist. HAMi is designed to provide a finer-grained resource model for this class of workload.

What is HAMi?

It is easy to describe HAMi as "GPU sharing" and stop there. That misses the architecture. The project combines several pieces, including:

• Kubernetes admission logic
• scheduling
• device allocation
• in-container runtime controls
• accelerator-specific resource management

A simplified flow looks like this:

The exact control flow depends on the deployment mode and project version, but the separation of concerns is the important part.

Scheduling answers:

Which device should this workload use?

Runtime enforcement answers:

What is this workload allowed to consume after it starts?

Those are different problems.

HAMi-Core is the part enforcing the limits

HAMi-Core sits inside the container execution path and provides the runtime control layer for supported accelerators. For NVIDIA CUDA workloads, its documented design places it between the CUDA Runtime and the NVIDIA CUDA Driver by intercepting relevant API calls.

The simplified picture is:

This is why HAMi is more than a scheduler. A scheduler can decide that three workloads belong on one device. It does not, by itself, guarantee that each process stays within the intended memory or compute envelope. HAMi-Core is one of the mechanisms used to enforce those runtime limits for supported workloads. This is also where compatibility testing becomes important. Runtime interception, driver versions, accelerator models, CUDA behavior and framework behavior all matter. A design that looks good on a diagram still needs to be validated against the actual GPU and software stack you intend to operate.

Before and after GPU sharing

A simple whole-device allocation pattern might look like this:

If the inference service needs most of GPU 1 and the embedding service needs most of GPU 2, this is fine. The problem appears when the workloads are small relative to the device. With a sharing layer, the same physical GPU can be represented as a shared resource:

The value here is better utilization of a scarce accelerator. It is not "creating more GPU." You still have one physical device with one physical memory system, one set of compute resources and the performance characteristics of that hardware. Sharing changes how that capacity is allocated and controlled.

GPU sharing is not one technology

"GPU virtualization" is not one thing. The broader Kubernetes and accelerator ecosystem includes mechanisms such as:

• NVIDIA MIG
• GPU time slicing
• vGPU
• MPS
• software-level sharing layers such as HAMi
• Kubernetes Dynamic Resource Allocation
• vendor-specific device plugins
• scheduler-level policies

These approaches differ in hardware requirements, isolation model, scheduling semantics, observability, compatibility and performance characteristics.

So the right question is not:

Which technology is the best GPU virtualization technology?

The useful question is:

What isolation and allocation model does this workload actually need?

For example, MIG can provide hardware-level partitioning on supported NVIDIA GPUs. A software sharing layer can provide a different trade-off. DRA provides a native Kubernetes API model for dynamic device allocation and, in newer Kubernetes releases, more expressive capacity semantics.

The architecture should follow the required guarantees.

Where Kubernetes DRA changes the conversation

Kubernetes itself has been moving toward a more expressive device allocation model through Dynamic Resource Allocation (DRA). DRA is stable in Kubernetes v1.35 according to the current Kubernetes documentation. It provides a resource-claim model that lets workloads request devices through Kubernetes APIs instead of relying only on traditional whole-device resource requests. That matters because the original GPU resource model is intentionally simple:

resources:
  limits:
    nvidia.com/gpu: 1

It effectively says:

Give this container one GPU device.

DRA allows the resource model to express more complex device requirements and configurations. Kubernetes has also been adding support for consumable capacity, allowing compatible devices to be shared by multiple independent claims while the scheduler tracks the aggregate consumed capacity. So where does HAMi fit? This is not simply a story of "DRA replaces HAMi." The projects are moving at different layers of the stack and evolving quickly. HAMi has been building toward DRA integration while retaining its own virtualization and runtime-control components. The practical takeaway is that platform teams should understand both the Kubernetes-native direction and the capabilities required from the accelerator layer. Do not evaluate a GPU sharing platform only by asking whether it can create fractional resources.

Ask how allocation, isolation, enforcement, scheduling, observability and lifecycle management are implemented.

llm-d and HAMi solve different layers

This is the cleanest way to connect them:

The relationship can be summarized as:

llm-d manages the inference-serving problem. HAMi manages accelerator allocation and sharing.

They can therefore operate underneath the same Kubernetes platform without being redundant. There is overlap in the broader scheduling discussion, but the primary objectives are different.

A useful way to think about the stack

Start with the smallest possible architecture.

For one model, one GPU and low traffic:

That can be enough. There is no prize for building the most sophisticated platform before you have a workload that needs it. Now suppose traffic grows and you have several replicas:

The platform now has a reason to care about inference-aware routing, cache locality, endpoint state and autoscaling. Then suppose those model servers are sharing a cluster with embeddings and rerankers:

Now GPU allocation becomes its own problem. Put the two layers together and the architecture starts to resemble a private inference platform:

The key idea is separation of responsibilities.

Inference efficiency       → llm-d
Accelerator allocation     → HAMi / DRA / vendor mechanisms
Model execution            → vLLM / SGLang / other runtimes
Cluster scheduling         → Kubernetes
Traffic entry              → Gateway / inference gateway
Observability               → Prometheus / Grafana / OpenTelemetry / vendor telemetry

That composability is one of the more important trends in Kubernetes AI infrastructure.

The modern AI infrastructure stack is becoming layered

There is a temptation to search for one project that "does AI infrastructure."

In practice, production systems are becoming a collection of specialized layers.

None of these layers is automatically required.

That is exactly the point. A healthy platform starts from workload requirements and adds complexity only where it buys something measurable.

What is changing in the ecosystem

The timing is interesting because both projects have crossed meaningful governance milestones. llm-d entered the CNCF Sandbox in March 2026. Its current public architecture explicitly targets distributed inference on Kubernetes, with support across several accelerator ecosystems and an emphasis on inference-aware scheduling, cache locality and production deployment patterns. HAMi moved to CNCF Incubating in July 2026. Its project direction is focused on heterogeneous accelerator virtualization, resource isolation, scheduling and increasing Kubernetes-native integration, including DRA.

At the same time, Kubernetes itself is expanding its resource-management APIs through DRA. That creates a bigger shift than any one project. The Kubernetes ecosystem is moving from:

Schedule a whole accelerator to a Pod.

toward:

Describe the accelerator resource a workload actually needs, and let the platform reason about it.

For inference, another parallel shift is happening:

Send requests to a healthy replica.

is becoming:

Route requests using inference state, cache locality, workload characteristics and latency objectives.

Those are two different evolutions happening at the same time.

llm-d's current direction matters

It is also worth looking at what llm-d is trying not to become. The project is not implementing a replacement model runtime that forks vLLM or SGLang. Instead, it builds around existing inference engines and Kubernetes-native interfaces. That has a practical advantage. Model runtimes evolve quickly. Their kernels, parallelism strategies, memory management and model support can change rapidly. An orchestration layer that can sit above them without owning the full execution engine has a better chance of remaining composable. The current llm-d router architecture uses the Kubernetes Gateway API Inference Extension model and separates routing policy into filters, scorers and a data layer. That makes the router itself extensible.

For platform engineers, that is more interesting than a single "smart load balancer" feature because it creates a place for scheduling policy to evolve independently. But do not confuse architecture with operational maturity This is where AI infrastructure discussions often become too optimistic. A component can have an excellent architecture and still be the wrong dependency for your environment.

Before adopting llm-d or HAMi, validate:

• version maturity
• accelerator compatibility
• driver compatibility
• Kubernetes version
• observability
• failure behavior
• upgrade path
• security model
• debugging workflow
• performance under your actual workload
• operational ownership inside your team

A platform is not production-ready because the components installed successfully. If the only person who knows why a router is selecting a particular endpoint is the engineer who wrote the original Helm values, the platform has an operational problem. The same is true for GPU virtualization. If a workload fails only under contention, or behaves differently after a driver update, or becomes unstable under memory pressure, your team needs enough visibility to distinguish an application problem from a runtime-control problem. That is why observability belongs in the architecture from day one.

A basic deployment is often enough when

You have:

• one or two models
• very low traffic
• dedicated GPUs
• no meaningful cache-locality problem
• simple operational requirements
• no requirement to pack many independent workloads onto the same accelerator

A simple Kubernetes deployment with vLLM or SGLang can be entirely reasonable in this environment.

llm-d becomes more relevant when

You have:

• multiple inference replicas
• more than one model or model variant
• meaningful queueing and latency pressure
• long or repeated prefixes
• cache locality that affects performance
• interest in prefill/decode disaggregation
• more complex autoscaling requirements
• a need for reusable production deployment patterns

The important word is relevant, not mandatory.

A single-model cluster should not be forced into a distributed inference architecture just because the project exists.

HAMi becomes more relevant when

You have:

• multiple smaller AI workloads
• GPUs that are often partially idle
• several teams sharing accelerator capacity
• requirements for GPU memory or compute limits
• heterogeneous accelerators
• pressure to improve utilization without purchasing more devices

Again, the actual utilization profile matters more than the marketing description.

If every workload already consumes an entire GPU, introducing fractional sharing may add complexity without solving the underlying problem.

What about air-gapped and private AI?

Private and air-gapped environments make these decisions more interesting because the platform often has to own more of the stack. In a public cloud API model, the infrastructure boundary is relatively small:

Application
    ↓
External model API

With private inference, the boundary expands:

Application
    ↓
Internal gateway
    ↓
Authentication / authorization
    ↓
Inference routing
    ↓
Model servers
    ↓
Kubernetes
    ↓
Accelerator management
    ↓
GPU / NPU hardware
    ↓
Observability + security + lifecycle

That means the platform team owns latency, capacity planning, failures, patching, upgrades, model rollout and hardware utilization. For an air-gapped environment, the operational boundary becomes even larger.

You may need:

• private image registries
• mirrored package repositories
• offline model artifacts
• internal certificate management
• secrets management
• strict egress controls
• image and dependency scanning
• signed artifacts
• controlled upgrade workflows
• hardware telemetry
• backup and disaster recovery
• audit logging

In that context, llm-d and HAMi are useful as infrastructure components, but neither one removes the need for the surrounding platform engineering.

A realistic private AI architecture

A private AI platform can evolve toward something like this:

This is much closer to what an enterprise platform looks like in practice. The exact components will vary. The important thing is that the model-serving layer sits inside a larger operating system for AI workloads.

Where people usually overbuild

There is a recurring failure mode in infrastructure projects: A team sees several new AI infrastructure projects and installs all of them.

The final cluster looks like this:

Kubernetes
+ multiple operators
+ multiple schedulers
+ multiple gateways
+ service mesh
+ GPU plugins
+ several observability agents
+ a custom controller
+ a large pile of Helm values

The architecture diagram looks impressive. The operational model is miserable. Every additional control plane adds upgrade work, failure modes, security review and debugging complexity. A better approach is boring on purpose.

1. Start with the workload.
2. Measure it.
3. Find the bottleneck.
4. Add one layer.
5. Measure again.

Then keep the layer if it improves the system enough to justify owning it. That is a much better platform engineering loop than collecting infrastructure components because they are popular.

The bigger picture: inference is now a scheduling problem

The interesting shift in private AI infrastructure is not simply that organizations can run open models themselves. The bigger shift is that inference becomes a distributed scheduling problem.

At small scale, the path looks like:

request
  ↓
model server
  ↓
GPU

At larger scale, it becomes:

request
  ↓
gateway
  ↓
model selection
  ↓
inference routing
  ↓
cache locality
  ↓
queue state
  ↓
prefill / decode decisions
  ↓
model server
  ↓
GPU scheduling
  ↓
accelerator capacity
  ↓
hardware

Every stage influences the next one.

The simplest way to remember llm-d and HAMi

Think of two questions.

llm-d asks:

Where should this inference request run, and what should the inference system do with it?

HAMi asks:

Where should this workload run, and how much accelerator capacity should it be allowed to consume?

The first is primarily about inference orchestration. The second is primarily about accelerator resource management. Underneath both of them, Kubernetes provides the control plane. And below that, the hardware still wins every argument.

Building private AI infrastructure?

For teams moving from external model APIs to private, hybrid or air-gapped inference, the hard part is usually not getting the first model online. It is designing the layers around it so that routing, GPU capacity, security, observability and upgrades remain manageable as usage grows.

At Covaratech, we work across Kubernetes, AI infrastructure, GPU platforms, inference stacks, observability and private or air-gapped environments. If you are evaluating llm-d, HAMi, vLLM, SGLang, GPU sharing or a complete 0→1 private AI platform, the architecture should start with your workload and constraints before you start buying GPUs or accumulating YAML.

References

• llm-d GitHub
• llm-d Architecture
• llm-d Router Architecture
• llm-d Disaggregated Serving
• llm-d Well-Lit Paths
• CNCF: llm-d
• HAMi Documentation
• HAMi Architecture
• CNCF: HAMi
• CNCF: Does Kubernetes DRA Replace HAMi?
• Kubernetes: Dynamic Resource Allocation
• Kubernetes: DRA Features
• Kubernetes: Schedule GPUs
• vLLM: Automatic Prefix Caching