Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

llm-d

Open-source distributed LLM inference serving with intelligent routing and production operations on Kubernetes.

llm-d is a distributed inference serving stack that adds routing, cache management, and operational controls above model servers on Kubernetes. llm-d serves teams deploying production LLM workloads, including agent sessions and offline batches, and is available as an open-source project under Apache License 2.0.

Updated 9 Nov 20266 sections

At a glance

Pricing
FreeOpen-source project available through its public GitHub repository under Apache License 2.0.
Added
9 Nov 2026
llm-d website screenshot

What is llm-d?

llm-d is an open-source distributed inference serving stack optimized for production deployments on Kubernetes. llm-d provides orchestration and optimization above model servers such as vLLM and SGLang, which run large language models on accelerators. The project addresses high-scale inference traffic and the operational requirements of maintaining these services in production.

llm-d is a Cloud Native Computing Foundation sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The repository describes production validation across NVIDIA, AMD, and Intel GPUs and Google TPUs. Published adoption examples include Google Cloud, DigitalOcean, IBM Research, Mistral AI, and Red Hat.

What llm-d can do

llm-d organizes its capabilities around routing, cache management, large-model serving, agent workloads, production operations, and batch processing. These capabilities target efficient handling of real-world inference traffic rather than model execution alone. The repository also provides benchmarked recipes and architectural patterns for deploying and testing the stack.

  • llm-d routes requests using prefix-cache state and load awareness.
  • llm-d offloads KV-cache data to CPU or disk and globally indexes cache state.
  • llm-d supports prefill/decode disaggregation and wide expert-parallelism.
  • llm-d maintains agent context with session-aware routing and native Responses API state.
  • llm-d provides priority- and fairness-aware flow control for multiple tenants.
  • llm-d supports SLO-aware autoscaling and end-to-end observability.
  • llm-d processes offline workloads through OpenAI-compatible Batch APIs.

How llm-d works

llm-d integrates Kubernetes with model-serving technologies including vLLM and SGLang. Model servers handle execution on accelerators, while llm-d supplies orchestration and serving optimizations above them. Depending on the deployment recipe, these optimizations include intelligent routing, disaggregated serving, and cache management across inference infrastructure.

llm-d provides a Quickstart Guide covering stack setup, intelligent router configuration, and performance validation with benchmarks. The repository recommends the Optimized Baseline as the starting point for most users. Additional guides supply Helm charts, while accelerator documentation identifies tested accelerators, networks, and configurations.

Who llm-d is for

llm-d is intended for teams deploying and operating large language model inference on Kubernetes. The documented use cases span cloud providers, enterprises, and organizations self-hosting open models. Production examples include coding and chip-design agents at IBM Research, regional inference at Mistral AI, and distributed serving across AMD and NVIDIA hardware at DigitalOcean.

  • Infrastructure teams can use llm-d to operate multi-tenant inference services.
  • Agent developers can use llm-d to reuse context across conversation turns.
  • Model-serving teams can use llm-d for large-model disaggregation.
  • Batch-processing teams can use llm-d for asynchronous offline inference.

llm-d pricing and access

llm-d is available as an open-source project through its public GitHub repository under Apache License 2.0. The repository provides source code, deployment guides, Helm charts, and release information rather than a hosted-service subscription offer. The supplied GitHub pricing material describes GitHub plans, not llm-d pricing, and does not establish paid llm-d tiers.

llm-d strengths and limitations

llm-d combines inference optimizations with production controls such as router high availability, coordinated prefill/decode rollouts, tenant fairness, and observability. The repository supports its performance claims with workload-specific benchmarks and provides reproducible results through Prism. Deployment still involves configuring Kubernetes inference infrastructure, and the documentation distinguishes tested configurations from broader accelerator coverage.

  • llm-d requires Kubernetes deployment and technical configuration.
  • llm-d describes predicted latency-based scheduling as experimental.
  • llm-d benchmark results refer to specified models, hardware, and serving configurations.

llm-d FAQ

What is llm-d?

llm-d is an open-source distributed inference serving stack optimized for production deployments on Kubernetes. llm-d adds orchestration, intelligent routing, cache management, and operational controls above model servers such as vLLM and SGLang.

Is llm-d free?

llm-d is available as an open-source project under Apache License 2.0 through its public GitHub repository. The repository includes deployment guides and Helm charts, but does not describe a hosted llm-d subscription or paid product tiers.

Who is llm-d for?

llm-d is for teams deploying and operating production LLM inference on Kubernetes. Documented audiences include cloud providers, enterprises, and organizations self-hosting open models for workloads such as coding agents, chip-design agents, and regional inference services.

Does llm-d work with vLLM and SGLang?

llm-d works with vLLM and SGLang as underlying model servers. llm-d provides orchestration and serving optimizations above those servers, including routing, cache management, and production operations on Kubernetes.

Can llm-d serve agents and batch workloads?

llm-d supports agent workloads through session-aware routing, turn-priority scheduling, and native Responses API state that reuses context across turns. llm-d also supports large-scale offline inference through OpenAI-compatible Batch APIs and asynchronous processing.

How do you deploy llm-d?

llm-d provides a Quickstart Guide for deploying an inference service on Kubernetes, configuring the intelligent router, and validating performance. llm-d recommends the Optimized Baseline for most users and supplies benchmarked deployment recipes and Helm charts.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like llm-d

LMCache

www.github.com

Open-source KV cache management for reusable, persistent caching across LLM inference engines and storage backends.

Free

vLLM

www.github.com

Open-source LLM inference and serving with PagedAttention, distributed execution and an OpenAI-compatible API.

Free

KServe

www.github.com

Open-source generative and predictive AI model serving on Kubernetes

Free

AgentField

www.github.com

Open-source control plane for building, running, and scaling AI agents as REST APIs.

Free