What is llm-d?
llm-d is an open-source distributed inference serving stack optimized for production deployments on Kubernetes. llm-d provides orchestration and optimization above model servers such as vLLM and SGLang, which run large language models on accelerators. The project addresses high-scale inference traffic and the operational requirements of maintaining these services in production.
llm-d is a Cloud Native Computing Foundation sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The repository describes production validation across NVIDIA, AMD, and Intel GPUs and Google TPUs. Published adoption examples include Google Cloud, DigitalOcean, IBM Research, Mistral AI, and Red Hat.
What llm-d can do
llm-d organizes its capabilities around routing, cache management, large-model serving, agent workloads, production operations, and batch processing. These capabilities target efficient handling of real-world inference traffic rather than model execution alone. The repository also provides benchmarked recipes and architectural patterns for deploying and testing the stack.
- llm-d routes requests using prefix-cache state and load awareness.
- llm-d offloads KV-cache data to CPU or disk and globally indexes cache state.
- llm-d supports prefill/decode disaggregation and wide expert-parallelism.
- llm-d maintains agent context with session-aware routing and native Responses API state.
- llm-d provides priority- and fairness-aware flow control for multiple tenants.
- llm-d supports SLO-aware autoscaling and end-to-end observability.
- llm-d processes offline workloads through OpenAI-compatible Batch APIs.
How llm-d works
llm-d integrates Kubernetes with model-serving technologies including vLLM and SGLang. Model servers handle execution on accelerators, while llm-d supplies orchestration and serving optimizations above them. Depending on the deployment recipe, these optimizations include intelligent routing, disaggregated serving, and cache management across inference infrastructure.
llm-d provides a Quickstart Guide covering stack setup, intelligent router configuration, and performance validation with benchmarks. The repository recommends the Optimized Baseline as the starting point for most users. Additional guides supply Helm charts, while accelerator documentation identifies tested accelerators, networks, and configurations.
Who llm-d is for
llm-d is intended for teams deploying and operating large language model inference on Kubernetes. The documented use cases span cloud providers, enterprises, and organizations self-hosting open models. Production examples include coding and chip-design agents at IBM Research, regional inference at Mistral AI, and distributed serving across AMD and NVIDIA hardware at DigitalOcean.
- Infrastructure teams can use llm-d to operate multi-tenant inference services.
- Agent developers can use llm-d to reuse context across conversation turns.
- Model-serving teams can use llm-d for large-model disaggregation.
- Batch-processing teams can use llm-d for asynchronous offline inference.
llm-d pricing and access
llm-d is available as an open-source project through its public GitHub repository under Apache License 2.0. The repository provides source code, deployment guides, Helm charts, and release information rather than a hosted-service subscription offer. The supplied GitHub pricing material describes GitHub plans, not llm-d pricing, and does not establish paid llm-d tiers.
llm-d strengths and limitations
llm-d combines inference optimizations with production controls such as router high availability, coordinated prefill/decode rollouts, tenant fairness, and observability. The repository supports its performance claims with workload-specific benchmarks and provides reproducible results through Prism. Deployment still involves configuring Kubernetes inference infrastructure, and the documentation distinguishes tested configurations from broader accelerator coverage.
- llm-d requires Kubernetes deployment and technical configuration.
- llm-d describes predicted latency-based scheduling as experimental.
- llm-d benchmark results refer to specified models, hardware, and serving configurations.
llm-d FAQ
What is llm-d?
llm-d is an open-source distributed inference serving stack optimized for production deployments on Kubernetes. llm-d adds orchestration, intelligent routing, cache management, and operational controls above model servers such as vLLM and SGLang.
Is llm-d free?
llm-d is available as an open-source project under Apache License 2.0 through its public GitHub repository. The repository includes deployment guides and Helm charts, but does not describe a hosted llm-d subscription or paid product tiers.
Who is llm-d for?
llm-d is for teams deploying and operating production LLM inference on Kubernetes. Documented audiences include cloud providers, enterprises, and organizations self-hosting open models for workloads such as coding agents, chip-design agents, and regional inference services.
Does llm-d work with vLLM and SGLang?
llm-d works with vLLM and SGLang as underlying model servers. llm-d provides orchestration and serving optimizations above those servers, including routing, cache management, and production operations on Kubernetes.
Can llm-d serve agents and batch workloads?
llm-d supports agent workloads through session-aware routing, turn-priority scheduling, and native Responses API state that reuses context across turns. llm-d also supports large-scale offline inference through OpenAI-compatible Batch APIs and asynchronous processing.
How do you deploy llm-d?
llm-d provides a Quickstart Guide for deploying an inference service on Kubernetes, configuring the intelligent router, and validating performance. llm-d recommends the Optimized Baseline for most users and supplies benchmarked deployment recipes and Helm charts.
From the community
No reviews yet
Be the first to review this tool. What you write helps the next person choose.




