Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons

LMCache

Open-source KV cache management for reusable, persistent caching across LLM inference engines and storage backends.

LMCache is a KV cache management layer for LLM inference that persistently stores, shares, and reuses cached context to reduce repeated computation. LMCache serves developers, infrastructure teams, and researchers working on agentic, conversational, and RAG workloads, and is available as an open-source project under Apache License 2.0.

Updated 9 Nov 20266 sections

LMCache website screenshot

What is LMCache?

LMCache is an open-source KV cache management layer for scalable LLM inference. LMCache turns cache data from temporary inference state into reusable information that can be stored persistently and shared across serving engines. The project serves developers, researchers, and infrastructure teams working on language model serving, particularly long-context agents, multi-turn conversations, and knowledge-augmented workloads such as RAG.

LMCache separates cache management from the inference engine and provides interfaces for storage, transport, monitoring, and transformation. The repository describes LMCache as vendor-neutral, allowing users to change serving engines or storage vendors while retaining reusable KV caches. The codebase is licensed under Apache License 2.0.

What LMCache can do

LMCache focuses on reducing repeated prefill computation by moving KV caches beyond GPU memory and making those caches reusable. LMCache also provides observability and extension interfaces for production operations and research. The documented capabilities cover persistent storage, cache reuse beyond prompt prefixes, transfers between inference workers, and custom transformations of cached data.

  • LMCache offloads KV caches to CPU memory, local disk, and remote storage backends.
  • LMCache reuses cached data across requests, sessions, and engine instances.
  • LMCache reports health, performance, prefix-hit, cache lifecycle, and user-specific usage metrics.
  • LMCache uses CacheBlend for non-prefix reuse with selective token recomputation.
  • LMCache transfers KV caches between prefill and decode workers over NVLink, RDMA, or TCP.
  • LMCache exposes a SERDE interface for compression, token dropping, and custom serialization.

How LMCache works

LMCache runs as a standalone daemon process that manages KV caches independently from the inference engine process. This separation means cached state is not lost solely because the inference engine crashes. LMCache moves caches into a tiered storage hierarchy and retrieves reusable data across requests, sessions, and engine instances instead of repeating the same prefill computation.

LMCache uses unified interfaces for pluggable storage and KV transfer backends. The repository lists CPU RAM, local SSD storage, Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS among supported backends. LMCache also documents multimodal model support in vLLM and an integration with NVIDIA Dynamo.

Who LMCache is for

LMCache is intended for developers, researchers, and industry teams building efficient LLM inference systems. LMCache is particularly relevant when workloads repeatedly use context, when caches need to survive engine restarts, or when infrastructure needs to move cached data between storage tiers and serving components. Researchers can also use the transformation interface to explore cache compression and serialization.

  • LMCache supports cache reuse for long-context agentic workloads.
  • LMCache reduces repeated prefill work in multi-turn conversation workloads.
  • LMCache supports knowledge-augmented workloads such as RAG.
  • LMCache enables KV cache transfers between separate prefill and decode workers.

LMCache licensing and access

LMCache is available as an independent open-source project, with its codebase licensed under Apache License 2.0. LMCache can be obtained through the public repository or installed using pip install lmcache. The repository provides links to an installation guide, quickstart, recipes, CLI reference, benchmarking guide, production deployment documentation, and FAQ.

LMCache strengths and setup considerations

LMCache combines persistent cache reuse, engine-independent operation, and a choice of storage and transport backends. These capabilities support cache management across different serving environments without tying stored caches to an individual engine process. LMCache requires installation and setup within an inference environment; the repository directs users to setup examples and production deployment guidance rather than presenting a standalone consumer application.

LMCache FAQ

What is LMCache?

LMCache is an open-source KV cache management layer for LLM inference. LMCache persistently stores and reuses caches across requests, sessions, and serving engines to reduce repeated prefill computation, lower time-to-first-token, and improve throughput.

Is LMCache open source?

LMCache is an independent open-source project licensed under Apache License 2.0. LMCache is available through its public repository and can be installed using pip install lmcache, with installation and deployment documentation linked from the repository.

Which storage backends does LMCache support?

LMCache supports CPU RAM, local disk storage, Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS. LMCache provides a unified interface for integrating remote storage and KV transfer backends for cache offloading and sharing.

Who is LMCache for?

LMCache serves developers, researchers, and teams building LLM inference systems. LMCache targets long-context agentic workloads, multi-turn conversations, and knowledge-augmented workloads such as RAG, where cache reuse can reduce repeated computation.

Does LMCache support non-prefix cache reuse?

LMCache supports reusing cached KV blocks at any position in a prompt, rather than only matching prefixes. LMCache uses CacheBlend to selectively recompute tokens for quality recovery when applying this non-prefix reuse.

How do you install LMCache?

LMCache can be installed with pip install lmcache. LMCache also provides links to installation instructions, a quickstart, recipes, a CLI reference, a benchmarking guide, and production deployment documentation for further setup.

From the community

No reviews yet

Be the first to review this tool. What you write helps the next person choose.

More tools like LMCache

llm-d

www.github.com

Open-source distributed LLM inference serving with intelligent routing and production operations on Kubernetes.

Free

vLLM

www.github.com

Open-source LLM inference and serving with PagedAttention, distributed execution and an OpenAI-compatible API.

Free

SGLang

www.github.com

Open-source inference and serving framework for large language, vision-language, and diffusion models.

Free

RunNburn

www.github.com

Memory-aware GGUF inference with CPU and GPU offloading and an OpenAI-compatible server.

Free