What is lithos-metal?
lithos-metal is an open-source inference engine for running and serving language models on Apple silicon Macs. lithos-metal generates specialized Metal megakernels and supports local serving for coding agents and other applications. The engine combines layer-wise mixer fusion, GPU-resident generation control, and DSpark speculative decoding rather than relying on a single whole-model megakernel.
lithos-metal is a standalone engine licensed under Apache 2.0 and does not depend on the MPK/Mirage runtime. The repository includes implementation documentation, model-adapter contracts, chip-backend information, and contribution instructions. Code adapted from other projects retains its license headers and attribution in the repository.
What lithos-metal can do
lithos-metal combines model execution with a local serving interface. Selected configurations fuse attention or recurrent mixers within a layer, while feed-forward networks remain separate. DSpark proposes token blocks for target verification, and GPU-resident state manages token positions, accepted tokens, recurrent-state changes, and stopping with minimal CPU involvement.
- lithos-metal generates specialized Metal code from operator tasks and dependencies.
- lithos-metal provides Chat Completions, Responses, and Anthropic Messages adapters.
- lithos-metal exposes Ollama's /api/chat endpoint for Ollama clients.
- lithos-metal supports text generation, tool calls, and SSE streaming.
- lithos-metal automatically selects matching DSpark heads for listed target models.
- lithos-metal offers target-only generation through the --no-draft option.
How lithos-metal works
lithos-metal can be installed as a precompiled package from the Lithos Homebrew tap. Starting the serving command with a supported model downloads the target and matching LithosAI NVFP4 DSpark head, prepares cached weight packs, and selects device-specific kernel recipes. Startup also includes compilation and warmup.
lithos-metal provides launchers for installed opencode, claude, codex, and hermes clients, plus environment and process-launch commands for other OpenAI-compatible clients. The launchers discover the served model and configure the child process without rewriting global settings. Clients must be installed separately, and tool execution happens in the client.
Who lithos-metal is for
lithos-metal serves developers who want local language-model inference on supported Apple silicon hardware. Coding-agent users can connect installed clients to the local server, while application developers can use compatible API adapters. Contributors can consult extension contracts for adding models, quantization formats, operations, drafters, or chip backends.
- Developers can connect coding agents to locally served models.
- Application developers can consume streaming text and tool calls through compatible APIs.
- Inference developers can extend model adapters, operations, and hardware backends.
lithos-metal licensing and access
lithos-metal is available as an open-source project under the Apache 2.0 license, with precompiled Homebrew installation and a source-build path. Source builds require Xcode Command Line Tools and Python 3.12. The package build compiles the native runtime and bundles Metal sources and chip recipes; an editable development installation is also documented.
lithos-metal strengths and limitations
lithos-metal combines local serving, multiple protocol adapters, and hardware-specific execution recipes in a standalone engine. Compatibility still depends on the chip backend, model checkpoint, and available memory. The published performance figures are target-compute projections with excluded work, not measured end-to-end generation rates or direct end-to-end speedup comparisons.
- lithos-metal requires an Apple silicon Mac with macOS 26+ and sufficient unified memory.
- lithos-metal runs one generation at a time, and the Responses adapter is stateless.
- lithos-metal disables automatic megakernel fusion and the tensor accelerator on unmeasured backends.
- lithos-metal labels the hybrid MoE integration experimental and does not guarantee every checkpoint within a model family.
lithos-metal FAQ
What is lithos-metal?
lithos-metal is an open-source LLM inference engine that generates Metal megakernels for Apple silicon Macs. lithos-metal provides local serving for coding agents and other applications, combining layer-wise mixer fusion, GPU-resident generation control, and DSpark speculative decoding.
Is lithos-metal open source?
lithos-metal is fully open source and licensed under Apache 2.0. lithos-metal is available through a precompiled Homebrew package or a source build, and adapted code retains its license headers and attribution in the repository.
What hardware does lithos-metal require?
lithos-metal requires an Apple silicon Mac with macOS 26+, a registered chip backend, and enough unified memory for the target model, draft model, and context. Backend validation varies, and unmeasured backends have no automatic megakernel fusion and keep the tensor accelerator disabled.
Does lithos-metal work with coding agents and APIs?
lithos-metal provides launchers for separately installed opencode, claude, codex, and hermes clients, plus support for other OpenAI-compatible clients. lithos-metal also provides Chat Completions, Responses, Anthropic Messages, and Ollama chat adapters, with text generation, tool calls, and SSE streaming.
What are lithos-metal's serving limitations?
lithos-metal runs one generation at a time, provides a stateless Responses adapter, and leaves tool execution to the client. lithos-metal also requires separately installed coding clients, and matching a supported model family does not imply support for every checkpoint.
From the community
No reviews yet
Be the first to review this tool. What you write helps the next person choose.




