Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons
All categories

LLM Infrastructure

Tools for serving, routing, evaluating and monitoring large language models

206tools listed

Pricing mix

  • 102 Free
  • 85 Freemium
  • 19 Paid

102 tools match

  • Vicuna

    www.lmsys.org

    A LLaMA-based research chatbot trained on user-shared conversations to handle multi-turn dialogue.

    Free
  • Local AI Playground

    www.localai.app

    Manage and verify AI models, run them offline on a CPU, and serve responses locally.

    Free
  • GGML

    www.ggml.ai

    A cross-platform tensor library for running large machine learning models on commodity hardware.

    Free
  • Gold Retriever

    www.jina.ai

    Create retrieval plugins that let ChatGPT search and use your own text data.

    Free
  • xTuring

    xturing.stochastic.ai

    Open-source LLM customization with LoRA fine-tuning and perplexity evaluation examples.

    Free
  • EnergeticAI

    www.energeticai.org

    Pre-trained English embeddings and text classifiers for Node.js, with a focus on serverless cold-start performance.

    Free
  • AI SDK

    sdk.vercel.ai

    An open-source TypeScript toolkit for building AI applications and agents with streaming and multi-provider support.

    Free
  • magpie

    www.usemagpie.ai

    A local gateway for choosing models, sharing subscriptions and masking secrets across coding agents.

    Free
  • BAML

    www.boundaryml.com

    An open-source programming language for AI agents with typed outputs, concurrency and built-in observability.

    Free
  • lithos-metal

    www.github.com

    Open-source LLM inference and local serving for Apple silicon Macs using Metal megakernels and DSpark decoding.

    Free
  • openTPU

    www.github.com

    Open-source AI accelerator with hardware design, a compiler, a simulator and profiling tools in one repository.

    Free
  • AxisRL

    www.github.com

    Open-source agentic reinforcement learning post-training with SGLang rollout and Megatron training.

    Free
  • LLM Graph Builder

    www.github.com

    Build Neo4j knowledge graphs from documents, videos and web pages, then query the data through chat.

    Free
  • KaaS

    www.github.com

    Compile notes, documents, and transcripts into a self-hosted Markdown wiki with cited answers and MCP access.

    Free
  • tokenizers

    www.github.com

    Open-source tokenization library for encoding and decoding text in research and production.

    Free
  • mistral.rs

    www.github.com

    Open-source LLM inference with multimodal models, local coding agents, compatible APIs, and Python and Rust SDKs.

    Free
  • AxonHub

    www.github.com

    Open-source AI gateway for model routing, SDK compatibility, request tracing and cost control.

    Free
  • tare

    www.github.com

    Audit Claude Code token usage, diagnose limits, and trace context overhead from local session logs.

    Free
  • agent-sandbox

    www.github.com

    Open-source Kubernetes orchestration for stateful sandboxes, AI agent runtimes, and reinforcement learning workloads.

    Free
  • gepa

    www.github.com

    Open-source optimization of prompts, code, agent architectures, and configurations using LLM reflection.

    Free
  • Swiftlet

    www.github.com

    Run Qwen Mixture-of-Experts models locally on Macs and iPhones by streaming expert weights from storage.

    Free
  • AI-Infra-Guard

    www.github.com

    Open-source AI red teaming with agent, skill, MCP, infrastructure and LLM security scanners.

    Free
  • Antigravity-Manager

    www.github.com

    Desktop account management, API protocol conversion, and request routing for AI services.

    Free
  • Automodel

    www.github.com

    Open-source PyTorch distributed training and fine-tuning for language and vision-language models

    Free
  • TokenHub

    www.github.com

    Open-source AI gateway for enterprise model routing, access control, usage attribution, and cost governance.

    Free
  • Repomix

    www.github.com

    Pack repositories into AI-friendly files with token counts, configurable exclusions, and optional code compression.

    Free
  • semantica

    www.github.com

    Open-source context graphs, knowledge graphs, deterministic reasoning, and decision provenance for AI systems.

    Free
  • Rivet

    www.github.com

    Open-source orchestration for durable AI agents, stateful applications, and replayable workflows.

    Free
  • ds4

    www.github.com

    Open-source local LLM inference for Metal, CUDA and ROCm, with a coding agent and HTTP server.

    Free
  • OptMem

    www.github.com

    Persistent memory for AI agents using a Python script, an append-only log and searchable summaries.

    Free
  • openai-python

    www.github.com

    Official Python library for the OpenAI API with synchronous, asynchronous, streaming, and Realtime support.

    Free
  • AgentField

    www.github.com

    Open-source control plane for building, running, and scaling AI agents as REST APIs.

    Free
  • SkillOpt

    www.github.com

    SkillOpt trains reusable natural-language skills for frozen LLM agents without changing model weights.

    Free
  • openmed

    www.github.com

    Local-first clinical entity extraction and PII de-identification across Python, Apple, Android and browser runtimes.

    Free
  • DSPy

    www.github.com

    An open-source Python framework for building language model systems and optimizing their prompts and weights.

    Free
  • A shared memory hub for AI agents that turns conversations, documents, and code into reusable team assets.

    Free