Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons
All categories

LLM Infrastructure

Tools for serving, routing, evaluating and monitoring large language models

206tools listed

Pricing mix

  • 102 Free
  • 85 Freemium
  • 19 Paid

206 tools

Filters

  • Hillock

    Local neuro-symbolic memory for offline AI, with verified fact retrieval and an OpenAI-compatible API.

    Code & Development
    Freemium
  • tare

    Audit Claude Code token usage, diagnose limits, and trace context overhead from local session logs.

    Code & Development
    Free
  • agent-sandbox

    Open-source Kubernetes orchestration for stateful sandboxes, AI agent runtimes, and reinforcement learning workloads.

    AI Agents & MCP
    Free
  • gepa

    Open-source optimization of prompts, code, agent architectures, and configurations using LLM reflection.

    Code & Development
    Free
  • Swiftlet

    Run Qwen Mixture-of-Experts models locally on Macs and iPhones by streaming expert weights from storage.

    Code & Development
    Free
  • AI-Infra-Guard

    Open-source AI red teaming with agent, skill, MCP, infrastructure and LLM security scanners.

    AI Agents & MCP
    Free
  • Antigravity-Manager

    Desktop account management, API protocol conversion, and request routing for AI services.

    Code & Development
    Free
  • Automodel

    Open-source PyTorch distributed training and fine-tuning for language and vision-language models

    Code & Development
    Free
  • TokenHub

    Open-source AI gateway for enterprise model routing, access control, usage attribution, and cost governance.

    LLM Infrastructure
    Free
  • Repomix

    Pack repositories into AI-friendly files with token counts, configurable exclusions, and optional code compression.

    Code & Development
    Free
  • semantica

    Open-source context graphs, knowledge graphs, deterministic reasoning, and decision provenance for AI systems.

    Data & Analytics
    Free
  • Rivet

    Open-source orchestration for durable AI agents, stateful applications, and replayable workflows.

    AI Agents & MCP
    Free
  • ds4

    Open-source local LLM inference for Metal, CUDA and ROCm, with a coding agent and HTTP server.

    Code & Development
    Free
  • OptMem

    Persistent memory for AI agents using a Python script, an append-only log and searchable summaries.

    AI Agents & MCP
    Free
  • openai-python

    Official Python library for the OpenAI API with synchronous, asynchronous, streaming, and Realtime support.

    Code & Development
    Free
  • AgentField

    Open-source control plane for building, running, and scaling AI agents as REST APIs.

    AI Agents & MCP
    Free
  • SkillOpt

    SkillOpt trains reusable natural-language skills for frozen LLM agents without changing model weights.

    AI Agents & MCP
    Free
  • openmed

    Local-first clinical entity extraction and PII de-identification across Python, Apple, Android and browser runtimes.

    Research & Analysis
    Free
  • DSPy

    An open-source Python framework for building language model systems and optimizing their prompts and weights.

    Code & Development
    Free
  • TencentDB-Agent-Memory

    A shared memory hub for AI agents that turns conversations, documents, and code into reusable team assets.

    AI Agents & MCP
    Free
  • guidance

    An open-source Python library for controlling language model output with grammars, regex and JSON schemas.

    Code & Development
    Free
  • SillyTavern

    A locally installed interface for LLM chat, image generation, and text-to-speech models.

    LLM Infrastructure
    Free
  • LlamaIndex

    Open-source framework for building RAG and agent applications with private data.

    Code & Development
    Free
  • Meilisearch

    Meilisearch provides full-text, semantic, and hybrid search through an API, with managed and self-hosted deployment.

    Code & Development
    Freemium
  • RAGFlow

    Open-source RAG engine for document understanding, agentic retrieval, and structured knowledge compilation.

    AI Agents & MCP
    Free
  • whichllm

    Find and run local LLMs ranked by hardware fit, estimated speed and benchmark quality.

    Code & Development
    Free
  • CubeSandbox

    Open-source sandbox service for AI agents with MicroVM isolation, E2B SDK compatibility, and snapshot management.

    AI Agents & MCP
    Free
  • Monty

    Rust-based Python sandbox for running AI-generated code with controlled access to host functions and mounts.

    Code & Development
    Freemium
  • llm-d

    Open-source distributed LLM inference serving with intelligent routing and production operations on Kubernetes.

    LLM Infrastructure
    Free
  • OpenShell

    An open-source runtime that isolates AI agents and enforces policies for files, networks, processes, and credentials.

    AI Agents & MCP
    Free
  • BitNet

    Open-source inference framework for running 1-bit language models on CPUs and GPUs.

    LLM Infrastructure
    Free
  • MemPalace

    Local-first AI memory with verbatim conversation storage, semantic search, CLI access and MCP integration.

    AI Agents & MCP
    Free
  • PaddleOCR

    Open-source OCR and document parsing that converts PDFs and images into structured Markdown and JSON for AI workflows.

    Productivity & Automation
    Free
  • claude-supermemory

    Persistent memory for Claude Code with automatic recall, conversation capture, and shared repository context.

    Code & Development
    Freemium
  • llmfit

    Find local LLMs that fit your CPU, RAM, GPU, and VRAM, with hardware detection and performance estimates.

    LLM Infrastructure
    Free
  • supermemory

    An open-source memory and context engine for AI agents, apps, and coding assistants.

    AI Agents & MCP
    Free