Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons
All categories

LLM Infrastructure

Tools for serving, routing, evaluating and monitoring large language models

206tools listed

Pricing mix

  • 102 Free
  • 85 Freemium
  • 19 Paid

121 tools match

  • BIG-bench

    www.github.com

    Open-source benchmark for evaluating language models with JSON and programmatic tasks

    Free
  • FalkorDB

    www.github.com

    Graph database for knowledge graphs, GraphRAG, and agent memory, with OpenCypher queries and Redis client access.

    Free
  • ToolPermit

    www.github.com

    Local-first permission policies, one-time approvals, and redacted audit logs for MCP tool calls.

    Free
  • nobodywho

    www.github.com

    Open-source inference engine for local LLMs, tool calling, speech recognition and speech synthesis

    Free
  • Open-source model optimization library for quantization, pruning, distillation and faster inference.

    Free
  • CLIProxyAPI

    www.github.com

    Open-source proxy server for accessing CLI models through OpenAI, Gemini, Claude, Codex, and Grok-compatible APIs.

    Free
  • Magnitude

    www.github.com

    Open-source inference engine that tunes local models for your hardware and connects them to coding agents.

    Free
  • reef

    www.github.com

    Open-source infrastructure for improving agent model weights, prompts, rules, and skills from feedback.

    Free
  • Miles

    www.github.com

    Open-source reinforcement learning framework for large-scale LLM and VLM post-training.

    Free
  • datasets

    www.github.com

    Open-source Python library for loading, streaming, and preprocessing machine learning datasets.

    Free
  • SGLang

    www.github.com

    Open-source inference and serving framework for large language, vision-language, and diffusion models.

    Free
  • TurboFieldfare

    www.github.com

    Run Gemma 4 26B-A4B locally on Apple Silicon Macs using about 2 GB of RAM.

    Free
  • Switchyard

    www.github.com

    Open-source model routing for AI agents, with Rust embedding and gateway integrations.

    Free
  • RunNburn

    www.github.com

    Memory-aware GGUF inference with CPU and GPU offloading and an OpenAI-compatible server.

    Free
  • kimi-k3-in-c

    www.github.com

    Open-source C99 inference engine for running Kimi K3 on a CPU without BLAS, a framework or a GPU.

    Free
  • Noisegate

    www.github.com

    An open-source differential privacy gateway for AI agents querying sensitive data over MCP.

    Free
  • AgentENV

    www.github.com

    Open-source infrastructure for running, snapshotting and forking agent environments across machines.

    Free
  • Candle

    www.github.com

    Open-source Rust machine learning framework for model training, GPU inference, and browser-based models.

    Free
  • DataFlow

    www.github.com

    Open-source LLM data preparation with reusable operators, visual pipelines, and data generation and cleaning workflows.

    Free
  • openai-oauth

    www.github.com

    Connect ChatGPT accounts to apps through an OpenAI-compatible proxy, TypeScript SDK and browser sign-in.

    Free
  • PageIndex

    www.github.com

    Open-source, reasoning-based document retrieval using hierarchical tree indexes instead of vector databases.

    Free
  • plano

    www.github.com

    Open-source AI proxy for agent orchestration, LLM routing, guardrails, and OpenTelemetry observability.

    Free
  • Heretic

    www.github.com

    Open-source language model modification for refusal suppression, shorter responses, and reduced sycophancy.

    Free
  • slime

    www.github.com

    Open-source LLM reinforcement learning framework connecting Megatron training with SGLang rollout.

    Free
  • marin

    www.github.com

    Open-source framework for foundation model training, data preparation, evaluation, and reproducible experiments.

    Free
  • LMCache

    www.github.com

    Open-source KV cache management for reusable, persistent caching across LLM inference engines and storage backends.

    Free
  • CocoIndex

    www.github.com

    Open-source incremental data engine that keeps context fresh for AI agents and LLM applications.

    Free
  • vLLM

    www.github.com

    Open-source LLM inference and serving with PagedAttention, distributed execution and an OpenAI-compatible API.

    Free
  • Rig

    www.github.com

    Rust library and plugin-based terminal coding agent for building LLM applications.

    Free
  • NeMo

    www.github.com

    Open-source framework for creating, customizing, and deploying speech recognition, text-to-speech, and speech LLMs.

    Free
  • context-mode

    www.github.com

    A free local plugin that sandboxes coding-agent tool output and retrieves relevant results through SQLite search.

    Free
  • Datasaur

    www.datasaur.ai

    Private enterprise AI agents and custom models deployed inside your own infrastructure.

    Paid
  • Klu.ai

    www.klu.ai

    Evaluate AI models on real workflows and track when new releases improve performance or cost.

    Paid
  • LiteLLM

    www.github.com

    Open-source AI gateway and Python SDK for calling LLM providers through a unified OpenAI-compatible interface.

    Free
  • Get Odin AI

    www.getodin.ai

    Build and deploy AI agents that automate enterprise workflows across connected apps, data, and systems.

    Paid
  • Langdock

    www.langdock.com

    Company-wide AI chat, agents, workflows and governance with access to multiple AI models.

    Paid