Skip to content
Sign in
  • Browse all toolsFilter by category and pricing
  • Browse categoriesWriting, code, image, audio and more
  • Compare toolsPut up to three side by side
  • Read the blogGuides and comparisons
All categories

LLM Infrastructure

Tools for serving, routing, evaluating and monitoring large language models

206tools listed

Pricing mix

  • 102 Free
  • 85 Freemium
  • 19 Paid

187 tools match

  • Google T5

    www.github.com

    Open-source text-to-text transformer library for training, fine-tuning, and evaluating NLP models.

    Free
  • GPT Prompt Engineer

    www.github.com

    Open-source prompt generation, testing and ELO ranking in Google Colab or local Jupyter notebooks.

    Free
  • BIG-bench

    www.github.com

    Open-source benchmark for evaluating language models with JSON and programmatic tasks

    Free
  • FalkorDB

    www.github.com

    Graph database for knowledge graphs, GraphRAG, and agent memory, with OpenCypher queries and Redis client access.

    Free
  • ToolPermit

    www.github.com

    Local-first permission policies, one-time approvals, and redacted audit logs for MCP tool calls.

    Free
  • nobodywho

    www.github.com

    Open-source inference engine for local LLMs, tool calling, speech recognition and speech synthesis

    Free
  • Open-source model optimization library for quantization, pruning, distillation and faster inference.

    Free
  • Hindsight

    www.github.com

    Open-source agent memory with retain, recall and reflect tools, self-hosted deployment and managed cloud access.

    Freemium
  • CLIProxyAPI

    www.github.com

    Open-source proxy server for accessing CLI models through OpenAI, Gemini, Claude, Codex, and Grok-compatible APIs.

    Free
  • Magnitude

    www.github.com

    Open-source inference engine that tunes local models for your hardware and connects them to coding agents.

    Free
  • reef

    www.github.com

    Open-source infrastructure for improving agent model weights, prompts, rules, and skills from feedback.

    Free
  • Miles

    www.github.com

    Open-source reinforcement learning framework for large-scale LLM and VLM post-training.

    Free
  • datasets

    www.github.com

    Open-source Python library for loading, streaming, and preprocessing machine learning datasets.

    Free
  • SGLang

    www.github.com

    Open-source inference and serving framework for large language, vision-language, and diffusion models.

    Free
  • TurboFieldfare

    www.github.com

    Run Gemma 4 26B-A4B locally on Apple Silicon Macs using about 2 GB of RAM.

    Free
  • Switchyard

    www.github.com

    Open-source model routing for AI agents, with Rust embedding and gateway integrations.

    Free
  • Crawl4AI

    www.github.com

    Open-source web crawling and extraction for AI, with Python, Docker, CLI and hosted API access.

    Freemium
  • RunNburn

    www.github.com

    Memory-aware GGUF inference with CPU and GPU offloading and an OpenAI-compatible server.

    Free
  • kimi-k3-in-c

    www.github.com

    Open-source C99 inference engine for running Kimi K3 on a CPU without BLAS, a framework or a GPU.

    Free
  • Noisegate

    www.github.com

    An open-source differential privacy gateway for AI agents querying sensitive data over MCP.

    Free
  • AgentENV

    www.github.com

    Open-source infrastructure for running, snapshotting and forking agent environments across machines.

    Free
  • Candle

    www.github.com

    Open-source Rust machine learning framework for model training, GPU inference, and browser-based models.

    Free
  • DataFlow

    www.github.com

    Open-source LLM data preparation with reusable operators, visual pipelines, and data generation and cleaning workflows.

    Free
  • openai-oauth

    www.github.com

    Connect ChatGPT accounts to apps through an OpenAI-compatible proxy, TypeScript SDK and browser sign-in.

    Free
  • PageIndex

    www.github.com

    Open-source, reasoning-based document retrieval using hierarchical tree indexes instead of vector databases.

    Free
  • SigNoz

    www.github.com

    OpenTelemetry-native observability for logs, metrics, traces, infrastructure, and AI applications.

    Freemium
  • plano

    www.github.com

    Open-source AI proxy for agent orchestration, LLM routing, guardrails, and OpenTelemetry observability.

    Free
  • Heretic

    www.github.com

    Open-source language model modification for refusal suppression, shorter responses, and reduced sycophancy.

    Free
  • slime

    www.github.com

    Open-source LLM reinforcement learning framework connecting Megatron training with SGLang rollout.

    Free
  • marin

    www.github.com

    Open-source framework for foundation model training, data preparation, evaluation, and reproducible experiments.

    Free
  • LMCache

    www.github.com

    Open-source KV cache management for reusable, persistent caching across LLM inference engines and storage backends.

    Free
  • CocoIndex

    www.github.com

    Open-source incremental data engine that keeps context fresh for AI agents and LLM applications.

    Free
  • vLLM

    www.github.com

    Open-source LLM inference and serving with PagedAttention, distributed execution and an OpenAI-compatible API.

    Free
  • Rig

    www.github.com

    Rust library and plugin-based terminal coding agent for building LLM applications.

    Free
  • NeMo

    www.github.com

    Open-source framework for creating, customizing, and deploying speech recognition, text-to-speech, and speech LLMs.

    Free
  • context-mode

    www.github.com

    A free local plugin that sandboxes coding-agent tool output and retrieves relevant results through SQLite search.

    Free