Open Source · Self-Hosted · OpenAI-Compatible
One gateway for every LLM your organization runs
Give your teams a single, governed endpoint for AI, running on infrastructure you control. Centralize access, track every request and its cost, and turn any model agentic without writing agent code.
One endpoint for all your AI backends
LLM Gateway is an open-source, self-hosted proxy that places a single OpenAI-compatible endpoint in front of every model you run, local GPUs, on-prem clusters, or cloud providers. Point your existing tools and SDKs at it and instantly gain centralized authentication, access control, cost tracking, and built-in agentic tools, without changing a line of application code.
The problems it solves
Shadow AI and data you can't control
When teams reach for public AI services, sensitive prompts, documents, and code leave your network, creating compliance and confidentiality risks you can't audit. LLM Gateway keeps inference on infrastructure you own and gives every user a sanctioned endpoint, so data stays where it belongs.
No visibility into usage or cost
Without a central chokepoint, nobody knows who is spending what, on which model, for which project. LLM Gateway meters every request, token, and euro per user, model, and source, with quotas, rate limits, and live dashboards that make AI spend predictable.
Access sprawl across teams and models
API keys copied into scripts, no way to revoke access, no idea who can reach which model. LLM Gateway centralizes authentication with OIDC and role-based access control, issuing per-user tokens that gate exactly which models and tools each person may use.
Building agents is slow and expensive
Web search, document parsing, RAG, and code execution normally mean custom agent frameworks and months of engineering. LLM Gateway ships 40+ tools that the model runs server-side, mid-completion, turning any OpenAI-compatible model agentic with zero client-side code.
How it works
LLM Gateway exposes a single, OpenAI-compatible endpoint. Any SDK or application that already speaks OpenAI points at it unchanged, chat completions, embeddings, image generation, transcription, and text-to-speech all flow through the same gateway.
Behind that endpoint, it load-balances requests across all your backends with health checks and automatic failover, and discovers available models live, no static configuration to maintain as your fleet changes. Every request passes through OIDC authentication, role-based access control, and per-user quotas before it ever reaches a model.
When a model needs to act, the gateway executes tools server-side and feeds the results back into the completion, web search, RAG over your own data, code execution in isolated sandboxes, document generation, and per-user MCP connectors, so your applications stay simple while the model does the work.
Built for control
Centralized authentication & RBAC
OIDC browser login against your identity provider, per-user API tokens hashed at rest, and role-based access control that gates models and tools per user and role. Grant, scope, and revoke access from one place.
Full cost & usage visibility
Metering of requests, tokens, and spend per user, model, and source. Global, per-role, and per-user quotas and rate limits over rolling windows, plus live dashboards that turn AI cost into a number you can plan around.
Agentic capabilities out of the box
40+ built-in tools, web search, URL and PDF reading, RAG over your codebase and documents, code execution, document rendering, and per-user MCP connectors to systems like GitHub, Jira, GitLab, and Slack, all executed server-side.
Your data, your infrastructure
A single self-hosted binary with SQLite for state, no external vector database, no separate frontend, no data leaving your environment. Run it on the hardware you already trust, with a built-in chat UI for end users.
Deploys in minutes, not months
LLM Gateway ships as one hardened binary written in Rust, no compose sprawl, no vector database, no separate frontend to operate. Deploy it with a systemd-podman Quadlet or Docker, terminate TLS at your existing reverse proxy, and point your teams at the built-in chat interface. It fits the way your infrastructure already works instead of forcing a new platform on it.
Bring governed, self-hosted AI to your teams
Talk to the croit team about deploying LLM Gateway in your environment, from a first evaluation to enterprise licensing and support.
Open source, enterprise ready
Free and open source
LLM Gateway is released under the AGPL-3.0 licence, the complete gateway, agentic tools, chat UI, and access control included. Inspect it, run it, and adapt it to your environment with no per-seat fees and no vendor lock-in.
Backed by croit
Built by the team that operates large-scale infrastructure in demanding enterprise and HPC environments. Commercial licensing, expert deployment, and support are available from croit when you need production assurance and someone to call.