An intelligent LLM gateway that saves money, blocks attacks, and runs at the edge. Here's how each piece works.
Responses are cached in two layers: SimpleCache (per-POP, microsecond lookups) and the global KV Store. When a user at any edge PoP asks a question that's been answered before, the response is served instantly without ever hitting the LLM provider. This is Fastly's CDN cache, purpose-built for AI.
Every cache hit is a $0 LLM call. At 50% cache hit rate on a 10k req/day workload with GPT-4o, you save ~$150/day in API costs. Edge cache hits return in <1ms vs 500ms+ for a live LLM call.
Bring Your Own Keys (BYOK) — connect your existing API keys and ModelRouter routes across all of them. 7 providers: Ollama (self-hosted), OpenAI, Anthropic, Groq, Together AI, Mistral, and DeepSeek. 26 models total across all tiers. We never see your LLM bills — you pay providers directly.
Multiple providers offer included tiers. Between Ollama, Groq, Together AI, Mistral, and DeepSeek, you can run production workloads at minimal cost before scaling to premium providers.
30+ heuristic patterns run at the edge in under 1ms. No external API calls, no ML inference latency. Detects instruction overrides, role hijacking, jailbreaks, delimiter injection, and system prompt extraction attempts. Attacks are blocked before they ever reach your LLM.
Blocked requests cost $0 in LLM tokens. Without edge detection, you'd pay for every attack attempt that reaches the model. At scale, prompt injection attempts can be 5-15% of traffic.
Define regex, substring, and exact-match rules with if/any/all/none logic. Rules are stored in the KV store and deployed to every edge PoP instantly — no redeploy needed. Think of it as a WAF for AI: block PII extraction attempts, forbidden topics, or specific model access in real time. Use DLP rules to prevent employees from leaking SSNs, source code, trade secrets, or any sensitive data pattern through AI prompts.
Block SSN patterns (\d{3}-\d{2}-\d{4}), credit card numbers, source code snippets, internal project names, or trade secrets. Rule updates propagate globally in <60 seconds. Test rules against sample prompts before enabling.
Every request is audited to the KV store: provider, model, injection score, latency, token usage, and routing decisions. Entries are keyed by tenant and date for fast retrieval. Know exactly which employees are using AI, what they're asking, and which models are being invoked. This gives you a full compliance trail for SOC 2, HIPAA, AI governance policies, or internal security audits.
Fire-and-forget writes mean zero added latency to your LLM responses. Audit logs are exportable as CSV directly from the UI. Full visibility into AI usage across your organization for compliance reporting, cost allocation, and policy enforcement.
ModelRouter speaks the same API format as OpenAI and OpenRouter. Point any existing client library at your ModelRouter URL and it just works. 10 endpoints cover completions, model listing, prompt inspection, audit logs, custom rules (CRUD + testing), provider registry, provider connection testing, and health checks. No SDK changes, no code rewrites.
Zero integration cost. Swap in ModelRouter in one line of config. No vendor lock-in — you can switch back to direct API calls at any time. BYOK means your keys, your bills, zero markup.
Prompts are classified into 9 tiers — fast, balanced, quality, reasoning, code, vision, ocr, image_gen, and embedding — using content-aware heuristic analysis. Specialized tasks route to purpose-built models: OCR to Mistral OCR (10x cheaper, more accurate than GPT-4o, preserves table structure and formulas), image generation to FLUX.1 via Together AI (faster and cheaper than DALL-E 3), code to DeepSeek Coder, general vision to GPT-4o or Claude Sonnet 4, and reasoning to Claude Opus 4. Each model has a quality ranking (Claude Opus 4 tops at 0.98, Mistral OCR at 0.95 for OCR, FLUX.1 Pro at 0.93 for image gen). Models without required capabilities are automatically excluded. User-controlled levers for cost, speed, and quality weight the final model selection across all 7 providers.
"What time is it in Tokyo?" doesn't need GPT-4o. Automatic tiering means you stop paying premium prices for simple prompts. OCR goes to Mistral OCR (10x cheaper than GPT-4o). Image generation goes to FLUX.1 via Together AI (faster and cheaper than DALL-E 3). Code goes to DeepSeek Coder. General vision to GPT-4o or Claude Sonnet 4. Embeddings go to text-embedding-3-small. Fallback chains across 7 providers ensure reliability if any provider is down.
Runs as WASM at the edge with sub-millisecond cold starts. Routes 26 models across 7 providers. Fastly's 120-second wall-clock limit is plenty for LLM responses. Your routing logic executes at the nearest PoP in Fastly's global network, minimizing the distance between your users and the gateway. 530 tests (Vitest + Playwright) ensure reliability across all tiers and providers.
<1ms cold start. Zero servers to manage. Auto-scaling to any traffic volume. KV Store and SimpleCache are built into the platform — no external Redis or Memcached needed.