Introduction

Crabllm is a high-performance LLM API gateway written in Rust. It sits between your application and LLM providers, exposing an OpenAI-compatible API surface.

One API format. Many providers. Low overhead.

What It Does

You send requests in OpenAI format to crabllm. It routes them to the configured provider — OpenAI, Anthropic, Google Gemini, Azure OpenAI, or Ollama — translating the request and response as needed.

The full HTTP API surface is documented interactively at crabtalk.github.io/crabllm/api.

Your application talks to one endpoint. Crabllm handles the rest:

• Provider translation — Anthropic and Google have their own API formats. Crabllm translates automatically.
• Routing — Weighted random selection across multiple providers for the same model. Automatic fallback when a provider fails.
• Streaming — SSE streaming proxied without buffering.
• Auth — Virtual API keys with per-key model access control.
• Extensions — Rate limiting, caching, cost tracking, budget enforcement.

Why Rust

• Sub-millisecond overhead — no GC pauses, no interpreter startup.
• Memory safety — without runtime cost.
• Concurrency — Tokio async runtime handles thousands of concurrent streaming connections efficiently.
• Deployment — single static binary. No interpreter, no virtualenv, no Docker required.

Feature Comparison

Getting Started

Install

cargo install crabllm

Configure

Create a crabllm.toml file:

listen = "0.0.0.0:8080"

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o", "gpt-4o-mini"]

[providers.anthropic]
kind = "anthropic"
api_key = "${ANTHROPIC_API_KEY}"
models = ["claude-sonnet-4-20250514"]

Environment variables in ${VAR} syntax are expanded at startup.

Run

crabllm --config crabllm.toml

You’ll see:

crabllm listening on 0.0.0.0:8080 (3 models, 2 providers, 0 extensions)

Send a Request

All requests use the OpenAI format, regardless of which provider handles them:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

To use Anthropic, just change the model name:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The request format is the same. Crabllm translates it to the Anthropic Messages API internally.

Streaming

Add "stream": true to get SSE streaming:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Model Aliasing

Map friendly names to canonical model names:

[aliases]
gpt4 = "gpt-4o"
claude = "claude-sonnet-4-20250514"

Now "model": "gpt4" routes to gpt-4o.

Next Steps

• Configuration — full reference for all config options
• Providers — setup guides for each provider
• Features — routing, auth, extensions, and more

Running with Docker

The crabllm image is published to the GitHub Container Registry as ghcr.io/crabtalk/crabllm:latest. It’s a minimal debian:bookworm-slim image with the statically-linked binary inside.

Quick start

mkdir -p ./crabllm-data
docker run -d --name crabllm \
  -p 5632:5632 \
  -v ./crabllm-data:/data \
  ghcr.io/crabtalk/crabllm:latest \
  serve --config /data/crabllm.toml --bind 0.0.0.0:5632

First run: crabllm.toml doesn’t exist, so the server generates one in ./crabllm-data/crabllm.toml with a fresh admin token and default API key. Inspect it:

cat ./crabllm-data/crabllm.toml | grep -E 'admin_token|key'

Copy the two sk-… values — you need the admin token to manage providers, and the default key to call the gateway.

Connecting with crabctl

Install crabctl on the host (or any machine with network access to the container):

cargo install crabctl

Export the admin token and point it at the container:

export CRABLLM_URL=http://127.0.0.1:5632
export CRABLLM_TOKEN=<admin_token-from-crabllm.toml>

Add a provider dynamically (no restart, no TOML edits):

# Known kind — OpenAI
crabctl providers create openai \
  --kind openai \
  --api-key "$OPENAI_API_KEY"

# Self-defined kind for any OpenAI-compatible upstream
crabctl providers create openrouter \
  --kind openrouter \
  --base-url https://openrouter.ai/api/v1 \
  --api-key "$OPENROUTER_API_KEY"

With --models omitted, the server calls {base_url}/models and populates the list automatically.

List what’s live:

crabctl providers list
crabctl keys list

Calling the gateway

Use the default key (not the admin token) for completions:

curl http://127.0.0.1:5632/v1/chat/completions \
  -H "Authorization: Bearer <default_key-from-crabllm.toml>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

OpenAPI docs live at http://127.0.0.1:5632/docs. An online snapshot is hosted at crabtalk.github.io/crabllm/api.

Passing upstream API keys via environment

crabllm.toml expands ${VAR} at load time, so secrets don’t have to live in the file. Edit the generated config:

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o"]

Then pass the variable to the container:

docker run -d --name crabllm \
  -p 5632:5632 \
  -v ./crabllm-data:/data \
  -e OPENAI_API_KEY \
  ghcr.io/crabtalk/crabllm:latest \
  serve --config /data/crabllm.toml --bind 0.0.0.0:5632

docker-compose

services:
  crabllm:
    image: ghcr.io/crabtalk/crabllm:latest
    restart: unless-stopped
    ports:
      - "5632:5632"
    volumes:
      - ./crabllm-data:/data
    environment:
      - OPENAI_API_KEY
      - ANTHROPIC_API_KEY
    command:
      - serve
      - --config
      - /data/crabllm.toml
      - --bind
      - 0.0.0.0:5632

Verbose logs

Pass -v (info), -vv (debug), or -vvv (trace) before the subcommand to see request/response lines and outbound provider calls:

docker run --rm ghcr.io/crabtalk/crabllm:latest -vv serve --config /data/crabllm.toml

RUST_LOG is honored when no -v is given.

TLS backends

The published image uses the native-tls default, which links libssl3 at runtime (installed in the image). If you need the pure-Rust rustls stack — for example to drop libssl from a derived image — rebuild from source:

cargo install crabllm --no-default-features --features rustls,openapi

Configuration

Crabllm is configured via a TOML file, passed with --config:

crabllm --config crabllm.toml

The --bind flag overrides the listen address.

Environment Variables

Strings containing ${VAR} are expanded from environment variables at startup. Unknown variables expand to empty string. Use this for secrets:

api_key = "${OPENAI_API_KEY}"

Top-Level Fields

Providers

Each provider is a named entry under [providers]:

[providers.my_openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o", "gpt-4o-mini"]

Virtual Keys

[[keys]]
name = "team-a"
key = "sk-team-a-secret"
models = ["gpt-4o", "claude-sonnet-4-20250514"]

[[keys]]
name = "admin"
key = "sk-admin-secret"
models = ["*"]

When no keys are configured, authentication is disabled.

Aliases

[aliases]
gpt4 = "gpt-4o"
claude = "claude-sonnet-4-20250514"

Maps friendly model names to canonical names. Single-hop lookup.

Pricing

[pricing.gpt-4o]
prompt_cost_per_million = 2.50
completion_cost_per_million = 10.00

[pricing.claude-sonnet-4-20250514]
prompt_cost_per_million = 3.00
completion_cost_per_million = 15.00

Per-model token pricing in USD. Used by the budget extension for spend tracking.

Extensions

[extensions.cache]
ttl = 3600

[extensions.rate_limit]
rpm = 60

[extensions.usage]

[extensions.budget]
default_limit = 10000000

[extensions.logging]
level = "info"

See Extensions for details on each.

Storage

[storage]
kind = "memory"

See Storage for details.

Full Example

listen = "0.0.0.0:8080"
shutdown_timeout = 30

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o", "gpt-4o-mini"]
weight = 2
max_retries = 2
timeout = 30

[providers.anthropic]
kind = "anthropic"
api_key = "${ANTHROPIC_API_KEY}"
models = ["claude-sonnet-4-20250514"]

[providers.ollama]
kind = "ollama"
models = ["llama3.2"]

[aliases]
gpt4 = "gpt-4o"
claude = "claude-sonnet-4-20250514"

[[keys]]
name = "default"
key = "${CRABTALK_API_KEY}"
models = ["*"]

[pricing.gpt-4o]
prompt_cost_per_million = 2.50
completion_cost_per_million = 10.00

[extensions.rate_limit]
rpm = 100

[extensions.usage]

[extensions.logging]
level = "info"

[storage]
kind = "sqlite"
path = "crabllm.db"

Providers

A provider is an LLM service that crabllm routes requests to. Each provider has its own API format and authentication mechanism. Crabllm translates between the OpenAI-compatible format your application uses and the provider’s native format.

Supported Providers

The last five (deepseek … kimi) share one compat implementation — each is just a name plus two base URLs in the provider crate’s compat table. Adding another OpenAI+Anthropic provider is a one-line entry there.

The proxy is dialect-pure: each endpoint forwards raw bytes only to providers that natively speak that dialect, with no format translation. So an OpenAI-only provider (e.g. xAI Grok or Meta, configured as kind = "openai" with a base_url) is reachable through /v1/chat/completions but not through the /v1/messages Anthropic endpoint. To serve Anthropic-format traffic, a provider needs a native Anthropic endpoint — that’s what the compat table is for.

Common Fields

Every provider supports these fields:

[providers.name]
kind = "..."           # optional — defaults to the section name (`name`)
api_key = "..."        # API key (supports ${ENV_VAR})
base_url = "..."       # base URL override
models = ["..."]       # model names this provider serves
weight = 1             # routing weight (higher = more traffic)
max_retries = 2        # retries on transient errors (429, 5xx)
timeout = 30           # per-request timeout in seconds

When kind is omitted, the section name is used as the kind — [providers.zai] resolves to kind zai. Set kind explicitly only when the section name isn’t the kind, e.g. running two openai instances under different names for routing.

Multiple Providers for the Same Model

When multiple providers list the same model, crabllm selects between them using weighted random selection. If the selected provider fails, it falls back to the next provider by weight. See Routing.

[providers.openai_primary]
kind = "openai"
api_key = "${OPENAI_KEY_1}"
models = ["gpt-4o"]
weight = 3

[providers.openai_backup]
kind = "openai"
api_key = "${OPENAI_KEY_2}"
models = ["gpt-4o"]
weight = 1

Endpoint Support

The Compat column covers every compat-table provider (deepseek, zai, qwen, minimax, kimi) — they share one implementation, so their endpoint support is identical. Whether a given provider actually serves embeddings varies; see its page.

OpenAI

The openai provider works with OpenAI and any OpenAI-compatible API (Groq, Together AI, vLLM, etc.). Requests are forwarded as-is with URL and auth rewrite — no translation needed.

Configuration

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o", "gpt-4o-mini", "text-embedding-3-small"]

Custom Base URL

For OpenAI-compatible services, set base_url:

[providers.groq]
kind = "openai"
api_key = "${GROQ_API_KEY}"
base_url = "https://api.groq.com/openai/v1"
models = ["llama-3.3-70b-versatile"]

[providers.together]
kind = "openai"
api_key = "${TOGETHER_API_KEY}"
base_url = "https://api.together.xyz/v1"
models = ["meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo"]

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings
• Image generation
• Audio speech (TTS)
• Audio transcription

Tool Calling

Tool calling works as-is — the request body is forwarded directly:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }'

Anthropic

The anthropic provider translates OpenAI-format requests to the Anthropic Messages API and back.

Configuration

[providers.anthropic]
kind = "anthropic"
api_key = "${ANTHROPIC_API_KEY}"
models = ["claude-sonnet-4-20250514", "claude-haiku-4-20250514"]

Translation

Crabllm handles the full translation between OpenAI and Anthropic formats:

• System messages — extracted from the messages array and sent as the Anthropic system parameter.
• Stop reasons — mapped between formats (end_turn to stop, etc.).
• Tool calling — fully supported. Tool definitions, tool use responses, and tool result messages are all translated.
• Streaming — Anthropic’s event stream (message_start, content_block_delta, etc.) is translated to OpenAI-format SSE chunks.

Usage

Send requests in OpenAI format as usual:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ]
  }'

Limitations

• Embeddings, image generation, and audio endpoints are not supported by the Anthropic API.

Google Gemini

The google provider translates OpenAI-format requests to the Google Gemini API (generativeai).

Configuration

[providers.google]
kind = "google"
api_key = "${GOOGLE_API_KEY}"
models = ["gemini-2.0-flash", "gemini-2.5-pro"]

Translation

• System messages — mapped to Gemini’s systemInstruction field.
• Roles — assistant mapped to model, user stays user.
• Content — mapped to Gemini’s parts array format.
• Tool calling — tool definitions mapped to functionDeclarations, tool messages to functionResponse parts, responses extract functionCall parts.
• Streaming — uses streamGenerateContent?alt=sse and translates the Gemini event stream to OpenAI-format SSE chunks.

Usage

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.0-flash",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Limitations

• Embeddings, image generation, and audio endpoints are not supported.

Azure OpenAI

The azure provider routes to Azure OpenAI deployments. The request body is OpenAI-format (no translation needed), but the URL pattern and authentication differ.

Configuration

[providers.azure]
kind = "azure"
api_key = "${AZURE_OPENAI_KEY}"
base_url = "https://my-resource.openai.azure.com"
api_version = "2024-02-01"
models = ["gpt-4o"]

• base_url — your Azure OpenAI resource URL.
• api_version — the Azure API version string.

How It Works

Crabllm rewrites the URL to Azure’s deployment-based pattern:

POST /openai/deployments/{model}/chat/completions?api-version={api_version}

Authentication uses the api-key header instead of Authorization: Bearer.

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings
• Image generation
• Audio speech (TTS)
• Audio transcription

Usage

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Ollama

The ollama provider connects to a local Ollama instance. Ollama exposes an OpenAI-compatible API, so requests are forwarded as-is.

Configuration

[providers.ollama]
kind = "ollama"
models = ["llama3.2", "mistral"]

The default base URL is http://localhost:11434/v1. Override it if Ollama runs on a different host:

[providers.ollama]
kind = "ollama"
base_url = "http://192.168.1.100:11434/v1"
models = ["llama3.2"]

No API key is needed for local Ollama.

Usage

Start Ollama, pull a model, then send requests through crabllm:

ollama pull llama3.2
crabllm --config crabllm.toml

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings (if supported by the Ollama model)

DeepSeek

The deepseek provider routes to DeepSeek’s models. DeepSeek exposes two compatible surfaces and crabllm uses both: an OpenAI-compatible endpoint for chat and streaming, and a native Anthropic-compatible endpoint for Anthropic-format passthrough. Authentication is a bearer token on both.

Configuration

[providers.deepseek]   # section name doubles as kind; add `kind = "deepseek"` only if you rename it
api_key = "${DEEPSEEK_API_KEY}"
models = ["deepseek-chat", "deepseek-reasoner"]

Defaults, no base_url needed:

• OpenAI-compatible: https://api.deepseek.com/v1
• Anthropic-compatible: https://api.deepseek.com/anthropic

Set base_url to override the origin (both endpoints derive from it).

Usage

Send requests in OpenAI format as usual — just set the model to a DeepSeek id:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-chat",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Anthropic-format requests are served natively by DeepSeek’s Anthropic-compatible endpoint rather than translated.

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Anthropic Messages (native passthrough, streaming and non-streaming)
• Tool/function calling

Limitations

• Embeddings, image generation, and audio endpoints are not offered by DeepSeek.

z.ai

The zai provider routes to z.ai’s GLM models. z.ai exposes two compatible surfaces and crabllm uses both: an OpenAI-compatible endpoint for chat, streaming, and embeddings, and a native Anthropic-compatible endpoint for Anthropic-format passthrough. Authentication is a bearer token on both.

Configuration

[providers.zai]        # section name doubles as kind; add `kind = "zai"` only if you rename it
api_key = "${ZAI_API_KEY}"
models = ["glm-4.7", "glm-4.7-flash", "glm-5"]

Defaults, no base_url needed:

• OpenAI-compatible: https://api.z.ai/api/paas/v4
• Anthropic-compatible: https://api.z.ai/api/anthropic/v1 (the /v1/messages path the Anthropic SDK / Claude Code resolve from ANTHROPIC_BASE_URL)

Set base_url to override the OpenAI-compatible origin (e.g. a regional or proxied endpoint).

Usage

Send requests in OpenAI format as usual — just set the model to a GLM id:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-4.7",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Anthropic-format requests are served natively by z.ai’s Anthropic-compatible endpoint rather than translated.

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings
• Anthropic Messages (native passthrough, streaming and non-streaming)
• Tool/function calling

Limitations

• Image generation and audio endpoints are not offered by z.ai.

Qwen

The qwen provider routes to Alibaba’s Qwen models via DashScope. Qwen exposes two compatible surfaces and crabllm uses both: an OpenAI-compatible endpoint for chat, streaming, and embeddings, and a native Anthropic-compatible endpoint for Anthropic-format passthrough. Authentication is a bearer token on both (the Anthropic endpoint also accepts x-api-key).

Configuration

[providers.qwen]       # section name doubles as kind; add `kind = "qwen"` only if you rename it
api_key = "${DASHSCOPE_API_KEY}"
models = ["qwen-max", "qwen-plus", "qwen-turbo"]

Defaults, no base_url needed (international / -intl region):

• OpenAI-compatible: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
• Anthropic-compatible: https://dashscope-intl.aliyuncs.com/apps/anthropic/v1 (the /v1/messages path the Anthropic SDK / Claude Code resolve from base_url)

Set base_url to override the OpenAI-compatible origin — e.g. the mainland-China endpoint https://dashscope.aliyuncs.com/compatible-mode/v1.

Usage

Send requests in OpenAI format as usual — just set the model to a Qwen id:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-max",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Anthropic-format requests are served natively by Qwen’s Anthropic-compatible endpoint rather than translated.

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings
• Anthropic Messages (native passthrough, streaming and non-streaming)
• Tool/function calling

Limitations

• The Anthropic-compatible endpoint serves only /v1/messages — it has no /v1/models discovery route, which some Anthropic-native tools expect.

MiniMax

The minimax provider routes to MiniMax’s models. It exposes an OpenAI-compatible endpoint for chat, streaming, and embeddings, and a native Anthropic-compatible endpoint for Anthropic-format passthrough. Authentication is a bearer token on both.

Configuration

[providers.minimax]    # section name doubles as kind; add `kind = "minimax"` only if you rename it
api_key = "${MINIMAX_API_KEY}"
models = ["MiniMax-M2"]

Defaults, no base_url needed:

• OpenAI-compatible: https://api.minimax.io/v1
• Anthropic-compatible: https://api.minimax.io/anthropic/v1 (the /v1/messages path the Anthropic SDK / Claude Code resolve from ANTHROPIC_BASE_URL)

Set base_url to override the OpenAI-compatible origin (e.g. the api.minimaxi.com region).

Usage

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMax-M2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Anthropic-format requests are served natively by MiniMax’s Anthropic-compatible endpoint rather than translated.

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings
• Anthropic Messages (native passthrough, streaming and non-streaming)
• Tool/function calling

Limitations

• The Anthropic-compatible endpoint reports a smaller context_window in model metadata than some MiniMax models actually support; clients that trust that metadata (e.g. Claude Code) may cap their budget early.

Kimi

The kimi provider routes to Moonshot AI’s Kimi models. It exposes an OpenAI-compatible endpoint for chat, streaming, and embeddings, and a native Anthropic-compatible endpoint for Anthropic-format passthrough. Authentication is a bearer token on both.

Configuration

[providers.kimi]       # section name doubles as kind; add `kind = "kimi"` only if you rename it
api_key = "${MOONSHOT_API_KEY}"
models = ["kimi-k2-0711-preview"]

Defaults, no base_url needed:

• OpenAI-compatible: https://api.moonshot.ai/v1
• Anthropic-compatible: https://api.moonshot.ai/anthropic/v1 (the /v1/messages path the Anthropic SDK / Claude Code resolve from ANTHROPIC_BASE_URL)

Set base_url to override the OpenAI-compatible origin (e.g. the mainland-China endpoint https://api.moonshot.cn/v1).

Usage

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2-0711-preview",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Anthropic-format requests are served natively by Moonshot’s Anthropic-compatible endpoint rather than translated.

Supported Endpoints

• Chat completions (streaming and non-streaming)
• Embeddings
• Anthropic Messages (native passthrough, streaming and non-streaming)
• Tool/function calling

Limitations

• Moonshot rescales the sampling temperature (real = requested * 0.6), so a given temperature behaves cooler than on other providers.

Routing

Crabllm decides which provider handles a request based on model name, routing weights, and fallback logic.

Model Resolution

When a request arrives, crabllm looks up the model name in the configured providers. If the model is an alias, it resolves to the canonical name first (single-hop lookup).

Weighted Selection

When multiple providers serve the same model, one is selected via weighted random selection. Higher weight values mean more traffic:

[providers.primary]
kind = "openai"
api_key = "${OPENAI_KEY_1}"
models = ["gpt-4o"]
weight = 3                    # 75% of traffic

[providers.secondary]
kind = "openai"
api_key = "${OPENAI_KEY_2}"
models = ["gpt-4o"]
weight = 1                    # 25% of traffic

Selection is stateless — no shared counters. Each request picks independently.

Retry

When a provider returns a transient error (HTTP 429, 500, 502, 503, 504), crabllm retries the same provider with exponential backoff:

• Base delay: 100ms, doubling each retry.
• Full jitter: each sleep is a random duration in [backoff/2, backoff] to prevent thundering herd.
• Max retries: configurable per provider via max_retries (default 2).

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o"]
max_retries = 3               # retry up to 3 times

Set max_retries = 0 to disable retry entirely.

Fallback

When retries are exhausted on a provider, crabllm tries the next provider by descending weight. This continues until a provider succeeds or all providers have been tried.

# Primary provider (tried first)
[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o"]
weight = 2

# Fallback provider (tried if primary fails)
[providers.azure]
kind = "azure"
api_key = "${AZURE_KEY}"
base_url = "https://my-resource.openai.azure.com"
api_version = "2024-02-01"
models = ["gpt-4o"]
weight = 1

Timeouts

Each provider call is wrapped in a timeout. If the timeout expires, the request is treated as a transient error (triggers retry/fallback):

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o"]
timeout = 60                  # seconds (default: 30)

Timeout errors return HTTP 504 Gateway Timeout if all providers time out.

Streaming Behavior

For streaming requests, retry and fallback only apply to connection errors (before the stream starts). Once the first SSE chunk is sent to the client, the connection is committed to that provider.

Streaming

Crabllm supports Server-Sent Events (SSE) streaming for chat completions across all providers. Streams are proxied without buffering — tokens arrive incrementally as the provider generates them.

Usage

Set "stream": true in the request body:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Write a haiku."}],
    "stream": true
  }'

The response is a stream of SSE events:

data: {"id":"...","object":"chat.completion.chunk","choices":[{"delta":{"content":"An"}}]}

data: {"id":"...","object":"chat.completion.chunk","choices":[{"delta":{"content":" old"}}]}

data: [DONE]

Provider Translation

For non-OpenAI providers, crabllm translates the provider’s native streaming format to OpenAI-compatible SSE chunks:

• Anthropic — message_start, content_block_delta events translated to chat.completion.chunk format.
• Google Gemini — streamGenerateContent response parts translated to OpenAI chunks.
• Azure — same SSE format as OpenAI, no translation needed.

Extension Hooks

Extensions can observe each streaming chunk via the on_chunk hook. The rate limiter and budget extension use this to count tokens in real-time as they arrive.

Keep-Alive

SSE connections include automatic keep-alive pings to prevent proxy/load balancer timeouts during long generation pauses.

Error Handling

If an error occurs mid-stream (after the first chunk has been sent), it is delivered as an SSE event with an error payload. The stream then terminates. Retry and fallback only apply before the stream starts.

Authentication

Crabllm supports virtual API keys for client authentication and model access control.

Virtual Keys

Define keys in the config:

[[keys]]
name = "team-frontend"
key = "sk-frontend-abc123"
models = ["gpt-4o-mini"]

[[keys]]
name = "team-backend"
key = "sk-backend-xyz789"
models = ["gpt-4o", "claude-sonnet-4-20250514"]

[[keys]]
name = "admin"
key = "${ADMIN_API_KEY}"
models = ["*"]

Clients send the key in the Authorization header:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-frontend-abc123" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Hi"}]}'

Model Access Control

The models field controls which models a key can access:

• ["gpt-4o", "gpt-4o-mini"] — only these models.
• ["*"] — all models.

Requests for unauthorized models return HTTP 401.

No Auth Mode

When no keys are configured, authentication is disabled entirely. All requests pass through without checking the Authorization header.

# No [[keys]] section = auth disabled
listen = "0.0.0.0:8080"

[providers.openai]
kind = "openai"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o"]

Key Name Tracking

The key name field is used by extensions for per-key tracking:

• Rate limiting — enforced per key name.
• Usage tracking — tokens counted per key name.
• Budget — spend limits per key name.
• Logging — key name included in log entries.

Extensions

Extensions add functionality to the request pipeline via hooks. They run in-handler (not as middleware), giving direct access to typed request and response data.

Available Extensions

Cache

Caches non-streaming chat completion responses. Cache key is a SHA-256 hash of the serialized request body.

[extensions.cache]
ttl_seconds = 3600           # default: 300 (5 minutes)

Admin route: DELETE /v1/cache — clears all cached entries.

Rate Limit

Enforces per-key request and token rate limits using a per-minute sliding window.

[extensions.rate_limit]
requests_per_minute = 60      # required
tokens_per_minute = 100000    # optional

Returns HTTP 429 when limits are exceeded. Token counting uses actual usage from provider responses (both streaming and non-streaming).

Usage Tracker

Accumulates prompt and completion token counts per key and model.

[extensions.usage]

No configuration needed. Admin route: GET /v1/usage — returns JSON array of usage entries with key, model, prompt_tokens, and completion_tokens.

Budget

Enforces per-key spend limits. Requires pricing to be configured for the models in use.

[extensions.budget]
default_budget = 10.00        # USD, required

[extensions.budget.keys.team-a]
budget = 50.00                # USD override for this key

Returns HTTP 429 when a key’s spend exceeds its budget. Admin route: GET /v1/budget — returns JSON array with key, spent_usd, budget_usd, and remaining_usd.

Logging

Structured request logging via the tracing framework.

[extensions.logging]
level = "info"

Logs completed requests (model, provider, key, latency, token counts) and errors. Initializes the tracing_subscriber when enabled.

Hook Pipeline

Extensions run in config order at these points:

1. on_request — before provider dispatch. Can short-circuit (rate limit, budget).
2. on_cache_lookup — before provider dispatch for non-streaming. Returns cached response if available.
3. on_response — after successful non-streaming response.
4. on_chunk — for each SSE chunk during streaming.
5. on_error — when a provider call fails.

Combining Extensions

Multiple extensions can be enabled simultaneously:

[extensions.logging]
level = "info"

[extensions.rate_limit]
requests_per_minute = 100

[extensions.usage]

[extensions.cache]
ttl_seconds = 600

[extensions.budget]
default_budget = 100.00

All extensions share the same storage backend.

Storage

Extensions that persist data (cache, rate limits, usage, budget) use a shared storage backend. Three backends are available.

Memory (default)

In-memory storage using concurrent hash maps. Fast, but data is lost on restart.

[storage]
kind = "memory"

This is the default when no [storage] section is present. No feature flag required.

SQLite

Persistent storage using SQLite via async pooled connections.

[storage]
kind = "sqlite"
path = "crabllm.db"

Requires the storage-sqlite feature:

cargo install crabllm --features storage-sqlite

The database file is created automatically if it doesn’t exist. Uses two tables (kv and counters) with atomic increment via INSERT ... ON CONFLICT ... RETURNING.

Redis

Remote persistent storage using Redis async multiplexed connections.

[storage]
kind = "redis"
path = "redis://127.0.0.1:6379"

Requires the storage-redis feature:

cargo install crabllm --features storage-redis

Supports standard Redis URLs. Increment maps to INCRBY, key listing uses SCAN with prefix glob patterns.

How Extensions Use Storage

Each extension namespaces its keys with a 4-byte prefix to avoid collisions:

Architecture

Principles

• Simplicity over abstraction. No trait where a function suffices.
• Single responsibility. Each crate has one focused job.
• OpenAI as canonical format. Providers translate to/from it.
• Streaming first-class. Never buffer a full response when streaming.
• Configuration-driven. Provider setup and routing from config, not code.
• Minimal gateway latency. Avoid hot-path allocations.

Workspace Layout

crabllm/
  crates/
    crabllm/   — binary, wires everything together
    core/       — shared types, config, errors
    provider/   — provider enum + translation modules
    proxy/      — HTTP server, routing, extensions
    bench/      — benchmark mock backend

Crates

crabllm

Binary entry point. Loads TOML config, builds the provider registry, initializes the storage backend and extensions, starts the Axum HTTP server. CLI args: --config and --bind.

core

Shared types with no business logic. Contains:

• Config — GatewayConfig with env var interpolation.
• Types — OpenAI-compatible wire format structs (request, response, chunk).
• Error — error enum with transient detection for retry logic.
• Storage — async KV trait with memory, SQLite, and Redis backends.
• Extension — hook trait for the request pipeline.

provider

Provider dispatch. The Provider enum has variants for each supported provider. Each variant dispatches to a per-provider module that handles request/response translation. ProviderRegistry maps model names to weighted deployment lists.

proxy

Axum HTTP server. Route handlers implement retry + fallback across deployments. Auth middleware validates virtual keys. Five built-in extensions run as in-handler hooks.

Request Flow

1. Client sends OpenAI-format request to crabllm.
2. Auth middleware validates the bearer token.
3. Handler resolves model name (aliases) and gets deployment list.
4. Extension on_request hooks run (rate limit, budget check).
5. Cache lookup for non-streaming requests.
6. Provider dispatch with retry + fallback.
7. Provider translates request, calls upstream, translates response.
8. Extension on_response/on_chunk hooks run (usage, budget, cache store).
9. Response returned to client.

Benchmarks

Gateway overhead measured against a mock LLM server with instant responses — numbers reflect pure proxy cost.

Latency: P50 / P99 in milliseconds. Lower is better.

Chat Completions

Streaming

Embeddings

Memory (Peak RSS)