Open Source AI Gateway

Key Features

Multi-Provider Support

Seamlessly integrate with OpenAI, Anthropic, Gemini, Ollama, Mistral, and Cohere

HTTP or gRPC

OpenAI API compatible HTTP interface and high performance gRPC interface

Smart Failover

Automatic failover between providers ensures high availability

Intelligent Caching

Built-in caching system with configurable TTL for cost savings

Rate Limiting

Fully configurable rate limiting policy

Admin Dashboard

Monitor usage, tokens, errors and configure gateway

Content Guardrails

Configurable content filtering and safety measures

Enterprise Logging

Integration with Splunk, Datadog, and Elasticsearch

System Prompt Injection

Intercept and inject system prompts for all outgoing requests

Getting Started

Configure

Config.toml file

[openAIConfig]
apiKey = "Your_API_Key"
model = "gpt-4"
endpoint = "https://api.openai.com"

Create a Config.toml file with your API configuration

Run

Docker command

docker run -p 8080:8080 -p 8081:8081 -p 8082:8082 \
-v $(pwd)/Config.toml:/home/ballerina/Config.toml \
chintana/ai-gateway:v1.1.0

Start the container with your configuration mounted

Use

curl command

curl -X POST http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "x-llm-provider: openai" \
    -d '{"messages": [{"role": "user","content": "When will we have AGI? In 10 words"}]}'

Start making API requests to your gateway