AI Gateway

The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

✅  Blazing fast (9.9x faster) with a tiny footprint (~100kb build)
✅  Load balance across multiple models, providers, and keys
✅  Fallbacks make sure your app stays resilient
✅  Automatic Retries with exponential fallbacks come by default
✅  Configurable Request Timeouts to easily handle unresponsive LLM requests
✅  Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models
✅  Plug-in middleware as needed
✅  Battle tested over 480B tokens
✅  Enterprise-ready for enhanced security, scale, and custom deployments

[!TIP] ⭐️ Star this repo to get Github release notifications for new provider integrations and features.

Star History

Use the AI gateway through the hosted API or self-host the open-source or enterprise versions on your environment.

The hosted API is the fastest way to setup an AI Gateway for your Gen AI application. We process billions of tokens daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.

To run the AI gateway locally, execute the following command in your terminal. (Needs npx installed) Or, explore deployment guides for Cloudflare, Docker, Node.js and more here.

npx @portkey-ai/gateway

^{Your AI Gateway is now running on https://localhost:8787 🚀}

The AI Gateway's enterprise version offers enterprise-ready capabilities for org management, governance, security and more out of the box. Compare the open source, hosted and enterprise versions here.

The enterprise deployment architecture, supported platforms is available here - Enterprise Private Cloud Deployments

The AI Gateway is compatible with the OpenAI API & SDKs, and extends them to call 200+ LLMs reliably. To use the Gateway through OpenAI, update the client to include the gateway's URL and headers and make requests as usual. The AI gateway can translate requests written in the OpenAI format to the signature expected by the specified provider. View examples

Portkey Python SDK is a wrapper over the OpenAI Python SDK with added support for additional parameters across all other providers. If you're building with Python, this is the recommended library to connect to the Gateway.

pip install -qU portkey-ai

Portkey JS/TS SDK is a wrapper over the OpenAI JS SDK with added support for additional parameters across all other providers. If you're building with JS or TS, this is the recommended library to connect to the Gateway.

npm install --save portkey-ai

The AI gateway supports OpenAI compatible endpoints with added parameter support for all other providers and models. View API Reference.

• Use models from Nvidia NIM with AI Gateway
• Monitor CrewAI Agents with Portkey!
• Comparing Top 10 LMSYS Models with AI Gateway.

• Create Synthetic Datasets using Nemotron
• Use Portkey Gateway with Vercel's AI SDK
• Monitor Llama Agents with Portkey

Explore Gateway integrations with 25+ providers and 6+ frameworks.

	Provider	Support	Stream
	OpenAI	✅	✅
	Azure OpenAI	✅	✅
	Anyscale	✅	✅
	Google Gemini & Palm	✅	✅
	Anthropic	✅	✅
	Cohere	✅	✅
	Together AI	✅	✅
	Perplexity	✅	✅
	Mistral	✅	✅
	Nomic	✅	✅
	AI21	✅	✅
	Stability AI	✅	✅
	DeepInfra	✅	✅
	Ollama	✅	✅
	Novita AI	✅	✅

View the complete list of 200+ supported models here

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

*Only available on the hosted app. For detailed documentation click here.

Fallbacks Fallback to another provider or model on failed requests. You can specify the errors on which to trigger the fallback. Improves reliability of your application

Automatic Retries Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.

Load Balancing Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.

Request Timeouts Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.

Multi-modal LLM Gateway Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers — all using the familiar OpenAI signature

Guardrails Verify your LLM inputs AND outputs to adhere to your specified checks. Build your own checks or choose from the 20+ pre-built guardrails.

These features are configured through the Gateway Config added to the x-portkey-config header or the config parameter in the SDKs.

Here's a sample config JSON showcasing the above features. All the features are optional

{
	"retry": { "attempts": 5 },
	"request_timeout": 10000,
	"strategy": { "mode": "fallback" }, // or 'loadbalance', etc
	"targets": [{
		"provider": "openai",
		"api_key": "sk-***"
	},{
		"strategy": {"mode": "loadbalance"}, // Optional nesting
		"targets": {...}
	}]
}

Then use the config in your API requests to the gateway.

Here's a guide to use the config object in your request.

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

✅  Secure Key Management - for role-based access control and tracking
✅  Simple & Semantic Caching - to serve repeat queries faster & save costs
✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅  PII Redaction - to automatically remove sensitive data from your requests to prevent inadvertent exposure
✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments

The easiest way to contribute is to pick an issue with the good first issue tag 💪. Read the contribution guidelines here.

Bug Report? File here | Feature Request? File here

Join our growing community around the world, for help, ideas, and discussions on AI.

• View our official Blog
• Chat with us on Discord
• Follow us on Twitter
• Connect with us on LinkedIn