Glide is your go-to cloud-native LLM gateway, delivering high-performance LLMOps in a lightweight, all-in-one package.

We take all problems of managing and communicating with external providers out of your applications, so you can dive into tackling your core challenges.

Glide sits between your application and model providers to seamlessly handle various LLMOps tasks like model failover, caching, key management, etc.

Take a look at the develop branch.

Check out our documentation!

• Unified REST API across providers. Avoid vendor lock-in and changes in your applications when you swap model providers.
• High availability and resiliency when working with external model providers. Automatic fallbacks on provider failures, rate limits, transient errors. Smart retries to reduce communication latency.
• Support popular LLM providers.
• High performance. Performance is our priority. We want to keep Glide "invisible" for your latency-wise, while providing rich functionality.
• Production-ready observability via OpenTelemetry, emit metrics on models health, allows whitebox monitoring (coming soon)
• Straightforward and simple maintenance and configuration, centralized API key control & management & rotation, etc.

	Provider	Support Status
	OpenAI	👍 Supported
	Azure OpenAI	👍 Supported
	Cohere	👍 Supported
	OctoML	👍 Supported
	Anthropic	👍 Supported
	Google Gemini	🏗️ Coming Soon

Routers are a core functionality of Glide. Think of routers as a group of models with some predefined logic. For example, the resilience router allows a user to define a set of backup models should the initial model fail. Another example, would be to leverage the least-latency router to make latency sensitive LLM calls in the most efficient manner.

Detailed info on routers can be found here.

The easiest way to deploy Glide is to our demo repository and docker-compose.

git clone https://github.com/EinStack/glide-demo.git

The demo repository comes with a basic config. Additionally, you need to init your secrets by running:

make init # from the demo root

This will create the secrets directory with one .OPENAI_API_KEY file that you need to put your key to.

After that, just use docker compose via this command to start your demo environment:

make up

See API Reference for more details.

{
 "model": "gpt-3.5-turbo", # this is not required but can be used to specify different prompts to different models
 "message":
      {
        "role": "user",
        "content": "Where was it played?"
      },
  "messageHistory": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Who won the world series in 2020?"},
      {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."}
    ]
}

Finally, Glide comes with OpenAPI documentation that is accessible via https://127.0.0.1:9099/v1/swagger/index.html

That's it 🙌

Use our documentation to further learn about Glide capabilities and configs.

Other ways to install Glide are available:

brew tap einstack/tap
brew install einstack/tap/glide

Coming Soon

Glide provides official images in our GHCR:

• Alpine 3.19:

docker pull ghcr.io/einstack/glide:latest-alpine 

• Ubuntu 22.04 LTS:

docker pull ghcr.io/einstack/glide:latest-ubuntu

• Google Distroless (non-root)

docker pull ghcr.io/einstack/glide:latest-distroless

• RedHat UBI 8.9 Micro

docker pull ghcr.io/einstack/glide:latest-redhat

Coming Soon

• Join Discord for real-time discussion

Open an issue or start a discussion if there is a feature or an enhancement you'd like to see in Glide.

• Maintainers

  • Roman Hlushko, Software Engineer, Distributed Systems & MLOps
  • Max Krueger, Data & ML Engineer

Thanks everyone for already put their effort to make Glide better and more feature-rich:

Apache 2.0