🌈 A Robust cloud-native AI Gateway - core LLMOps infrastructure stack for building production-ready AI Applications. It provides an Universal API for inferencing 100+ LLMs(OpenAI, Azure, Cohere, Anthropic, HuggingFace, Replicate, Stable Diffusion).
✅ Seamless Integration with Universal API
✅ Reliable LLM Routing with AI Router
✅ Load balance across multiple models and providers
✅ Automatic Retries with exponential fallbacks
✅ High availability and resiliency using production-ready LLMOps
🚧 Detailed Usage Analytics
🚧 PII detection and masking
🚧 Simple & Semantic Caching for cost reduction
🚧 No Vendor lock-in Observability - Logging, monitoring and tracing
✅ Enterprise-ready with enhanced security, reliability, and scale with custom deployments support.
| Provider | Provider Name | Support | Supported Endpoints | |
|---|---|---|---|---|
 |
OpenAI | openai | ✅ | /chat/completions, /chat/completions:stream |
 |
Groq | groq | ✅ | /chat/completions, /chat/completions:stream |
 |
Anyscale | anyscale | ✅ | /chat/completions |
 |
Deepinfra | deepinfra | ✅ | /chat/completions |
 |
Together AI | togetherai | ✅ | /chat/completions |
Not supported (yet): images, audio, files, fine-tunes, moderations
AI gateway can be intall on macOS, Windows, Linux, OpenBSD, FreeBSD, and on any machine
Download the appropriate version for your platform from releases page. Once downloaded, the binary can be run from anywhere. Ideally, you should install it somewhere in your PATH for easy use. /usr/local/bin is the most probable location.
gateway is available via a Homebrew Tap, and as downloadable binary from the releases page:
brew install missingstudio/tap/gatewayTo upgrade to the latest version:
brew upgrade gatewaygateway is available as downloadable binaries from the releases page. Download the .deb or .rpm from the releases page and install with sudo dpkg -i and sudo rpm -i respectively.
gateway is available via scoop, and as a downloadable binary from the releases page:
scoop bucket add gateway https://github.com/missingstudio/scoop-bucket.gitTo upgrade to the latest version:
scoop update gatewayWe provide ready to use Docker container images. To pull the latest image:
docker pull missingstudio/gateway:latestTo pull a specific version:
docker pull missingstudio/gateway:v0.0.1To start missing studio AI gateway, simply run the following command:
make upYour AI Gateway is now running on http://localhost:8080 💥
Let's make a chat completion request to OpenAI through the AI Gateway using both REST and gRPC protocols
curl \
--header "Content-Type: application/json" \
--header "x-ms-provider: openai" \
--header "Authorization: Bearer {{OPENAI_API_KEY}}" \
--data '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"who are you?"}]}' \
http://localhost:8080/v1/chat/completionsgrpcurl \
-d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"hi"}]}' \
-H 'x-ms-provider: openai' \
-H 'Authorization: Bearer {{OPENAI_API_KEY}}' \
-plaintext localhost:8080 llm.v1.LLMService.ChatCompletionsAI studio is an open-source project, and contributions are welcome. If you want to contribute, you can create new features, fix bugs, or improve the infrastructure.
It's still very early days for this so your mileage will vary here and lots of things will break. But almost any contribution will be beneficial at this point. Check the current Issues to see where you can jump in!
If you've got an improvement, just send in a pull request!
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'feat(module): add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
If you've got feature ideas, simply open a new issues!
Please refer to the CONTRIBUTING.md file in the repository for more information on how to contribute.
AI Studio is Apache 2.0 licensed.