I'm currently using Azure OpenAI for a couple of projects and have encountered a major concern: Azure doesn't provide a built-in hard spending limit on the resource or API level that would automatically disable the service once a defined dollar amount is reached. The Budget Alerts in Azure Cost Management only notify you but don't cut off the service in real-time.
I'm searching for a self-hosted or open-source gateway or proxy that I could place between my applications and the Azure OpenAI endpoint to help manage this.
Here's what I need:
- A hard spending limit that allows me to set a maximum budget (for example, $50/month) and gives an error response (like 429 or 402) when that limit is reached.
- Compatibility with Azure OpenAI API by supporting specific headers and deployment routing.
- The ability to estimate costs in real-time based on usage (prompt and completion tokens).
- Ideally, it should be lightweight enough to run within a Docker container or as a lightweight service using Go, Node.js, or Python.
It would be great if it could also:
- Manage different budgets for various API keys or users.
- Include a simple dashboard for tracking current month-to-date spending.
- Preferably, be open source with an MIT or Apache license.
2 Answers
You might want to check out Microsoft’s Azure API Management. It seems like a solid option for your needs. API Management can serve as the intermediary between your apps and the OpenAI endpoint, and from what I understand, it has a quota-by-key policy that can help manage spending.
Absolutely! All you need to do is set up your API behind APIM, and you’re good to go!
Have you looked into Litellm? It seems to fulfill quite a few of the requirements you have listed, including cost management functionalities.
Perfect! I'll explore that option, thanks a lot!
Does this allow for setting a hard spending limit? Like stopping once I exceed $1,000 in a month? I’ve looked at the docs but didn’t see that feature mentioned.