Looking for a Gateway/Proxy to Handle Spending Limits on Azure OpenAI
Asked by
TechWizard42
I'm currently using Azure OpenAI for a couple of projects and have encountered a major concern: Azure doesn't provide a built-in hard spending limit on the resource or API level that would automatically disable the service once a defined dollar amount is reached. The Budget Alerts in Azure Cost Management only notify you but don't cut off the service in real-time.
I'm searching for a self-hosted or open-source gateway or proxy that I could place between my applications and the Azure OpenAI endpoint to help manage this.
Here's what I need:
- A hard spending limit that allows me to set a maximum budget (for example, $50/month) and gives an error response (like 429 or 402) when that limit is reached.
- Compatibility with Azure OpenAI API by supporting specific headers and deployment routing.
- The ability to estimate costs in real-time based on usage (prompt and completion tokens).
- Ideally, it should be lightweight enough to run within a Docker container or as a lightweight service using Go, Node.js, or Python.
It would be great if it could also:
- Manage different budgets for various API keys or users.
- Include a simple dashboard for tracking current month-to-date spending.
- Preferably, be open source with an MIT or Apache license.
Answers
Have you looked into Litellm? It seems to fulfill quite a few of the requirements you have listed, including cost management functionalities.
Reply to this answer
You might want to check out Microsoftโs Azure API Management. It seems like a solid option for your needs. API Management can serve as the intermediary between your apps and the OpenAI endpoint, and from what I understand, it has a quota-by-key policy that can help manage spending.
BudgetBoss87: Does this allow for setting a hard spending limit? Like stopping once I exceed $1,000 in a month? Iโve looked at the docs but didnโt see that feature mentioned.
CleverCoder33: Absolutely! All you need to do is set up your API behind APIM, and youโre good to go!
Reply to this answer
Add your answer
Answers are reviewed before publication.
TechWizard42: Perfect! I'll explore that option, thanks a lot!