We're making OpenAI and Anthropic calls from several services, but we have no reliable month-to-month view of which service or team is driving the bill until the provider invoice arrives. Vendor dashboards aren't giving us enough attribution, so we're considering better usage logging or putting a gateway in front of the APIs. What approaches have worked for tracking tokens, costs, budgets, and ownership across multiple services?
4 Answers
The quickest starting point is to log the usage fields returned by each SDK, such as prompt and completion tokens, alongside the service, team, model, timestamp, environment, and request status. Send that to whatever observability system you already use. You can also assign separate provider API keys to each service or team, which gives you basic attribution without adding infrastructure. That works well at small scale, although it gets harder to manage as the number of services grows.
Budgets should be enforced at the same level as attribution: service, team, and environment. Add soft-limit alerts and hard-limit behavior such as blocking requests or falling back to a cheaper model. Keep failed and retried requests separate from successful user-facing work so an unreliable integration doesn’t look like the biggest source of product usage. Also make sure every client uses the standard gateway or instrumentation path; teams bypassing it with direct API calls are a common source of blind spots.
Treat attribution as part of the request contract rather than something you reconstruct later. Require metadata such as service, team, environment, feature or workflow, customer or account when appropriate, and a job or run ID. Record model, prompt and completion usage, cache status, retries, latency, status code, and the upstream request ID. Keep request and response bodies out of ordinary logs unless you have a specific redaction and retention policy. Useful dashboards include cost by service and feature, cost per successful operation, retry waste, cache hit rate, model mix, and unusually expensive requests.
For several teams and providers, I’d put a gateway such as LiteLLM or another AI gateway in front of both APIs. Give every service, team, and environment its own virtual key instead of sharing one key across everything. The gateway can centralize token logging, model routing, budgets, alerts, and spend dashboards with minimal client changes. The separate virtual keys are the important part—routing all traffic through a proxy while keeping one shared key still leaves you with an undifferentiated bill.
Exactly. A gateway is useful for routing, but key-level attribution is what makes the spend breakdown actionable. I’d also reconcile the gateway’s estimates against provider exports because caching, batch pricing, and pricing changes can make token-count calculations drift.