How would you handle LLM traffic from Kubernetes apps?

I’d separate configuration from enforcement. An operator and custom resources can give teams a declarative way to request providers, credentials, budgets, and policies. A central gateway can then apply those settings, keep provider keys out of individual apps, and record usage consistently. That avoids making every workload reinvent the same controls.

The tradeoff is that a central gateway becomes part of the critical path, so plan for availability, capacity, and clear failure behavior. Decide whether a gateway outage should block calls or allow a tightly limited fallback.

For a first version, I’d start with a central proxy and require workloads to send an application or team identity that the proxy can verify. That gives you one place to enforce budgets, route requests, and log which workload initiated a call. Keep provider credentials there rather than distributing them across pods.

Don’t assume the gateway can reliably identify a prompt’s origin from its contents. Tie requests to authenticated workload identity and explicit app metadata, and make missing or invalid identity a policy decision.

The sensitive-data check is the tricky part. If it must happen before egress, it needs to be on the synchronous request path, not just in an audit pipeline after the fact. A central gateway can do that, but it adds a network hop. A sidecar or mesh interception point may put enforcement nearer to the workload, though it also adds deployment and operational complexity.

I’d avoid choosing a sidecar just for the idea of proximity. First measure whether the extra hop matters, then decide where the checks belong and what happens when the policy service is unavailable.

An SDK is useful for making provider APIs easier to consume, but I wouldn’t make it the only enforcement layer if teams can call providers directly. Libraries can be skipped, misconfigured, or lag behind policy changes. Use them as a convenience, with network controls and a gateway or equivalent enforcement point providing the actual boundary.

Keep the first version narrow. Centralized keys, authenticated workload identity, per-team limits, and basic request and usage records are a solid starting set. Add routing and more sophisticated content policies once you know what information you can reliably inspect and how you want denials to behave.

Also be explicit about what gets logged. Prompt and response bodies can contain sensitive information, so useful observability does not automatically mean retaining full content.