I'm rebuilding self-hosted observability on EKS for six internal teams, each with its own AWS account, logs, and metrics. Our previous ECS deployment used one dedicated observability stack per team and became expensive, but the real cause appeared to be oversized instances and unnecessarily high autoscaling minimums rather than the topology itself.
The current plan is to deploy Loki and Mimir in microservices mode for each team, with multitenancy disabled so each stack represents one implicit tenant. Tempo would be deployed only for teams that request tracing. Grafana would be shared, while Argo CD manages the fleet from Git.
Each Loki/Mimir pair needs roughly 30 pods, 3.85 vCPU, and 11.4 GiB of memory. Across six teams that is about 180 pods, 23 vCPU, and 68 GiB of memory. Both systems use replication factor 3 and seven-day retention. The cluster would run on arm64 nodes provisioned by Karpenter across two availability zones. At this size, pod count is the limiting factor rather than CPU or memory.
The concern is that every stack has a substantial fixed cost regardless of traffic. One team currently stores only about 1.2 GiB of logs, yet still needs components such as a compactor, ruler, index gateway, query scheduler, and rollout-related services. Reducing replicas only cuts the footprint by about 20 percent, so it seems that running fewer stacks may be the only meaningful way to reduce cost and operational work.
I'd appreciate practical experience with both designs. How painful are per-tenant limits and shuffle sharding in one shared Loki/Mimir deployment? At what number of isolated stacks does the operational overhead become noticeable? For Mimir, has anyone handled the move from classic ingesters to the newer ingest-storage architecture, or used the bundled Kafka setup outside of production? What is the cleanest way to automate Grafana datasources for three backends per team as the team list changes? Is replication factor 3 across three ingesters reasonable for non-production, or excessive? Finally, have arm64 deployments of the Loki, Mimir, Tempo, Grafana, and agent ecosystem produced runtime surprises despite publishing multi-architecture images?
3 Answers
At roughly 1.2 GiB of logs per team, this looks like six copies of control-plane overhead rather than meaningful isolation. Keep the AWS-account and IAM boundaries, but strongly consider sharing the Loki and Mimir data plane with tenant limits. A separate stack is easier to justify for sensitive data, unusual recovery requirements, or a genuinely different blast-radius boundary. The old ECS bill demonstrates that the previous sizing was bad; it doesn’t by itself prove that six compactors and six sets of supporting components are a good design.
For a shared Grafana instance, use declarative provisioning rather than hand-maintained datasource files. A Grafana Operator with datasource custom resources, reconciled through Argo CD, is a reasonable Kubernetes-native approach. Terraform or OpenTofu also works well if the team inventory is already managed there. Generate datasource definitions from the same team registry that creates the Loki, Mimir, and Tempo resources so additions and removals happen together. Whichever tool you choose, make stable datasource UIDs and names part of the configuration so dashboards don’t break when endpoints change.
A separate Tempo instance per team can create a visibility problem when services call across team boundaries. A trace may cross several teams, so decide whether traces should be queried centrally or whether there is a deliberate boundary between them. Logs and metrics can sometimes be isolated more easily than distributed traces; deploying Tempo independently just because the other systems are independent may make cross-team troubleshooting harder.
If cross-team calls are common, a shared tracing backend or at least a centrally queryable tracing layer is probably more useful than strict per-team isolation.
The account boundaries still make sense, but the expensive part is duplicating the observability data plane for six teams that currently produce very little data.