I'm rebuilding self-hosted observability on EKS for six internal teams, each with its own AWS account, logs, and metrics. Our previous ECS deployment used one complete stack per team and became expensive, although the main problem turned out to be oversized instances and unnecessarily high autoscaling minimums rather than the topology itself.
The current plan is to run Loki and Mimir in microservices mode per team, with multitenancy disabled so each stack effectively serves one tenant. Tempo would be added only for teams that request tracing, while one shared Grafana instance would query all backends. Argo CD would manage the fleet from Git.
Each Loki/Mimir pair needs roughly 30 pods, 3.85 vCPUs, and 11.4 GiB of memory. Across six teams that is about 180 pods, 23 vCPUs, and 68 GiB of memory. Both systems would use replication factor 3, seven-day retention, and fixed replica counts on an arm64 Karpenter pool across two availability zones. At this size, pod count becomes the limiting factor before CPU or memory.
The concern is that every stack has a substantial baseline of components—compactors, rulers, index gateways, query schedulers, and so on—even when a team only stores about 1.2 GiB of logs. Reducing replicas only cuts the footprint by around 20%, so it seems that running fewer stacks may save more than trying to make each stack smaller.
I'd appreciate practical feedback on whether six isolated stacks are a mistake compared with one shared multi-tenant deployment. In particular: how much day-to-day work do tenant limits and shuffle sharding add in a unified stack; whether isolated stacks become painful at six or only at much larger numbers; how people approach Mimir's Kafka-based architecture instead of deprecated classic ingesters; how they automate a growing set of Grafana datasources; whether replication factor 3 is justified in non-production; and whether arm64 has caused runtime surprises across the Loki, Mimir, Tempo, and Grafana ecosystem. I'm looking for candid advice before committing to an architecture that may need to be dismantled later.
5 Answers
Replication factor 3 for non-production is usually hard to justify at this data volume unless you specifically need to test quorum behavior or survive a node loss during rolling changes. RF2 on only two ingesters leaves no room for maintenance, so adding a third ingester is understandable, but you could instead use fewer replicas or a deliberately reduced-fidelity development profile. Make the production-like profile opt-in for resilience testing rather than paying that baseline everywhere.
The operational concern is larger than the raw compute bill. A single EKS cluster hosting the telemetry system is also a failure-domain problem: if the cluster has a serious outage, the observability used to diagnose it disappears too. If the budget permits, compare this design with a managed service or at least an independent monitoring plane. If self-hosting is mandatory, a simpler single-binary or lower-component-count backend may be a better fit than deploying two full microservice systems per team.
For a shared Grafana instance, automate datasource creation rather than maintaining generated files by hand. A Grafana operator with datasource resources reconciled from the same team inventory used by Argo CD is a clean Kubernetes-native option. Terraform or OpenTofu also works well for the infrastructure and datasource layer, while dashboards can remain Git-managed. The important part is having one source of truth for teams, endpoints, credentials, and permissions so a new team creates its datasources automatically.
Be careful with tracing isolation. If services owned by different teams call one another, separate Tempo instances make cross-team traces difficult or impossible to follow. A shared tracing backend with tenant and access controls may provide better visibility, while particularly sensitive telemetry can be isolated separately. Also, image manifests proving arm64 support are only the first check; test the actual collectors, storage clients, compression libraries, and Helm hooks on arm64 before treating the platform as production-ready.
At roughly 1.2 GiB of logs per team, this looks like six copies of control-plane overhead rather than meaningful isolation. I’d keep the AWS-account and IAM boundaries, but strongly consider sharing the Loki and Mimir data plane with explicit tenant limits and resource controls. Separate stacks make more sense when teams need different recovery boundaries, strict security isolation, independent upgrades, or protection for sensitive data such as PII. Otherwise every team gets its own compactor, ruler, scheduler, and failure modes.
The account boundaries are useful, but the data plane is where you’re currently paying for six copies of almost nothing.
Before changing products, I’d confirm the actual requirements. Multiple retention policies, security boundaries, and query behavior can matter more than the headline pod count.