Should I run one Loki and Mimir stack per team, or consolidate into a shared platform?
Asked by
MapleQuill47
I'm rebuilding self-hosted observability on EKS for six internal teams, each with its own AWS account, logs, and metrics. Our previous ECS deployment used one dedicated observability stack per team and became expensive, but the real cause appeared to be oversized instances and unnecessarily high autoscaling minimums rather than the topology itself.
The current plan is to deploy Loki and Mimir in microservices mode for each team, with multitenancy disabled so each stack represents one implicit tenant. Tempo would be deployed only for teams that request tracing. Grafana would be shared, while Argo CD manages the fleet from Git.
Each Loki/Mimir pair needs roughly 30 pods, 3.85 vCPU, and 11.4 GiB of memory. Across six teams that is about 180 pods, 23 vCPU, and 68 GiB of memory. Both systems use replication factor 3 and seven-day retention. The cluster would run on arm64 nodes provisioned by Karpenter across two availability zones. At this size, pod count is the limiting factor rather than CPU or memory.
The concern is that every stack has a substantial fixed cost regardless of traffic. One team currently stores only about 1.2 GiB of logs, yet still needs components such as a compactor, ruler, index gateway, query scheduler, and rollout-related services. Reducing replicas only cuts the footprint by about 20 percent, so it seems that running fewer stacks may be the only meaningful way to reduce cost and operational work.
I'd appreciate practical experience with both designs. How painful are per-tenant limits and shuffle sharding in one shared Loki/Mimir deployment? At what number of isolated stacks does the operational overhead become noticeable? For Mimir, has anyone handled the move from classic ingesters to the newer ingest-storage architecture, or used the bundled Kafka setup outside of production? What is the cleanest way to automate Grafana datasources for three backends per team as the team list changes? Is replication factor 3 across three ingesters reasonable for non-production, or excessive? Finally, have arm64 deployments of the Loki, Mimir, Tempo, Grafana, and agent ecosystem produced runtime surprises despite publishing multi-architecture images?
Answers
A separate Tempo instance per team can create a visibility problem when services call across team boundaries. A trace may cross several teams, so decide whether traces should be queried centrally or whether there is a deliberate boundary between them. Logs and metrics can sometimes be isolated more easily than distributed traces; deploying Tempo independently just because the other systems are independent may make cross-team troubleshooting harder.
Reply to this answer
For a shared Grafana instance, use declarative provisioning rather than hand-maintained datasource files. A Grafana Operator with datasource custom resources, reconciled through Argo CD, is a reasonable Kubernetes-native approach. Terraform or OpenTofu also works well if the team inventory is already managed there. Generate datasource definitions from the same team registry that creates the Loki, Mimir, and Tempo resources so additions and removals happen together. Whichever tool you choose, make stable datasource UIDs and names part of the configuration so dashboards don’t break when endpoints change.
Reply to this answer
At roughly 1.2 GiB of logs per team, this looks like six copies of control-plane overhead rather than meaningful isolation. Keep the AWS-account and IAM boundaries, but strongly consider sharing the Loki and Mimir data plane with tenant limits. A separate stack is easier to justify for sensitive data, unusual recovery requirements, or a genuinely different blast-radius boundary. The old ECS bill demonstrates that the previous sizing was bad; it doesn’t by itself prove that six compactors and six sets of supporting components are a good design.
SatinPebble6: The account boundaries still make sense, but the expensive part is duplicating the observability data plane for six teams that currently produce very little data.
Reply to this answer
The strongest argument against six stacks is not storage cost; it is the repeated operational surface: upgrades, alerts, capacity planning, Kafka or ingest-storage decisions, compaction, backups, and failure testing. A shared stack with explicit tenant quotas and limits usually gives you one place to operate and makes spare capacity useful across teams. Keep isolation only where it provides a concrete security, compliance, recovery, or blast-radius benefit.
One separate concern is that running the observability system in the same EKS failure domain as the workloads means a cluster-wide outage can take out both the applications and their telemetry. If the telemetry must remain available during that kind of incident, consider a separate cluster or a managed service rather than spending the budget on duplicated small stacks.
Reply to this answer
Replication factor 3 in non-production is defensible when you need rolling-restart tolerance and want failures to resemble production, but it is not free. If the environment is disposable and losing telemetry is acceptable, lower replication or fewer replicas can save a lot of compute. The important question is whether non-production is validating durability and upgrade behavior or merely providing a place to test queries and dashboards. Those goals may justify different sizes.
Also, be cautious about inheriting production’s full microservices topology for very small installations. A simpler single-binary or consolidated deployment can be a better fit for low-volume team environments, while a shared production-scale Mimir/Loki platform handles workloads that actually need the extra components.
Reply to this answer
Add your answer
Answers are reviewed before publication.
GrayCedar24: If cross-team calls are common, a shared tracing backend or at least a centrally queryable tracing layer is probably more useful than strict per-team isolation.