How should we build a Terraform-based observability platform for a large hybrid estate?

Be careful not to assume that centralizing every observability detail will scale. The platform team should own collectors, integrations, common telemetry conventions, security controls, and baseline monitors. Service teams should own instrumentation correctness, useful business metrics, service-level objectives, and alerts that require domain knowledge. OpenTelemetry can provide a useful portability layer for traces and metrics, but it does not eliminate the need for ownership, semantic conventions, upgrade testing, or cost controls.

Treat this as a platform product rather than a dashboard-cleanup exercise. Build a paved path where a service can declare a small amount of configuration and automatically receive standard tags, ownership metadata, baseline monitors, log collection, cost attribution, and runbook links. Start with a required tag and ownership schema, inventory the exceptions, and migrate one service category at a time. The compliant path needs to be easier than creating a custom setup. Every alert should have a clear owner, service identifier, and runbook; otherwise it probably should not page someone.

Before building a large internal system, compare the operational and licensing costs of your current setup with managed alternatives. In AWS environments, managed scraping and managed telemetry services can remove collector-management and configuration-drift problems, while Terraform modules and official deployment accelerators can make the setup reproducible. Prometheus-compatible metrics, OpenTelemetry instrumentation, and provider-neutral metadata can reduce lock-in, but self-hosting Grafana, Prometheus, or Loki still creates a substantial maintenance obligation. Run the numbers using your actual host, log, metric, and trace volumes instead of assuming either vendor tooling or open source will automatically be cheaper.

Terraform modules gave us the biggest improvement for agent deployment and core log configuration. Pin module and agent versions, keep configurations in version control, test them in a representative environment, and roll them out through a controlled pipeline. Centralize the defaults and guardrails, but allow narrowly defined extension points so application teams can add service-specific logs or monitors without forking the platform. Policy checks can catch missing tags, unowned monitors, unsupported versions, and invalid configurations before deployment.