I'm managing a mixed infrastructure where most of our resources are on cloud X, but we also have some on-prem servers that I'd like to monitor. My plan is to run Prometheus on these remote servers and upload the data to a cloud storage occasionally. I want to know if there's a way to import this data into our central Prometheus server. Is this feasible, and are there any tools that could assist with this setup?
3 Answers
We actually do something similar by implementing a Thanos sidecar in our on-prem deployment, which integrates with Thanos in our central setup. It works pretty smoothly for our needs.
Another option is to use vmagent to scrape metrics from your Prometheus-compatible targets on-prem and send them to a centralized database like VictoriaMetrics. It's quite effective for this purpose.
You can definitely use Prometheus federation along with the `remote_write` feature to push metrics from your on-prem servers to your central server. Just ensure you're clear on when to use each feature, as they serve slightly different purposes for data collection and transfer.
Just a heads-up, federation and `remote_write` are different features. If you’re using `remote_write`, you won’t utilize federation.
Also, consider checking out Thanos. It offers some great additional features for this type of data management, but keep in mind it may require more configuration to set up.