Hey everyone! This year, I've started managing several MSK clusters for different teams. Each cluster has multiple topics with different configurations, and I'm finding it increasingly difficult to keep everything organized as these clusters become more complex. Right now, I'm using a bastion EC2 host to connect via IAM for sending Kafka commands, but it's turning into quite the hassle. Every time I need to create or modify a topic or add ACLs, I end up copy-pasting commands repeatedly.
I've looked into a few Docker images and UI tools, but most seem outdated. I'm wondering if anyone has experience or recommendations for tools that can help? Ideally, I'm looking for something that can run in ECS with full access to the cluster using task roles instead of SCRAM authentication.
4 Answers
We utilize Terraform for managing our MSK, and we have AKHQ deployed, which our developers can use to view and manage topics as needed. It really makes things easier!
I hear you—managing MSK can get tricky as it scales. A few people in my circle have had success using Terraform or Pulumi along with open-source tools like Kafka UI from Provectus or AKHQ. These can run in ECS and, with a bit of configuration, support IAM auth. For automating topics and ACLs, using Terraform's Confluent provider or scripting with kafka-python can really cut down on the manual workload. If you're juggling multiple clusters, consider standardizing your configs in code and treating topic changes like infrastructure changes.
You might also want to take a look at the Strimzi operator, where you can define all your topic configurations in git, managed by ArgoCD. Another option could be setting up an API for handling topic requests or modifications.
We’ve set up a purpose-built CI agent pool to manage Terraform applies, and it works great! Check out this [terraform-provider-kafka](https://github.com/Mongey/terraform-provider-kafka) for an example.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures