We recently faced a frustrating outage because a third-party API key expired unexpectedly. Despite knowing about the expiration and having it documented in a spreadsheet, the person responsible was on leave and no one else checked it. We use UptimeRobot to monitor public SSL certs, which works great, but we're struggling with monitoring offline items like Apple Push certs, API tokens, and internal signing keys. I'm curious about how other teams manage these expirations. Do you have a specific tool for tracking them, or is it just a shared spreadsheet and calendar reminders, hoping nothing slips through the cracks? I'm considering building a cron job for reminders, but it seems like a temporary fix. What solutions have you found effective?
5 Answers
What works for us is setting calendar appointments at 31, 14, 7, and 1 day before an expiration in our shared calendar. This way, multiple people stay informed about upcoming issues. We also have a paid service that opens tickets 30 days prior for cert renewals. It really helps with accountability if we include detailed instructions on what needs to be done.
We rely on shared calendars for reminders, which works well for our team. It keeps everyone in the loop. The key challenge is ensuring everything that needs monitoring gets added to that calendar promptly. Have you found that any key expirations consistently slip through the cracks?
For unscannable items, I suggest using a password manager with expiration dates. You can set up a script that integrates your monitoring system with the secrets manager to check expiration dates automatically. This way, you're alerted ahead of time, rather than relying on manual checks.
That makes sense. Establishing a system that automatically alerts for expiring API keys sounds much safer. Have you had success implementing that?
From my experience, having a monitoring system is essential. We track all expiring items in our CMDB, and tickets are generated automatically when something is about to expire. This way, no single person is responsible for tracking these expirations, even though it does rely on process rather than memory. It’s a big improvement over relying on people to remember.
We've typically used spreadsheets alongside email reminders for this. It's crucial to add multiple people to the reminder list and set several alerts ahead of the expiration date. One strategy we're working on is to enhance our automation by setting up automatic emails that generate tickets for upcoming expirations. It sounds promising, but we're still testing its utility.
That approach is pretty common. Adding multiple reminders helps, but it often falters when team responsibilities shift. I’m curious how well the automated email-to-ticket system works once it's fully implemented.
That’s a solid approach! Calendar invites definitely help keep everyone on the same page, but have you ever had issues with people assuming others will handle it?