I fixed what seemed like an obvious bug in one of our services: a retry mechanism was triggering more times than intended. The deployment looked fine at first, but two days later a downstream team reported that their backfill reconciliation was failing.
It turned out that the extra retry behavior had quietly become part of their workaround for eventual consistency. There was no documentation or test explaining this dependency, so the behavior looked like a simple bug even though it was doing something important.
How do you investigate changes like this in older systems with incomplete test coverage? What steps do you take to avoid removing behavior that another service has started relying on?
4 Answers
This is basically the practical consequence of systems having more observable behavior than their official API. If clients can see it, depend on it, or build a workaround around it, they may treat it as part of the contract whether you intended that or not. I usually add a regression test for the newly discovered dependency, then work toward removing the dependency rather than immediately restoring the old behavior forever.
Before changing behavior, check request logs, endpoint metrics, and traces to see who uses the affected path and how often. In a system without much test coverage, run the service in staging with tracing enabled and replay representative production traffic. Comparing before-and-after traces can reveal downstream calls or traffic patterns that disappear after the change.
Sometimes the safest fix is not to correct the historical behavior all at once. I once found that a calculation had been producing incorrect values for years, so changing it immediately would have created a different customer problem. We had to introduce a gradual transition while communicating the behavior and deciding what the correct long-term result should be.
Roll back first if the change is causing active problems. Once things are stable, document the dependency in the code and wherever your team tracks system behavior, then create a follow-up task to replace the workaround with an intentional design. A load-bearing bug is usually a sign that the real dependency needs to be made explicit.
The documentation step is easy to skip when everyone is rushing, which is exactly how the same surprise tends to happen again later.
That gradual approach makes sense when the old output has already become part of what users or other systems expect. The hard part is making sure the temporary adjustment has an owner and an eventual removal plan.