I need some insight from the community. We performed a systemctl stop and then a start on our RabbitMQ servers, one at a time. After restarting, we noticed that nearly 200k messages disappeared from some queues, although not all of them. All our queues are set to persistent, so we're puzzled about what could have happened to the messages. We've already tried rebooting, restarting the service, and digging through logs and data files. The servers seem to be operating normally, but we are really missing a lot of data. Any help in recovering these lost messages would be greatly appreciated!
4 Answers
Unfortunately, the data could be lost. Just setting your queues to persistent or durable isn’t enough; the client has to mark the messages as persistent too by using the right delivery mode. Make sure that's configured correctly!
You might want to check your queue configuration and the settings for the messages. There can be some complicated interactions between how messages are configured and how your queues are set up. Don't overlook that!
Thanks for the info, I will take a look at that!
It's possible that some messages were delivered but not acknowledged. You might want to check on that too.
How fast did you restart the servers? In a clustered setup, a restarted node may come back empty initially and take some time to resynchronize the messages. Also, ensure that queues have a high-availability policy; otherwise, messages won't be mirrored.
What flag? We've never had issues before, maybe this is something that's changed?