How to Safely Store Private Keys for Firmware Signing?

I'm working on a firmware signing pipeline and looking for some advice from those who have experience in a production environment. Here's the situation: each firmware image consists of multiple files, and there's a need to sign each file individually, though I'm also open to hearing opinions about whether a signed manifest would be a better approach. Right now, we're using Jenkins for our CI/CD, but we're transitioning to GitLab. Since our devices use secure boot, it's crucial that we protect our private key because losing it could lead to unauthorized firmware deployment. I'm considering a few options for managing these keys: - Hardware Security Module (HSM), either on-premises or cloud-based. - Smart cards or USB tokens. - TPM-bound keys on a designated signing host. - Encrypted keys stored in a secrets manager (which is my least favorite option). I have a few specific questions: 1. What setup do you use for firmware signing in production? 2. Are you signing each artifact or going for a manifest approach? 3. How do you ensure signing is isolated from CI runners? 4. Any lessons learned regarding key rotation, audit tracking, or defending against pipeline attacks? 5. If you're using GitLab, are protected environments and stages enough, or do you need an extra signing service? I'm particularly mindful of the threat of supply-chain attacks and compromised CI workers; my goal is to create a robust system rather than just a convenient one. Any real-world insights or strategies that have proven effective would be greatly appreciated!