I've inherited two production OpenStack clusters used for internal development. The operating systems and OpenStack haven't been upgraded in years, the documentation is stale, and much of the knowledge sits with the original administrators.
Would you try to upgrade them or build replacements? Where should I start?
I'd seriously consider building a replacement rather than trying to upgrade a decade of neglect in place. That doesn't mean tearing anything down yet. First establish what workloads and services need to move, then price the new environment and a migration.
A small lab would give you room to learn the deployment and test failures before committing to a production design. Keep the old clusters running until the workloads have been migrated and checked, and document the replacement so its operation doesn't depend on one person.
Inventory first. OpenStack isn't one service, so find out which components are actually in use. Are these mainly Nova instances, or are Neutron networking, Cinder volumes, Swift storage, or other services involved too?
Map the workloads and their dependencies before comparing an upgrade with a rebuild. A VM migration is a different problem once you also have to preserve its networking and attached storage.
Get a written handover going with the original administrators. Ask for the deployment method, access, service dependencies, known problems, and whatever recovery procedures exist. Record gaps as you discover them, especially where the only answer is that someone knows how it works.
Then give management a proposal with the risks, estimated effort, options, and decisions you need from them. Ownership should come with time and resources to make the systems manageable.
tandem-bike-auditor:
That gives me a starting point for the handover. I'll document the gaps along with what I can verify.
Don't assume a rebuild is automatically quick or that an upgrade is impossible. Work out the installed versions and deployment method, then investigate the upgrade path and test it away from production. Compare that effort and risk with building and migrating to a replacement.
Answers are reviewed before publication.
cataloguing_cloudberries:
Budget and capacity could decide whether a parallel build is possible. I'd put those constraints in the proposal rather than assuming there's room for another cluster.