Why did OpenAI’s rollback of the faulty model take so long?

I think the 24-hour estimate is reasonable, mainly because these models are massive. GPT-4o has an estimated 1.8 trillion parameters! They have big, expensive clusters running all this, and they need to roll back without any interruptions. If they were to shut down everything, sure it could go faster, but they’re probably doing this gradually to avoid chaos. Plus, there’s likely some behind-the-scenes stuff going on that isn't publicly shared about their data centers.

Rolling back a large language model is way more complex than redeploying an app on Vercel. You’ve got to manage global load balancers and a multi-region setup with model sharding and GPU management, not to mention interactions between tightly coupled services. The 24 hours is about ensuring they can safely redirect traffic, backtrack without messing up user sessions, and keep everything stable. Plus, coordinating with partners can slow things down, but it’s absolutely necessary for trust at that scale.

Consider the data size too—like the LLaMA 3.1 model has around 750GB of data! GPT-4o could be even bigger, so you can imagine the time needed to manage all that data and migrate it back to the cluster-nodes. It's not just flipping a switch.

During the rollback, OpenAI still has to service a huge number of requests, much more than most small developers deal with. Their architecture must be quite complex, and transitioning to new instances while keeping the old ones operational is no small feat. Plus, considering the historical data GPT tracks, it adds another layer of complexity to the rollback process. It’s a big operation!