I've noticed that Google has been offering a 1 million token context since the launch of their G2.5 model last year, and it seems like a game-changer. I'm curious why OpenAI and Anthropic haven't ramped up to that scale. Is it all about GPU versus TPU limitations? If so, what's stopping OpenAI from just purchasing more GPUs? They have the financial resources for it.
5 Answers
The main issue is actually related to manufacturing and supply chain logistics. Ordering GPUs in large quantities takes a long time, often years, and right now, companies like Elon Musk's have massive GPU orders that monopolize the supply. Google had the infrastructure set up ahead of time, while OpenAI is still in the building stage.
Exactly! Even with access to the hardware, the trade-offs in quality can be a big reason to hold back.
Cost and availability are the big blockers here. Increasing context size exponentially boosts memory needs, which means they need more GPUs. Plus, companies like Google have developed newer methods that use less memory but need faster GPUs, forcing others to figure out how to catch up while keeping costs manageable.
Totally makes sense. They’re trying to attract users while balancing the budget; it’s a tough game.
Right! They want to keep the lights on without burning a hole in their pockets.
I suspect Google's architecture is more advanced compared to OAI's. Google might be employing a newer type of transformer model that allows for larger contexts without the same drawbacks. This puts them ahead in the game.
Agreed! It’s like they’re playing a different game with better pieces.
Exactly, it really seems like Google is leveraging newer technologies better than OAI and Anthropic.
Inference costs are a major concern too. Even though OAI's third version has high performance, they seem to be short on GPUs right now, which affected how they handle the features of ChatGPT. They're really aiming to keep their main user base from being overloaded with just a few high-end features.
And that makes sense! With so many users, they have to be strategic about how they allocate hardware.
Yes! It's tough to balance between performance and what the average user needs.
Another thing to consider is the GPU supply chain. Even if OAI had the funds, if the demand is higher than the supply, they can't just snap their fingers and have more GPUs available. Nvidia is a crucial player, and if they stall, it stalls the entire industry.
Totally! And it also highlights the risks of having so much reliance on one supplier.
Exactly! It’s just like any other market, supply is king.
But don’t you think the performance of 1 million context tokens also plays a role? Other models like LLAMA 4 offer cheaper options, but they still avoid that size due to performance issues.