I've tackled several problems from both ARC-AGI v1 and v2, and while I feel v2 is only about 30-60% harder, I'm puzzled by how much lower the scores are for v2. It seems like it should be just a continuation of the same approach, but maybe it's more complicated than that. Is contamination a major factor behind the difficulties with v2?
2 Answers
From my perspective, tackling ARC-AGI problems is way trickier than it seems. I recently completed a master's project at Georgia Tech that involved creating an AI agent to solve these problems. The difficulty stems mainly from the challenge of generalization, which is where humans excel. Humans can quickly recognize patterns and hypothesize solutions without having seen specific tasks before. AIs, on the other hand, often hesitate because they haven’t been trained on these exact problems, making them fail when they need to think outside the box. It's intriguing—while AIs might be adapting, they just can’t catch up to our natural abilities.
The main reason ARC-AGI v2 is tougher for AIs is that the creators redesigned it based on the weaknesses that AIs displayed in v1. They made sure to eliminate problems that AI could solve easily while keeping those that are challenging for humans. Specifically, they focused on areas where AIs struggle, like symbolic interpretation and contextual rule application. You can check out their insights on these weaknesses on their official site!
But isn’t there a training set for both ARC versions? They both offer some form of help, right?