In a shocking twist, the world of Pokémon has become the center of a fierce debate in AI benchmarking! Last week, a viral claim took the internet by storm, alleging that Google’s latest AI, Gemini, had outpaced Anthropic’s prominent Claude model in the original Pokémon video game trilogy.

The battleground? Lavender Town. According to a Twitch stream from a developer, Gemini reached this iconic location, while Claude was left struggling at Mount Moon as of late February. But hold on—there’s more to the story!

Critics on Reddit pointed out a crucial detail: the developer behind the Gemini stream crafted a custom minimap to assist the model in recognizing vital game elements, such as cuttable trees. This nifty tool significantly minimized the need for Gemini to analyze screenshots before making gameplay decisions—giving it an unfair edge!

While some might shrug off Pokémon as a quirky AI benchmark, it actually raises important questions about how variations in testing methods can drastically skew results. Take Anthropic’s recent performance with its 3.7 Sonnet model on the SWE-bench Verified benchmark, for instance. It achieved a 62.3% accuracy score, but when they utilized a 'custom scaffold,' the accuracy soared to 70.3%!

And the competition doesn’t stop there! Meta recently revamped one of its AI models, Llama 4 Maverick, to shine on the LM Arena benchmark. Interestingly, the basic version of this model performed much worse on the same evaluation.

As the lines blur between gaming and AI, one thing is clear: the race to claim the title of most advanced AI model is more intense than ever, and Pokémon has unwittingly become a front line in this ongoing battle!