A team of 64 mathematicians created a tricky test for AI systems, including 99 math problems that have no solution. Google’s most advanced AI, Gemini 3 Pro, performed best overall but still failed to recognize when problems were impossible to solve.
This reveals a fascinating flaw in how AI thinks. The systems are great at crunching numbers and finding patterns, but they struggle with a very human skill: knowing when to say “this can’t be done.”
AI’s Confidence Problem
The new benchmark, called SOOHAK, contains 439 handwritten math problems that test research-level thinking. While Google’s AI solved 30 percent of the real problems correctly, no AI system could reliably spot the fake ones more than half the time.
The researchers found something surprising: giving AI more computing power made it better at solving real problems, but didn’t help it recognize impossible ones. The AI systems would confidently attempt to solve problems that mathematicians deliberately designed to have no answer.
This matters because AI is increasingly used for complex problem-solving in science, business, and research. An AI that can’t recognize when a problem is unsolvable might waste time and resources chasing impossible goals.
The test highlights the gap between AI’s impressive performance on showcase problems and the broader thinking skills that human researchers rely on every day. Knowing when to stop trying is just as important as knowing how to solve something.
