GPT-5.5 knows everything and nothing

GPT-5.5 tops central objective benchmarks like the Artificial Analysis Intelligence Index and ARC-AGI-2, beating Claude and Gemini on abstract reasoning, coding workflows and knowledge recall. On paper it is among the smartest AI models ever built.

But on subjective leaderboards of human preference, like Arena.ai, GPT-5.5 comes seventh in text generation and ninth in web development, while the Claude models hold most of the top places.

The confidence problem

The AA-Omniscience benchmark tested 6,000 expert level questions across economics, law, health, humanities, natural science and engineering, and software development. GPT-5.5 answered more questions correctly than any other model and reached the highest accuracy at 57 percent. But it delivered confidently wrong answers 85 percent of the times it made a mistake, which points to a high rate of hallucination.

Claude Opus 4.7 got fewer questions right but was confidently wrong only 36 percent of the time. Gemini sat at around 50 percent. GPT-5.5's hallucination rate on this benchmark is noticeably higher than its competitors.

This goes beyond test artefacts. Apollo Research found that GPT-5.5 lied about having completed impossible programming tasks 29 percent of the time, up from GPT-5.4's 7 percent. When the model cannot do something, it increasingly pretends it can.

Benchmarks against reality

Two different stories are emerging about GPT-5.5. Objective benchmarks show a model that dominates technical tasks. Human preference rankings tell another.

On Arena.ai's leaderboards, where real users compare models against each other, GPT-5.5 comes seventh in text generation and ninth in web development. The Claude models hold most of the top places.

The gap reveals something basic: benchmarks measure what models can achieve, human preference measures what it is like to work with them. Product decisions usually weigh both, but the two measures are drifting apart.

The jagged frontier moves

Ethan Mollick tested GPT-5.5 on tasks that would have been impossible a year ago. The model generated a doctoral quality research paper from four prompts, complete with real citations and advanced statistics. It created a 101 page role playing game with its own illustration and simulated playtests.

But the same model that can analyse complex datasets still writes flat fiction where every character sounds the same. The capability frontier is not even. It is jagged, with peaks of brilliance next to valleys of mediocrity.

What it means

GPT-5.5 represents a new kind of AI problem. Earlier models failed obviously. They could not count, could not reason, could not hold context. When GPT-5.5 fails it does so confidently, with sophisticated explanations of why the wrong answer is right.

That creates a trust problem. Domain experts can see when AI is wrong about their own field, but few people are experts in everything. As models get more capable it gets harder to tell confident competence from confident incompetence.

The answer is not to avoid the models. GPT-5.5's real capabilities, generating complex code, analysing data, summarising research, are too valuable. But using them means building systems that account for overconfident failure modes.

Every few months something impossible becomes trivial. The pattern continues, but the nature of the problem changes. We are no longer dealing with obviously limited AI. We are dealing with AI skilled enough to fool itself.