Mid-July 2026, a strange word suddenly popped up in every tech conversation: pelicanmaxxing. Coined by developer Dylan Castillo, the term instantly went viral on Hacker News, racking up over 400 points in just a few hours. The question it raises is as simple as it is uncomfortable: are major AI labs spending their time biasing their models to perform well on anecdotal but highly publicized benchmarks? In a climate where Satya Nadella is warning enterprises about labs capturing their know-how (Fortune, July 16, 2026) and The Economist is documenting the Chinese threat to American labs, distrust is on the rise. Pelicanmaxxing crystallizes that unease. But do the facts live up to the hype?
What is pelicanmaxxing, exactly?
The term is a mashup of “pelican” and “benchmaxxing.” The latter refers to a practice well known among model evaluators: optimizing an LLM to excel on a specific benchmark, at the expense of its real-world performance. It also goes by “benchmark hacking” or “teaching to the test.”
Pelicanmaxxing applies that same logic to one very particular benchmark: a model’s ability to generate an SVG of a pelican riding a bicycle. This test, popularized by developer Simon Willison, has become a ritual every time a new model drops. The prompt is always the same: “Generate an SVG of a pelican riding a bicycle.”
If a lab specifically trains its model to nail this drawing, it could win over the tech community without actually making meaningful progress. Dylan Castillo decided to find out for real.
A cult benchmark, almost by accident
Simon Willison has been testing LLMs with this prompt for several years. At every major launch (GPT-5, Claude 4, Gemini 3, etc.), he publishes the result. On Hacker News, his comments often rank among the most upvoted.
The benchmark is anything but scientific. Yet it has become an informal gauge of a model’s visual quality and creativity. The reasoning is straightforward: if an LLM can draw a pelican on a bicycle in SVG, it likely masters spatial composition, proportions, and vector syntax. Failure, on the other hand, is instantly obvious.
With billions of dollars at stake and guaranteed media coverage, the temptation to optimize for this test seems obvious. Hence the pelicanmaxxing suspicion.
The experiment: 1008 images, 7 models, a solid protocol
Dylan Castillo designed a comparative test of remarkable rigor. He cross-referenced 8 animals (including the pelican) with 6 vehicles (including the bicycle), for a total of 48 combinations. The original prompt is just one cell among those 48.
Here are the models tested via OpenRouter:
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.5 Flash
- Grok 4.5
- Qwen3.7-Max
- GLM-5.2
- DeepSeek V4 Pro
Each combination was generated 3 times, at temperature 1.0, with the same reasoning effort level requested from each model. Total: 1008 SVG images.
The evaluation pipeline has three stages. First, each SVG is converted to PNG to check that it renders correctly (only 11 failures out of 1008, a remarkable rate). Second, an LLM judge (GPT-5.6 Luna) assigns a score from 1 to 5 for the animal, the vehicle, and the coherence of the action. Finally, an extractor (Gemini 3.1 Flash-Lite) analyzes the scene elements: subject orientation, accessories, background.
The hypothesis is as follows: if a lab is pelicanmaxxing, it should show up on the “pelican” row, the “bicycle” column, or the “pelican + bicycle” cell compared to the other combinations.
Verdict: no pelicanmaxxing in sight
The results, published in full on Castillo’s blog and on GitHub, are clear. None of the seven models tested shows convincing signs of targeted optimization. Five key takeaways emerge.
1. Pelicans are not drawn any better than other animals. Out of the 8 species tested, the pelican ranks 6th in average score, behind the cat, the whale, the raccoon, the heron, and the antelope. If labs were training on this benchmark, pelicans would be at the top.
2. Bicycles score poorly. Among the 6 vehicles, the bicycle ranks second to last, just ahead of the airplane. The judge reports misaligned wheels or missing pedals in two-thirds of the images. A lab optimizing for this vehicle would get better scores.
3. The pelican + bicycle combo ranks 42nd out of 48. Even without adjusting for difficulty, the benchmark’s star prompt sits near the very bottom of the leaderboard.
4. No statistically significant effect after correction. Castillo applied a fixed-effects regression to isolate the specific impact of the benchmark. Out of 21 tests (7 labs × 3 effects), only one crosses the significance threshold: Gemini 3.5 Flash on the bicycle column, with p=0.022. But with 21 tests, randomness predicts roughly one false positive. After Bonferroni correction, this signal disappears.
5. The pelican always faces right, but that is not a smoking gun. All pelican + bicycle images (21 out of 21) show the animal facing right. That might seem suspicious. Yet 60% of the 1008 images follow that same orientation. Across the whole experiment, three other combinations reach 90% or more rightward orientation. A 21/21 out of 48 combinations is nothing out of the ordinary.
What the experiment cannot detect
Two important limitations are worth highlighting. First, the judge belongs to the same family as one of the competitors (GPT-5.6 Luna scoring GPT-5.6 Terra). Castillo anticipates the objection: since all models are evaluated across all 48 combinations, any judge bias would lift the entire grid for the favored model, without creating a false positive on the pelican + bicycle cell.
Second, the experiment cannot detect what Castillo calls “SVGmaxxing”: a global optimization of SVG generation across all combinations. Google DeepMind, for instance, openly claims this type of improvement. A lab that SVGmaxxes would be better everywhere, with no visible gap between the pelican and the other animals.
Key takeaways
- Pelicanmaxxing is a viral concept that questions the integrity of informal benchmarks in AI.
- Dylan Castillo’s experiment, conducted across 1008 images and 7 models, finds no evidence of targeted cheating on the “pelican on a bike” prompt.
- Labs do not draw pelicans any better than other animals, nor bicycles any better than other vehicles.
- The only intriguing result (all pelicans facing right) is explained by natural image composition constraints.
- Global SVG generation optimizations remain possible and undetectable by this protocol.
Got an opinion on the topic or a similar experiment to share? Drop by on LinkedIn or leave a comment, I am always happy to chat about these subjects. And if this kind of analysis interests you, the blog regularly publishes deep dives on security, DevOps, and the AI ecosystem.
Sources
- Are AI labs pelicanmaxxing ?, Dylan Castillo (original article, July 2026)
- Fortune, Satya Nadella warns enterprises that AI labs are stealing their know-how (July 16, 2026)
- The Economist, America’s AI labs are under threat from cheap Chinese rivals (July 21, 2026)
- Nextgov/FCW, US AI labs are emerging target for foreign spies (July 21, 2026)
- Experiment code and data, GitHub