LLMs are improving fast, but that progress comes with a hidden cost. Armin Ronacher, creator of Flask and Jinja2, just documented a concrete and unsettling case: the latest Anthropic models are less reliable than their predecessors on certain third-party tool schemas. That’s worth taking seriously.
The Problem: LLM Models Regressing on Tool Interoperability
Ronacher was working on Pi, his personal development tool, when he noticed something off. Claude Opus 4.8 and Sonnet 5 were hallucinating extra fields during calls to Pi’s editing tool. Phantom keys like requireUnique, matchCase, forceMatchCount, and oldText2 were showing up in the arguments, even though Pi’s schema expects none of them. Pi would reject the call, and the model would have to retry.
What’s striking: older Anthropic models didn’t show this behavior at all. Opus 4.5 adapted cleanly to alternative tool schemas. With Opus 4.8 and Sonnet 5, the failure rate hit 20% on some extended agentic sessions.
Why It Happens: The Post-Training / Harness Coupling
To understand the mechanism, you need to know how tool calls actually work in LLMs. They’re not classic function calls. The model receives a transcript with the available tools, generates structured text according to a learned format, and the server interprets that text as a tool call. The whole thing rests on a learned convention, not a mechanical constraint.
Ronacher puts forward a solid hypothesis: the new Anthropic models were trained via reinforcement in an environment very close to Claude Code. Claude Code uses a flat editing schema (old_string, new_string, file_path), which is structurally quite different from Pi’s nested schema (edits[] with oldText / newText).
The result is predictable once you think it through: the model has a very strong prior on the canonical Claude Code form. Faced with a semantically similar but structurally different schema, it’s out of distribution. And because its prior is stronger than before, it pushes back harder.
Error Tolerance as a Degradation Vector
There’s a second factor, even more subtle. Claude Code is a very forgiving harness. Ronacher inspected the minified code and listed what Claude Code silently tolerates:
- Unknown fields, filtered out without error
- Parameter aliases (
old_str,old_string,pathforfile_path) - Malformed Unicode sequences, automatically repaired
<invoke>blocks leaking into visible text, with automatic retry
The direct consequence: during reinforcement, a slightly malformed call could still succeed in that environment and receive a reward. The gradient against hallucinating extra fields was weak or nonexistent. The model learned that an extra field isn’t a problem, because in its training environment, it genuinely wasn’t.
What This Means for Third-Party Harnesses
Ronacher’s conclusion is uncomfortable for anyone building agents or tools on Anthropic’s APIs. A few key points:
- Strict mode (
strict: truein the tool definition) eliminates the problem, because the server then constrains sampling to the allowed keys. But Anthropic enforces complexity limits on schemas in strict mode, which can block certain use cases. - The Codex models Ronacher tested don’t show this regression, probably because the OpenAI harness (specifically the Harmony format with in-band JSON constraints) provides more visibility into the pipeline.
- The more post-training focuses on a closed proprietary harness, the more third-party harnesses will inherit its implicit quirks, whether they know it or not.
- Adapting your tool schema to match the canonical Claude Code form can be a short-term fix, but it’s not a sound architectural solution.
Ronacher also notes that failures are highly context-sensitive: on a simple single-turn prompt, the problem doesn’t reproduce. It’s in long agentic sessions, with file-reading history, accumulated thinking blocks, and multi-line edits, that the model goes off the rails. Removing thinking blocks from the history cut the failure rate in half in his tests.
Key Takeaways
- A more powerful LLM can be more rigid on tool interoperability, not less. This is a documented paradox, not a hypothesis.
- Post-training on a closed proprietary harness (like Claude Code) creates invisible interoperability debt for third-party developers.
- The error tolerance of a training harness gets encoded into model behavior, even in contexts where that tolerance doesn’t exist.
- Anthropic’s strict mode is an effective workaround but comes with constraints. Evaluate it against your schema’s complexity.
- Monitoring for regressions on tool calls needs to become a normal part of your agent observability stack, right alongside text output quality.
If you’re building agents on LLM APIs, this kind of silent regression is exactly what classic benchmarks don’t capture. It’s the sort of topic I enjoy digging into, so feel free to reach out or follow the blog for future analysis.
Sources
- Armin Ronacher, “Better Models: Worse Tools”, lucumr.pocoo.org, July 4, 2026: https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
- Anthropic documentation on strict mode and tool use: https://docs.anthropic.com/en/docs/build-with-claude/tool-use