15 posts tagged with "tool-calling"

Ask an agent team what their tool-call success rate is and you will get an answer. Ask them what their tool-hallucination rate is and the room goes quiet. Most teams do not track it, and the ones who do usually only count the catastrophic version — a function name that does not exist in the catalog — while the quieter, more expensive variants travel through production unmetered.

A hallucinated tool call is not only when the model invents delete_orphaned_users(older_than="30d") and your dispatcher throws ToolNotFoundError. That is the easy case. The harder case is when the fabricated call shadows into an adjacent real tool through fuzzy matching, or when the tool name is correct but the agent invents an argument your schema happily accepts because you marked it optional. Both of those pass your "did a tool call succeed" dashboard. Neither is what the user asked for.

Your agent passes every unit test, handles the happy path beautifully, and then one Tuesday afternoon it tries to call get_user_preferences_v2 — a function that has never existed in your codebase. The call looks syntactically perfect. The parameters are reasonable. The only problem: your agent fabricated the entire thing.

This is the phantom tool call — a hallucination that doesn't manifest as wrong text but as a wrong action. Unlike a hallucinated fact that a human might catch during review, a phantom tool call hits your runtime, throws a cryptic ToolNotFoundError, and derails a multi-step workflow that was otherwise running fine.

Every agent demo starts with three tools. A web search, a calculator, maybe a code executor. The agent nails it every time. So you ship it, and your team starts adding integrations — Slack, Jira, GitHub, email, database queries, internal APIs. Six months later, your agent has 150 tools and picks the wrong one 40% of the time.

This is the tool explosion problem, and it's one of the least discussed failure modes in production agent systems. The degradation isn't linear — it's a cliff. An agent that's 95% accurate with 5 tools can drop below 30% accuracy when you hand it 100, even if the model and prompts haven't changed at all.