The Hidden Token Tax: How System Prompts and Tool Schemas Silently Drain Your Context Window
· 9 min read
Most teams know how many tokens their users send. Almost none know how many tokens they're spending before a user says anything at all.
In a typical production LLM pipeline, system prompts, tool schemas, chat history, safety preambles, and RAG prologues silently consume 30–60% of your context window before the actual user query arrives. For agentic systems with dozens of registered tools, that overhead can hit 45% of a 128k window — roughly 55,000 tokens — on tool definitions that may never get called. This is the hidden token tax.
