Page 75
12 articlesSystem Prompts as Code, Config, or Data: The Architecture Decision That Cascades Into Everything
Most teams pick where their system prompt lives by accident, then fight the consequences for years. The choice between code, config, and data storage cascades into deploy cadence, eval scope, and tenant flexibility — here is the framework to apply before MVP.
The Three Tastes of an AI Engineer: Why Prompts, Evals, and Guardrails Don't Live in the Same Head
Prompt taste, eval taste, and guardrail taste are three separate intuitions that the AI engineer job title hides. Hire and promote as if they were one skill and you ship lopsided systems where every artifact is green and the user is leaving.
The Tip Jar Problem: When 5% of Your Users Burn 80% of Your Inference Budget
Flat-rate pricing for token-billed AI products produces a power-law usage distribution where a tiny minority of whales destroys margins. The standard fixes — caps, throttles, fair-use clauses — alienate the engaged users who would pay more if you let them. Here is the tier architecture, metering pre-work, and unit-economics discipline that actually fits how token costs behave.
Token Amplification: The Prompt-Injection Attack That Burns Your Bill
Most prompt-injection threat models focus on data exfiltration. The quieter attack class is bill amplification — a $0.01 request becomes a $40 inference invoice. Here is the defense discipline that stops it.
Token Budgets Are the New Internal IAM
When your AI bill crosses seven figures, token quota stops being a finance number and starts behaving like an authorization surface. Why allocation needs IAM-style discipline, not dashboard sliders.
Tokenizer Churn: The Silent Breaking Change Inside Your 'Compatible' Model Upgrade
A vendor model bump can leave the API byte-stable while quietly swapping the tokenizer underneath — silently breaking context budgets, stop sequences, and few-shot prompts. Here is how to audit, pin, and survive tokenizer churn.
The Expensive-to-Undo Tool Taxonomy: One Approval Gate Per Risk Class
Binary tool approval breaks under load: a single confirm dialog cannot gate a draft save and an outbound payment without training users to click through both. A six-class risk taxonomy fixes the conflation.
Your Tool Catalog Is a Power Law and You're Optimizing the Long Tail
Production tool usage follows a power law, but most agent frameworks treat the catalog as flat — and pay for it in token bloat, accuracy collapse past 100 tools, and silent long-tail regressions. A field guide to hot/cold partitioning.
Tool-Composition Privilege Escalation: Your Security Review Cleared the Nodes, Not the Edges
Per-tool security review clears nodes, but agents run trajectories. The composition graph of an agent's tool catalog is a permission set the security team never enumerated, and confused-deputy exploits live on the edges.
Trust Ceilings: The Autonomy Variable Your Product Team Can't See
AI agents stall at the autonomy ceiling — the level above which users start checking, intervening, or abandoning the feature. Treat it as a measurable product variable, not a model problem.
Abstain or Escalate: The Two-Threshold Problem in Confidence-Gated AI
A single confidence threshold collapses two distinct decisions — abstain and escalate — into one number, and that compromise is why your trust metric keeps sliding even when accuracy looks fine.
The Third Copy: Vector Stores, Deletion Completeness, and the GDPR Gap RAG Teams Keep Missing
When a user invokes their right to erasure, deleting the source text doesn't delete the embedding. Most teams never modeled the vector store as a third copy of user data — and the inversion-attack literature says they should have.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
