Page 116
12 articlesBuilding LLM Evals from Sparse Annotations: You Don't Need 10,000 Examples
Most teams delay eval investment waiting for enough labeled data. The evidence shows 50–200 carefully chosen examples, built with active learning, weak supervision, and LLM-bootstrapped labeling, produce reliable signal. Here's how to build trustworthy evals before you have a large dataset.
The Few-Shot Saturation Curve: Why Adding More Examples Eventually Hurts
Adding more few-shot examples to your prompts seems like a free win — it isn't. Here's the empirical evidence for where the curve turns against you, why it happens, and what to do instead.
Fine-Tuning Dataset Provenance: The Audit Question You Can't Answer Six Months Later
Most fine-tuned production models have no reliable answer to 'where did this training example come from.' Here's the provenance registry schema and audit workflow that gives you one before the regulator asks.
Graceful AI Feature Sunset: How to Deprecate a Model-Powered Feature Without Breaking User Trust
Deprecating an AI feature isn't like removing a button — users build workflows around model personality, output structure, and behavioral quirks. A four-phase lifecycle for retiring model-powered features without triggering churn.
Grammar-Constrained Generation: The Output Reliability Technique Most Teams Skip
Constrained decoding guarantees schema-valid LLM outputs at the token level — eliminating the validate-retry loop entirely. Here's how it works, why most teams skip it, and when it actually hurts you.
Hiring for LLM Engineering: What the Interview Actually Needs to Test
Standard coding screens and ML math questions fail to predict LLM engineering success. Here's what practical interview exercises actually reveal about a candidate's ability to ship AI products.
Hot-Path vs. Cold-Path AI: The Architectural Decision That Decides Your p99
A decision framework for which AI work belongs in the request path, which belongs in a queue, and how to migrate across the boundary once traffic shape changes.
The Implicit API Contract: What Your LLM Provider Doesn't Document
LLM providers guarantee uptime and latency SLAs. They don't guarantee that your prompts will produce the same output next month. Here's what engineers need to know about the implicit behavioral contract — and how to test against it.
The Intent Classification Layer Most Agent Routers Skip
Most agent routers load every tool schema on every request and let the LLM decide. At 417 tools, that approach collapses to 20% accuracy. Here's how an intent classification layer fixes it—and why skipping it quietly destroys both accuracy and cost at scale.
Judge Model Independence: Why Your Eval Breaks When the Grader Shares Blind Spots with the Graded
Using the same model family as both product and judge inflates scores by 8–16% because they share blind spots. Here's how to build evaluation systems that actually catch what your model misses.
Keeping Synthetic Eval Data Honest
Using LLMs to generate your own test cases creates a flattering but misleading feedback loop. Here's how adversarial seeding, human annotation triage, and diversity gap analysis fix the structural blindspots synthetic evals miss.
Knowledge Graphs as a RAG Alternative: When Structured Retrieval Beats Embeddings
Vector similarity search fails silently on multi-hop queries and schema-dependent facts. Here's when a property graph with traversal queries outperforms embedding lookup — and how to build the hybrid that covers both.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
