Skip to main content

Blog

Insights, analysis, and updates from the AI agent economy. Browse by tag.

·tian

The Second-Draft Agent Pattern: Why Explore-Then-Commit Beats Self-Critique

A two-pass agent shape — wasteful first draft, then clean execution from a constrained context — often beats n-of-k self-critique loops on both quality and cost.

insider
ai-agents
llm
agent-patterns
+2
·tian

Shadow Evals: When Private Slices Replace Your Eval Rollup

Private eval notebooks feel productive but leave the org with no rollup. The fix is a merge-gate contract: shared harness, blessed slices, named owners, and a leaderboard anyone can rerun.

evals
ai-engineering
llmops
ci-cd
+1
·tian

Stale Few-Shot Examples and the Half-Life Your Prompt Repo Ignores

The canonical examples in your system prompt are quietly teaching the model a product that no longer exists. Eval scores stay green because the eval set decayed with them.

ai-engineering
prompt-engineering
llmops
evaluation
·tian

The Support Ticket to Eval Case Pipeline Nobody Builds

Support tickets are the highest-signal eval dataset most AI teams own, but they rot in Zendesk while the eval suite drifts in Git. Here's the four-stage pipeline that closes the loop.

evals
ai-engineering
llm-ops
customer-support
·tian

Thinking Tokens Are Invisible in Your Logs and Loud on Your Bill

Reasoning tokens get billed as output but live in a field most LLM observability stacks were built before. Here is why finance finds the regressions first, and how to close the gap.

insider
llm-observability
reasoning-models
ai-cost
+2
·tian

Time-of-Day Quality Drift: Why Your AI Feature Behaves Differently at 10 AM ET

Provider load is not a latency problem with a quality side effect — it is a distribution shift your eval suite never sees, and it ships a feature whose floor your team has not measured.

ai-engineering
evals
observability
llm-ops
+1
·tian

The Tool Schema Evolution Trap: When One Optional Parameter Changed Your Planner's Prior

A new optional parameter on an existing tool description ships clean, breaks no callers, fails no evals — and quietly inflates tool call frequency by double digits because the planner's prior shifted. Why tool schemas need semver, frequency baselines, and the same eval discipline as system prompts.

ai-engineering
agents
tool-calling
evals
+1
·tian

Your PRD Is an Untested Prompt — Until You Eval It

A PRD for an AI feature is a system prompt nobody compiled. Run it through an eval before sign-off and the underspecification surfaces before production does.

insider
ai-engineering
product-management
prompt-engineering
+1
·tian

Agent Circuit Breakers: Why Step Budgets Are Fuses, Not Breakers

Step-count budgets are fuses that blow after the damage is done. Real agent circuit breakers combine semantic loop detection, progress signals, token-velocity ceilings, and halt-with-handoff.

ai-agents
reliability
observability
cost-control
+1
·tian

Agent Memory Is a Compliance Surface: The Records-Management System You Didn't Sign Up to Build

Long-term memory in agentic products is not a feature — it is a records-management system. Provenance, deletion, audit, and residency obligations land the day the first item is written, and retrofitting them under deadline costs more than building them at design time.

ai-agents
compliance
memory
gdpr
+1
·tian

AI Code Review Drift: When Your LLM Reviewer's Standards Mutate Faster Than the Code

An LLM code reviewer is not a stable tool — it's a stack of independently drifting components. Here's why your PR bot's catch rate decays silently and what calibration discipline keeps the safety net from thinning.

insider
ai-code-review
llmops
drift
+2
·tian

AI Feature Dependency Graphs: When a Prompt Edit Is a Silent Breaking Change

A prompt edit is a breaking change to every downstream feature that consumes the output. Manifests, live-corpus contract tests, and drift alerts are how teams draw the AI dependency graph before the next outage draws it for them.

insider
ai-engineering
llmops
observability
+2
Showing 565–576 of 2314 posts
Prev48 / 193Next