Page 4
12 articlesThe Model Rollout Flag That Bucketed by Session and Drifted Your A/B Cohort
A 4 percent satisfaction lift turned into a confound when a flag SDK cached per session, a ramp rotated the hash salt, and the analysis grouped by event-time flag value. Here is how cohort drift hides inside a working pipeline — and the disciplines that close the gap.
The Nightly Batch That Starved Your Interactive Traffic After a Quota Window Rewrite
A nightly LLM batch ran clean for ten months until a provider rewrote how its daily window was counted — turning a 00:05 UTC cron into a self-inflicted 429 storm on interactive traffic. Why workload segregation, jitter, and bucket-semantics contract tests are the structural fix.
The OAuth Scope One Tool Requested That Every Other Tool Quietly Inherited
How a per-provider OAuth token store silently grants every tool the union of every other tool's scopes — and the storage-key fix that puts the boundary back where the design said it was.
The OAuth Scope Your Agent Inherited When On-Behalf-Of Quietly Became Act-As
An AI agent holding a user-shaped OAuth token is impersonating the user, no matter what the security review called it. How prompt injection turns the gap into an incident — and the token-level patterns that fix it.
The On-Call Rotation That Muted LLM Pages Because Every One Looked Like The Last One
Alert fatigue on LLM features creeps in as pages that look identical produce no action, and the silence rule that follows is the rational adaptation that breaks production detection.
The OpenTelemetry Tail Sampler That Dropped Exactly the LLM Spans Your Post-Mortem Needed
Tail-based sampling was tuned for a request-response world where 200 OK and worth keeping were close enough to be the same thing. LLM systems break that contract — and the trace you need is the one your sampler is configured to drop.
The Persona Your System Prompt Offered That the Model Picked the Same Way Every Time
When your system prompt asks the model to choose between personas, the model's training prior decides — collapsing your A/B test arms before the experiment platform notices.
The PII Redactor That Protected Your Logs and Let the Model Leak the Outputs
An input-only PII redactor is half a control. Once the model can write, the outbound path becomes the exfiltration surface your review never named.
The PII Redactor That Scrubbed the User's Question and Left the Prompt Cache Untouched
A redactor that protects the analytics pipeline does nothing for the prompt cache on the inference path — and that uninventoried cache is where the next retention breach hides.
The Pinned Dependency Your Security Agent Upgraded Past the Comment It Could Not See
Two AI coding agents shipped an off-by-one date bug to European customers because neither knew the other existed. A field guide to multi-agent coordination as a distributed-systems problem in the repository.
The Presigned URL That Expired Before Your User Could Verify the Multimodal Model's Claim
When the asset clock and the claim clock run at different speeds, every correctly-grounded multimodal answer eventually looks like fabrication to whoever revisits the conversation.
The Prompt Engineer Who Quietly Became Your Only Eval Set Reader
Your eval set is a file. The judgment that makes it readable lives in one engineer's head — here is how that bus factor shows up only after they leave.
About Tian Pan
I'm Tian Pan, an engineer-founder focused on agentic engineering — building autonomous AI systems and scaling engineering teams. I write practical guides on system design, technical leadership, and shipping with AI agents. Previously an early engineer at Uber, Brex, and IoTeX.
