2 posts tagged with "release-management"

A prompt regression lands in production at 9 AM. On the web app, an engineer rolls back the system prompt by lunch and the trace logs go quiet. On iOS, the same regression sits in the binary the App Store reviewed three weeks ago — and the team now has to choose between a server-side prompt swap that voids the store's review of the actual user-facing behavior, or an expedited review that costs 24-48 hours plus a soft favor with the platform team. Neither option is on the runbook.

This is the deploy cadence collision: web AI features iterate on the team's clock, mobile AI features iterate on the platform's clock, and most release trains were laid down before anyone thought to ask whether the prompt belongs on the same train as the binary. The result is a quietly accumulating tax — review delays, asymmetric rollback latency, undisclosed AI surfaces that fail privacy review on resubmit, and an entire class of AI bugs that mobile engineers fix at one-tenth the speed their web colleagues do.

A platform team ships their weekly agent release on a Wednesday afternoon. The internal changelog is dutiful: three system-prompt commits, a model-alias bump from a -0815 snapshot to -1019, four edits to tool descriptions, a new eval-rubric weighting, and a refreshed retriever index. By Friday, the support queue has eighteen tickets that nobody on the platform team can pattern-match. Tickets two and seven say "the bot is suddenly refusing to summarize private repos." Ticket eleven says "every code block in the output now starts with a language tag, and our downstream parser breaks on it." Ticket fifteen says "tool X is being called twice as often on long inputs and we're hitting our rate limit."

None of these tickets reference any of the lines in the changelog. The platform team's release notes are a list of files moved. The integrator tickets are a list of behaviors changed. The two documents do not meet in the middle, and that gap is where the trust leaks out.