4 posts tagged with "liability"

Your agent works. It resolves the ticket, issues the refund, updates the record, closes the loop — unattended, at a quality your metrics say beats the median human doing the same job. You are ready to take the human out of the loop. And then the deployment stalls, not in engineering, but in a meeting where someone from legal or finance asks a single question you cannot answer: when it's wrong, who eats the loss?

This is the part of agent autonomy that the capability curve doesn't touch. You can push accuracy from 95% to 99% and the question doesn't move. Because the blocker was never "is the model good enough." The blocker is that a probabilistic system taking irreversible actions is a risk somebody has to hold, and right now nobody wants to hold it. Your errors-and-omissions policy was written for human mistakes and increasingly carves out automated ones. Your model vendor's contract disclaims the whole thing. And no carrier has an actuarial table for a system whose failure rate quietly drifts every time you edit a prompt.

Your agent just issued a $40,000 refund to the wrong account, re-routed a freight order that triggered expedited shipping fees, or pushed a config change that took down a customer's production environment for six hours. The action is done. It is irreversible, or close enough that reversing it costs real money. Now the only question that matters is the one nobody asked before you shipped the thing: whose budget eats the loss?

Most teams discover the answer the hard way, in a conference room three days later, with the vendor's account manager on speakerphone reading a liability cap back to them. The cap is the annual subscription fee. The loss is forty times that. The conversation is short.

An agent books the wrong flight. It sends an apology email to the wrong customer. It writes a database migration that drops a column three services still read from. In each case the model produced a plausible-looking action, executed it, and moved on. And in each case somebody absorbed a real cost — a rebooking fee, a damaged relationship, an incident bridge at 2 a.m.

Here is the uncomfortable part: most AI products have no answer for who that somebody is. The question never comes up in the design review. It surfaces later, one ticket at a time, in a support queue where an agent improvised a $40 credit because the customer sounded angry and the rep had no policy to point at. Multiply that by a few thousand tickets a month and the unit economics quietly rot — not from a dramatic failure, but from a slow leak nobody scoped.

"The model made a mistake" is not a support escalation. It is a billing event. And the products that survive the agentic era will be the ones that designed for that event before the first angry ticket, not the ones that improvised refunds by vibes until the gross margin went negative.

A customer's general counsel sends you a one-line email: "When the model invents a fact in our compliance workflow next week, whose insurance is paying?" You forward it to your VP of Engineering, who forwards it to Legal, who forwards it back to you. By the time the chain closes, three people have separately assumed that someone else read the model provider's terms carefully. None of them did. The contracts don't actually connect — and you are the layer in the middle that finds out first.

This is the AI indemnification gap. It exists because every enterprise AI product sits in a three-link liability chain — end customer, your product, model provider — where each link silently assumes the layer underneath is carrying the weight. The model provider's terms cap damages at roughly the last twelve months of fees and explicitly exclude output accuracy. Your MSA inherits those exclusions through a flow-down clause your customer's lawyer didn't read carefully. Your customer's contract with their downstream user — the actual end-of-chain victim when an output goes wrong — names your product as the responsible party with no clear upstream recourse.

The first claim discovers the gap. Until then, everyone in the chain is operating on a hopeful shrug.