3 posts tagged with "insurance"

Your agent works. It resolves the ticket, issues the refund, updates the record, closes the loop — unattended, at a quality your metrics say beats the median human doing the same job. You are ready to take the human out of the loop. And then the deployment stalls, not in engineering, but in a meeting where someone from legal or finance asks a single question you cannot answer: when it's wrong, who eats the loss?

This is the part of agent autonomy that the capability curve doesn't touch. You can push accuracy from 95% to 99% and the question doesn't move. Because the blocker was never "is the model good enough." The blocker is that a probabilistic system taking irreversible actions is a risk somebody has to hold, and right now nobody wants to hold it. Your errors-and-omissions policy was written for human mistakes and increasingly carves out automated ones. Your model vendor's contract disclaims the whole thing. And no carrier has an actuarial table for a system whose failure rate quietly drifts every time you edit a prompt.

Your agent just issued a $40,000 refund to the wrong account, re-routed a freight order that triggered expedited shipping fees, or pushed a config change that took down a customer's production environment for six hours. The action is done. It is irreversible, or close enough that reversing it costs real money. Now the only question that matters is the one nobody asked before you shipped the thing: whose budget eats the loss?

Most teams discover the answer the hard way, in a conference room three days later, with the vendor's account manager on speakerphone reading a liability cap back to them. The cap is the annual subscription fee. The loss is forty times that. The conversation is short.

Every software warranty ever written assumed deterministic behavior. You ship a function, it returns the same output for the same input, and your warranty covers the gap between documented behavior and actual behavior. AI features shatter that assumption entirely.

When your LLM-powered feature tells a customer something wrong — and that wrong thing costs them money — traditional warranty language leaves everyone pointing fingers at everyone else.

This is not hypothetical. Cumulative generative AI lawsuits in the U.S. passed 700 between 2020 and 2025, with year-over-year filings accelerating by 137%. The legal infrastructure governing software liability was built for a deterministic world, and the mismatch is already causing real damage.