When the AI Center of Excellence Becomes the Bottleneck

Two years ago, standing up an AI center of excellence was the responsible move. Nobody knew how to evaluate a model, procurement had no idea what an inference contract should look like, and legal wanted one throat to choke. Concentrating the ten people who understood any of it into a central team was obviously correct.

Today that same team is the reason your product engineers wait six weeks to change a prompt. The CoE reviews every system-message edit, owns the only eval harness in the company, and gates model upgrades behind a committee that meets biweekly. Teams have noticed. They ship on personal API keys, run evals in notebooks the CoE never sees, and paste customer data into whatever tool answers fastest. You did not prevent shadow AI — you created it, and gave it a governance body to hide from.

The uncomfortable part is that nobody made a mistake. The CoE was the right structure for the bootstrap phase and is the wrong structure now. Org charts have lifecycles the same way codebases do, and the failure mode here is treating scaffolding as architecture — leaving the temporary structure up so long that people start attaching load-bearing walls to it.

The bootstrap logic, and where it expires

A centralized CoE solves a specific problem: expertise is scarce and risk is unpriced. In year one of enterprise AI adoption, both conditions hold. A handful of people know what a hallucination-driven incident looks like; nobody has an eval pipeline; every vendor contract is being negotiated for the first time. Pooling that knowledge means every team's first AI project starts from the org's best understanding instead of from zero.

But both conditions decay. Expertise stops being scarce — after two years of shipping, your product teams have engineers who have debugged retrieval failures and written eval suites, and some of them are better at it than the CoE staff who have been reviewing documents instead of shipping. Risk stops being unpriced — the org has incident history, established vendor terms, and legal precedent. The inputs that justified centralization quietly disappear while the structure remains.

What remains is the queue. Centralized review scales linearly with headcount while demand scales with every team in the company adopting AI, so the gap widens every quarter. Industry analyses of AI CoE failures consistently flag the same trio: accumulating approval delays, business units building workaround systems, and a central team drowning in knowledge overload. If your CoE intake form has a "priority justification" field, you are rationing — and rationing a capability your competitors treat as ambient is a strategy problem, not a process detail.

Shadow AI is the org routing around damage

The network metaphor is exact. Surveys through 2025 put unsanctioned AI use somewhere between half and four-fifths of employees — one large study found 81% of employees and 88% of security leaders using unapproved tools, and executives are the worst offenders, not the exception. The cost is not hypothetical either: breach analyses now attribute roughly a fifth of incidents to shadow AI, at a measurable cost premium over ordinary breaches, because the data flows were invisible until they failed.

Here is the part CoE leaders resist: this usage is not defiance, it is demand. Every unsanctioned ChatGPT session is a signal that a workflow needed AI and the sanctioned path was too slow, too limited, or too invisible to find. A six-week review queue does not reduce AI risk. It relocates risk from a channel you can observe to channels you cannot. The gate becomes a filter that selects for exactly the usage you most wanted to prevent — the unreviewed, unlogged, personal-account kind.

The paradox is worth stating plainly: the stricter the central gate, the less governed your actual AI usage becomes. Governance that people route around is not governance. It is theater with a queue.

Scaffolding, not architecture

We have run this experiment before. Cloud centers of excellence followed the same arc a decade ago: indispensable during migration, then increasingly a review board that platform teams outgrew. The ones that aged well deliberately shifted from doing to enabling — they stopped approving deployments and started building landing zones, policy-as-code, and self-service infrastructure. The ones that aged badly kept the approval meetings and watched engineering route around them. Agile CoEs, data science CoEs, DevOps CoEs — same curve, same fork.

The lesson is that a CoE is a maturity-stage structure, not a permanent org unit. Its purpose is to make itself unnecessary in its original form. That reframing changes how you staff it (rotations, not careers), how you measure it (capabilities transferred, not projects reviewed), and how you talk about its end (graduation, not failure).

Platform engineering already has the vocabulary for the destination: paved roads. Instead of a team that reviews your deployment, you get a road so smooth that deviating from it is more work than following it. The governed path is the fast path — approved models behind a gateway with logging built in, eval harnesses as shared infrastructure any team can run in CI, prompt-change review as an automated policy check rather than a committee agenda item. Self-service replaces ticket queues; policy-as-code replaces manual gatekeeping. Compliance becomes a property of the platform, not an outcome of a meeting.

The signals it's time to dissolve

You do not need a consultant to tell you when the CoE has crossed from accelerant to bottleneck. The signals are measurable:

• Queue length trending up. If median time-to-approval grows quarter over quarter while the CoE's throughput stays flat, the math only ends one way.
• Exception rate. Count how often leadership grants "just this once" bypasses. Each exception is an admission that the process fails the org's actual risk-speed tradeoff. When exceptions become routine, the process is already dead — you just haven't buried it.
• Standards lag. When the CoE's approved-patterns document recommends techniques your teams abandoned two quarters ago — prompt patterns from a model generation back, evals that don't cover agents — the center is no longer where excellence lives.
• Shadow usage growth. If your CASB or network logs show unsanctioned AI traffic growing while official intake tickets shrink, the org has voted.
• Talent flow. When your best CoE engineers transfer into product teams to "actually ship," they are telling you where the interesting problems went.

Any two of these together mean the dissolution conversation is overdue. All five mean teams already dissolved the CoE themselves; the org chart just hasn't been updated.

What stays central forever

Dissolving the CoE as a gate does not mean decentralizing everything — and this is where the pendulum-swing crowd gets it wrong. McKinsey's 2025 State of AI research found that high performers pair aggressive, distributed AI adoption with centralized governance, human-in-the-loop rules, and visible executive accountability. The resolution to the apparent contradiction: centralize the substrate, distribute the decisions.

Three things earn permanent central ownership:

• Vendor leverage. Model contracts, rate negotiations, and capacity commitments benefit from aggregation. Twenty teams negotiating separately with the same model provider is money on fire, and a central gateway is also where usage visibility lives.
• Incident learning. When an AI failure happens in one business unit, the lesson must propagate to all of them. Incident review, red-team findings, and the taxonomy of what has actually gone wrong are natural monopolies — distributed teams have no incentive to share failures and every incentive to bury them.
• The eval substrate. Not the evals themselves — teams should own evals for their domains — but the harness, the shared benchmarks, the regression infrastructure that makes "we evaluated it" mean the same thing in every corner of the company. This is the AI equivalent of the build system: nobody wants to own it locally, everybody needs it to be excellent.

Notice what is absent from that list: use-case approval, prompt review, model selection for individual products. Those decisions belong to the teams that own the outcomes, operating on the paved road the platform provides.

Plan the dissolution on day one

The actionable version of all this depends on where you sit. If you are standing up a CoE now, write its sunset criteria into its charter: "this team converts to a platform function when N teams have shipped AI features and the eval harness is self-service." A CoE with a defined graduation condition behaves differently from one defending a permanent budget line — it builds tools instead of processes, because tools are what survive the transition.

If you are running a CoE that the signals say has expired, the move is conversion, not deletion. The review function becomes policy-as-code on a shared gateway. The expertise function becomes embedded rotations — CoE engineers spend two quarters inside product teams and mostly don't come back, which is the point. The governance function shrinks to the three permanent centrals: vendors, incidents, evals. Headcount mostly doesn't shrink; it redeploys from reading documents to building roads.

And if you are on a product team routing around your CoE today: the shadow tooling you built under a personal API key is evidence, not shame. Bring it to the dissolution conversation. The fastest way to get a paved road built is to show the desire path everyone is already walking.

The organizations that get this right will not be the ones that never built a CoE, nor the ones that keep theirs forever. They will be the ones that knew the difference between scaffolding and architecture — and took the scaffolding down while it was still a sign of success rather than a monument to the moment adoption stalled.