Last week, one of my senior engineers flagged a PR with a concerning note: “This library doesn’t exist.”
The code looked perfect. It imported a plausible-sounding package (financial-audit-utils), used it with sensible method names, had proper error handling, and even included tests. The AI coding assistant had generated it in response to “add audit trail validation to transaction processing.” Everything compiled. The tests passed. The code reviewer almost approved it.
But the package didn’t exist. The AI had hallucinated an entire library with plausible APIs.
The Plausibility Trap
This isn’t an isolated incident. A joint study testing 16 code-generation LLMs found that 19.7% of recommended packages were fabricated—they simply don’t exist. Even more concerning: 58% of hallucinated packages repeated across multiple queries, making them exploitable. An attacker can now register malicious packages with names LLMs commonly hallucinate.
Meanwhile, separate research shows 29-45% of AI-generated code contains security vulnerabilities, and performance can degrade dramatically—one study documented AI-generated Rust code running 20,171x slower than the equivalent hand-written implementation while looking completely correct.
Why Traditional Review Fails
In financial services—where I work—code review has always assumed human reasoning: you check for logic errors, edge cases, security issues. You trust that a human developer who imports lodash actually verified that lodash exists.
But LLMs don’t write correct code. They write plausible code. As Katana Quant puts it: “Plausible code compiles, it runs, and on the happy path with clean inputs and nominal traffic, it produces output that looks exactly right. However, plausible is not a semantic guarantee—it is a statistical one.”
This breaks our review assumptions:
- Package dependencies: Can no longer assume imports refer to real libraries
- API calls: Can no longer trust that methods exist on objects
- Security patterns: Can no longer assume defensive code handles actual threat vectors
- Performance: Can no longer trust algorithmic complexity matches apparent implementation
The Non-Deterministic Problem
Adding to the challenge: LLMs are non-deterministic by design. Even with temperature set to 0, the same prompt can yield different outputs due to floating-point math and implementation details. You can’t rely on reproducibility as a validation signal.
The Validation Gap
Here’s the uncomfortable reality in 2026: My team is merging 60% more PRs than last year thanks to AI assistance, but we’re spending 40% more time on code review trying to catch these issues.
Anthropic just launched Code Review specifically to address “the flood of AI-generated code”—acknowledging this is now an industry-wide problem. Early data shows AI code review can improve bug detection 42-48%, but only when combined with human oversight.
My Question for This Community
How do you validate AI-generated code when traditional reviews assume human reasoning?
In our financial services context, a hallucinated dependency could mean regulatory violations, audit failures, or security breaches. We can’t just “move fast and break things.”
I’m curious:
- What validation practices have you implemented?
- Do you use automated checks beyond standard linters?
- How do you train junior engineers to spot these issues?
- Are there entire categories of AI-generated code you now consider too risky?
The tools are incredible—but I’m increasingly convinced we need new paradigms for validation, not just faster review cycles.
What’s working in your organizations?