Property-Based Testing Is the Safety Net for AI-Generated Code — Why Aren't More Teams Using It?

General
1

We’ve been using GitHub Copilot and Claude for code generation across our team at TechFlow for the past eight months, and I want to share something that completely changed how we think about testing AI-generated code.

The Problem Nobody Talks About

AI-generated code has a surprisingly high defect rate. Various studies and industry reports put it at 50% or higher for non-trivial functions. The insidious part? The code looks correct. It’s well-structured, follows conventions, has reasonable variable names, and handles the obvious cases. Traditional unit tests — the kind we write based on expected inputs and outputs — often pass because we tend to test the same “happy path” that the AI was trained on.

Our team experienced this firsthand. After adopting Copilot, our unit test coverage stayed at 85% (our internal target), but production bugs increased by 40% over a three-month period. We were writing more code faster, our tests were green, and yet our bug tracker was filling up. Something was fundamentally broken in our quality feedback loop.

Enter Property-Based Testing

Property-based testing (PBT) flips the testing paradigm on its head. Instead of writing specific example-based tests like “given input X, expect output Y,” you define properties — invariants that must always hold true for any valid input — and the testing framework generates hundreds or thousands of random inputs to find violations.

For example, instead of testing sort([3, 1, 2]) equals [1, 2, 3], you define properties like:

  • The output length must equal the input length
  • Every element in the output must exist in the input
  • Each element must be less than or equal to the next element

The framework then generates random lists — empty lists, single-element lists, lists with duplicates, lists with negative numbers, lists with MAX_INT values — and checks if your properties hold.

The Tools

The ecosystem is more mature than most developers realize:

  • Hypothesis (Python) — The gold standard. Incredibly powerful generators and shrinking.
  • fast-check (TypeScript/JavaScript) — Excellent for full-stack TS teams. Integrates with Jest and Vitest.
  • QuickCheck (Haskell) — The original. Still influential in how PBT frameworks are designed.
  • PropCheck (Elixir) — Great integration with ExUnit, feels native to the language.
  • jqwik (Java/Kotlin) — JUnit 5 integration, great for enterprise Java shops.

Why PBT Catches What Unit Tests Miss

AI-generated code excels at the common case because that’s what dominates training data. Stack Overflow answers, GitHub repositories, and documentation examples all focus on typical usage. But production failures happen at the boundaries:

  • Empty inputs: AI-generated parsers that crash on empty strings
  • Unicode: String manipulation functions that break on emoji or RTL text
  • Negative numbers: Math functions that assume positive inputs
  • Concurrent access: Race conditions in shared state that only manifest under specific timing
  • Boundary values: Off-by-one errors at MAX_INT, zero-length arrays, null values

PBT randomizes exactly these categories. The generators are designed to produce adversarial inputs — the exact kind of edge cases that AI training data under-represents.

Our Results

After our production bug spike, we added property-based tests for every AI-generated function touching business logic. In the first week, we caught three critical bugs — all boundary condition failures:

  1. A currency conversion function that returned NaN for zero amounts (dividing by a rate that could be zero)
  2. A date range filter that included the end date in some time zones but not others
  3. A pagination function that returned duplicate results when the total count was exactly divisible by the page size

All three had passing unit tests. All three would have reached production without PBT.

The Generator + Tester Pattern

Here’s the workflow we’ve settled on:

  1. AI generates the implementation — Copilot or Claude writes the function
  2. Human defines the properties — The engineer identifies what invariants must hold
  3. PBT framework finds the gaps — Hypothesis/fast-check generates inputs and reports violations
  4. AI fixes the implementation — Feed the failing property back to the AI for a fix
  5. Repeat until all properties hold

This is a powerful loop. The AI is fast at generating code, the human is good at reasoning about correctness properties, and the PBT framework is thorough at finding violations.

Why Adoption Is Still Low

Here’s the honest answer: defining properties is hard. It requires a different kind of thinking than example-based testing. Writing expect(add(2, 3)).toBe(5) is straightforward. Writing “for all integers a and b, add(a, b) should equal add(b, a)” requires you to think abstractly about what the function guarantees.

Most developers find this intellectually demanding, especially mid-level engineers who haven’t been trained in formal reasoning. But here’s the twist — AI tools themselves can help. Ask Claude “what properties should this sorting function always satisfy?” and you’ll get a solid starting list. The AI is actually good at reasoning about properties even when it’s bad at implementing them correctly.

My Question to the Community

Is anyone else combining property-based testing with AI code generation? I’m particularly curious about:

  • Which PBT frameworks work best with your stack?
  • How do you decide which functions deserve PBT vs. traditional unit tests?
  • Have you tried using AI to generate the property definitions themselves?

I genuinely believe PBT is the missing piece in the AI-assisted development puzzle. We’re generating code faster than ever, but our quality assurance hasn’t kept pace. PBT is how we close that gap.

2

Alex, this resonates deeply from a security perspective. I’d argue PBT is even more critical for security testing than general correctness, and here’s why.

Security Vulnerabilities Are Property Violations

Think about the most common vulnerability classes:

  • SQL injection: The property is “no user input should appear unescaped in a database query.” That’s directly testable.
  • XSS: The property is “no user-supplied content should be rendered as executable markup in the browser.” Testable.
  • Buffer overflows: The property is “output length should never exceed the allocated buffer size for any input.” Testable.
  • Path traversal: The property is “resolved file paths should always be within the allowed directory.” Testable.

Every single one of these is an invariant — a property that must hold for all inputs, not just the benign ones your unit tests cover.

Custom Security Generators

I’ve been building custom Hypothesis generators specifically designed to produce adversarial security inputs. Instead of random strings, my generators produce:

  • Injection payloads: Classic injection patterns, Unicode-based variants, and encoding bypass attempts that attackers commonly use
  • Markup payloads: Script tags, event handler attributes, SVG-based vectors, and various encoding tricks
  • Oversized inputs: Strings at 10x the expected maximum length, deeply nested JSON structures, inputs designed to trigger catastrophic backtracking in regular expressions
  • Encoding attacks: Mixed UTF-8/Latin-1 encodings, null bytes mid-string, Unicode homoglyphs that look like ASCII characters but aren’t

The key insight is that these generators are reusable across projects. Once you build an injection_strings() generator, you can apply it to every function that touches user input.

Real Results

Last quarter I audited a codebase where a team had used Copilot to generate their input validation layer. Their SAST tool (Semgrep) gave it a clean bill of health. I ran my Hypothesis security generators against it and found two vulnerabilities in the first afternoon:

  1. A search endpoint that properly escaped inputs in the WHERE clause but forgot to escape the ORDER BY parameter — because the AI-generated code treated column names as “safe” inputs. My generator sent a malicious sort parameter and the property “query should only contain whitelisted column identifiers” failed immediately.

  2. A file upload handler that validated file extensions but not the actual file content. My generator produced a file with a benign extension but malicious content headers, and the property “uploaded file content-type must match the declared extension” failed immediately.

Neither of these would have been caught by example-based tests that use clean filenames and normal search terms.

The Compound Effect

When you combine PBT security generators with Alex’s Generator+Tester pattern, you get something powerful: AI generates the code, PBT with security generators stress-tests it, and you catch vulnerabilities before they reach production. This is shifting security left in a way that manual code review and SAST tools simply cannot match.

I’ve open-sourced my Hypothesis security generators — happy to share the repo if anyone’s interested. The hardest part was building the generators; using them is trivial once they exist.

3

This is a great thread, Alex. I want to add the data engineering perspective because PBT has been a game-changer for our pipeline validation work.

Data Pipelines Are Property Machines

Data transformations are naturally described by properties, which makes them ideal candidates for PBT. Here are some properties we test routinely:

  • Row count invariants: “A filter operation should always return fewer or equal rows than the input.” “A join operation output row count should be predictable based on join type and key cardinality.”
  • Null propagation: “No null values should exist in required fields after transformation.” “Nullable fields should remain nullable unless explicitly coalesced.”
  • Type preservation: “A string column should never silently become a numeric column after transformation.” “Date fields should maintain their timezone information through the pipeline.”
  • Idempotency: “Running the same transformation twice on the same input should produce identical output.”
  • Monotonicity: “For append-only tables, the output row count should always be greater than or equal to the previous run.”

These properties are universal across data pipelines. You write them once and they apply to every transformation in your DAG.

Hypothesis + Pandas: A Powerful Combination

I generate random DataFrames using Hypothesis’s @given decorator with custom strategies:

from hypothesis import given
from hypothesis.extra.pandas import columns, data_frames

@given(
    df=data_frames(
        columns=[
            columns("user_id", dtype=int),
            columns("amount", dtype=float),
            columns("currency", dtype=str),
        ]
    )
)
def test_currency_conversion_preserves_row_count(df):
    result = convert_currencies(df, target="USD")
    assert len(result) == len(df)
    assert result["amount"].notna().all()

This generates DataFrames with random integers, floats (including NaN, infinity, negative zero), and strings (including empty strings, Unicode, and extremely long values). It’s remarkable how many transformation bugs surface when your test data isn’t curated.

AI-Generated Transformations Are Especially Fragile

I’ve noticed a specific pattern with AI-generated data transformations: they handle type coercion incorrectly more than anything else. When Copilot generates a pandas transformation, it tends to assume columns are the type they “should” be rather than the type they actually are. Common failures:

  • String columns that contain numeric-looking values get silently cast to float, losing leading zeros (zip codes, product codes)
  • Date columns with mixed formats get partially parsed, leaving some rows as strings and others as datetime objects
  • Integer columns with null values get upcast to float (pandas classic), and the AI-generated code doesn’t account for this

PBT catches all of these because the random DataFrame generator naturally produces these edge cases. Our hand-written test data never included a DataFrame where a “numeric” column had one null value causing a type change — but Hypothesis found it in the first 50 examples.

Our Adoption Strategy

We don’t use PBT for everything. Our rule is:

  1. PBT for all transformations: Any function that takes a DataFrame in and produces a DataFrame out gets property-based tests
  2. PBT for all aggregations: Sum, count, average — define the mathematical properties and let Hypothesis verify them
  3. Traditional tests for integration logic: API calls, file I/O, database writes — these are better tested with mocks and fixtures

This split has reduced our data quality incidents by about 60% over six months. The ROI on PBT for data engineering is extraordinary because the properties are so naturally expressible.

Would love to hear if anyone else is using Hypothesis for data pipeline testing. I feel like the data engineering community hasn’t discovered PBT yet, and it’s a perfect fit.

4

Great discussion, and I want to bring the engineering management perspective because adopting PBT across a team is a very different challenge than adopting it individually.

The Learning Curve Is Real

We tried introducing PBT across my organization (40+ engineers) last quarter. I was enthusiastic after seeing the results on a proof-of-concept project. The rollout was… humbling.

What happened: Our senior engineers (Staff and Principal level) took to it immediately. They could look at a function and intuitively identify properties — commutativity, idempotency, monotonicity, invariant preservation. They started writing property-based tests within a day and were finding bugs within a week.

Our mid-level engineers struggled significantly. The feedback I kept hearing was: “I understand the concept, but I can’t figure out what properties to test.” The leap from “test that f(x) == y for this specific x” to “test that f always satisfies property P for all valid inputs” requires a kind of abstract reasoning that isn’t developed through typical software engineering training. It’s closer to writing formal specifications than writing tests.

Our Compromise: Tiered Testing Strategy

Rather than mandate PBT everywhere (which would slow down the team) or abandon it (which would waste its potential), we adopted a tiered approach:

Tier 1 — PBT Required (critical paths):

  • Payment processing and financial calculations
  • Authentication and authorization logic
  • Data transformations and ETL pipelines
  • Cryptographic operations
  • Any function flagged as AI-generated in our code review process

Tier 2 — PBT Encouraged (important but not critical):

  • Business logic with complex branching
  • API input validation
  • State machine transitions

Tier 3 — Traditional Unit Tests (everything else):

  • UI components
  • Simple CRUD operations
  • Integration tests
  • Configuration and setup code

This gives us the safety benefits of PBT where it matters most while not creating a bottleneck for the entire team. The senior engineers own Tier 1 properties, and we use those as teaching examples to gradually upskill the mid-level engineers.

Using AI to Generate Property Definitions

Here’s the most promising experiment we’re running. When an engineer finishes writing a function (whether AI-generated or not), they ask Claude: “What properties should this function always satisfy? List them as testable invariants.”

The results are surprisingly good. Claude typically generates 5-8 properties per function, and about 70% of them are meaningful and testable. The engineer then curates the list, removes the obvious or redundant ones, and implements the remaining properties as PBT specs.

This workflow dramatically lowers the barrier. The hardest part of PBT — identifying the properties — becomes an AI-assisted brainstorming session rather than a solo abstract reasoning exercise. The engineer still needs to understand and validate the properties, but they don’t need to generate them from scratch.

The ROI Question

For anyone trying to sell PBT to their leadership: the ROI argument is straightforward. Track your production bug rate before and after PBT adoption for the same category of code. In our case, Tier 1 code saw a 55% reduction in production incidents in the first quarter after PBT adoption. That’s a number any VP of Engineering will pay attention to.

The cost is real — PBT tests take longer to write and run slower than unit tests. But the cost of production bugs in critical systems is orders of magnitude higher.

Alex, Sam, Rachel — really appreciate the technical depth in this thread. I’m sharing it with my engineering managers as a case study for our next quarterly planning.