The Operational Model Card: Deployment Documentation Labs Don't Publish

A model card tells you whether a model was red-teamed for CBRN misuse and which demographic groups it underserves. What it doesn't tell you: the p95 TTFT at 10,000 concurrent requests, the accuracy cliff at 80% of the advertised context window, the percentage of complex JSON schemas it malforms, or how much the model's behavior has drifted since the card was published.

The gap is structural, not accidental. Model cards were designed in 2019 for fairness and safety documentation, with civil society organizations and regulators as the intended audience. Engineering teams shipping production systems were not the use case. Seven years of adoption later, that framing is unchanged — while the cost of treating a model card as a deployment specification has never been higher.

The 2025 Foundation Model Transparency Index (Stanford CRFM + Berkeley) confirmed the scope of the omission: OpenAI scored 24/100, Anthropic 32/100, Google 27/100 across 100 transparency indicators. Average scores dropped from 58 to 40 year-over-year, meaning AI transparency is getting worse, not better, as models get more capable. None of the four major labs disclose training data composition, energy usage, or deployment-relevant performance characteristics.

What follows is a test battery for producing the operational model card yourself — the documentation you need before committing a model to production, regardless of what the lab published.

What the Lab Card Covers (and What It Doesn't)

Published model cards follow a recognizable template: MMLU, HumanEval, GPQA scores; safety red-teaming results; alignment assessments; knowledge cutoff dates. This is genuinely valuable for evaluating whether a model is safe to expose to users.

It is not documentation for running the model. What's missing:

• Latency specifications. No p50/p95/p99 TTFT. No throughput figures at representative concurrency. No latency-vs-context-length curve. Cold start timing for serverless deployments: absent entirely.
• Context window degradation. Advertised window size is not the practical reliable limit. The two are different by 30–50% for most current models.
• Structured output failure rates. JSON schema adherence across schema complexities. What happens when the model can't comply — does it fail loudly or populate required fields with plausible garbage?
• Task-specific reliability. Accuracy on grounded summarization, extraction, and tool-calling behaves differently for the same model. Aggregate benchmark scores obscure this.
• Behavioral stability. How much has this model's behavior changed since the card was written? Named version pins (e.g., gpt-4o-2024-08-06) do not guarantee stable behavior.
• Cache behavior. Prompt cache hit/miss latency difference. Minimum cache threshold. TTL. None of this appears in published documentation.

This isn't a complaint about labs being secretive. It's that model cards answer different questions than engineering teams ask.

The Latency Profile

The first thing to measure is the latency decomposition: time-to-first-token (TTFT) and inter-token latency (ITL) behave differently and require different interventions.

TTFT is dominated by prefill. It scales with prompt length — for standard attention, roughly O(n²). A 10,000-token prompt takes significantly longer to prefill than a 1,000-token prompt, and the relationship is not linear. This matters because many production failures hide in P99: a system with a 200ms average TTFT can simultaneously have a P99 of 3,000ms under concurrency spikes.

ITL is memory-bandwidth bound once the model is generating. It's more predictable but sensitive to batch size and hardware. At high concurrency, queuing effects dominate.

To produce useful latency documentation, run tests at representative prompt lengths (short, medium, long), across representative concurrency levels (1, 10, 50, 100 concurrent requests), capturing:

• TTFT at p50, p95, p99 for each combination
• ITL at p50 and p95
• Total request latency for outputs at typical length

Cold start is a separate measurement. If you're deploying on serverless or using scale-to-zero infrastructure, the latency gap between cold and warm inference is enormous — typically 30–120 seconds for a cold container versus 200ms for a warm one. This gap belongs in your operational documentation with appropriate architecture implications: scale-to-zero is generally incompatible with interactive latency requirements.

Prompt cache behavior is also worth characterizing separately. Cache hits can reduce TTFT by 50–85% and cut token costs by 80–90%. Cache misses on content you expected to be cached are a common source of latency regressions that look like model performance issues.

The Context Degradation Curve

The advertised context window is the maximum the model will accept without an error. It is not the range over which the model reliably uses context.

The empirical finding — documented across multiple models and validated repeatedly — is a U-shaped performance curve: accuracy peaks when relevant information appears near the beginning or end of the context, and drops substantially when it's in the middle. Drops of 30–60% are typical for multi-document question answering when the relevant document moves from position 1 to position 10 of 20.

More recent work has quantified the degradation even when models have perfect retrieval access: on mathematical reasoning, code generation, and question answering tasks, accuracy degrades 14–85% as input length increases toward the advertised limit. No current frontier model is immune. Llama-3.1-8B-Instruct showed a 24% accuracy drop on a knowledge task at 30,000 tokens; Llama dropped 48% on arithmetic at the same length.

The practical rule of thumb: reliable performance typically degrades noticeably at roughly 50% of the advertised context window. For a model with a 128K token window, plan for 50,000 tokens as the practical limit for tasks where reliability matters. This should appear in your documentation with task-specific measurements, not generic estimates.

To characterize this, run your representative task type — whatever the model will actually do in production — at 10%, 25%, 50%, 75%, and 100% of the advertised window. Record accuracy at each level. The resulting curve is one of the most useful pieces of operational documentation you can produce.

Structured Output Failure Rates

Structured output reliability has improved substantially in the past two years, but the gap between different reliability tiers matters for how you architect error handling.

With constrained decoding (grammar-sampled outputs, native structured output modes), schema compliance exceeds 99.9% for syntactically well-formed requests. This is largely a solved problem if you use the right tooling.

What is not solved: semantic correctness. Schema compliance means the output matches the declared shape. It does not mean the values are correct. A model will confidently populate a confidence_score field with a plausible-looking float, or mark a status field as "complete" when the task was abandoned, because refusing to populate a required field is harder than inventing a value.

Eight reliability layers sit between raw text and a correct, trustworthy output: syntactic validity, schema compliance, referential integrity, semantic accuracy, logical consistency, temporal validity, policy compliance, and reasoning soundness. Constrained decoding enforces the top two. The remaining six require separate verification — or an architecture that doesn't assume a schema-valid response is a correct one.

To characterize your model's structured output behavior, test schema compliance across three levels of schema complexity: simple (flat, 3–5 fields), nested (2–3 levels of nesting), and array-heavy (arrays of objects with constrained elements). Also test semantic validity: what percentage of schema-valid outputs contain values that are factually wrong, internally contradictory, or plausible but incorrect? This is harder to automate but more useful to know.

Behavioral Stability

Model behavior drifts across versions. This is documented, not speculative. A Stanford and Berkeley study tracking GPT-4 behavior between March and June 2023 found: prime number identification accuracy dropped from 84% to 51%, chain-of-thought reasoning dropped from 97.6% to 2.4% of responses for that task, and directly executable code dropped from 52% to 10% of outputs — all without a major version change.

The pattern has continued into more recent model generations. Named version pins reduce but do not eliminate behavioral drift. A monitoring study found GPT-4 showing 23% variance in response length across monitoring periods; Mixtral exhibited 31% inconsistency in instruction adherence.

Your operational documentation should capture a behavioral baseline: a set of reference inputs with expected output characteristics (not exact outputs, but property-based expectations) that you can compare future model behavior against. When a model update lands — or when you're evaluating whether to upgrade to a newer version — running against this baseline gives you a concrete signal rather than relying on whether a regression feels qualitatively different.

This is also the mechanism for detecting silent model swaps. Some providers have shipped model updates without version bumps. A suite of characteristic probes — edge-case responses, calibration outputs, known capability markers — run continuously alongside production traffic catches this class of invisible change.

Agentic Failure Modes

If the model is being deployed in an agentic context — making tool calls, taking multi-step actions — the failure modes shift. Benchmarks on single-turn tasks don't predict these.

An analysis of 900 execution traces across three frontier models (ranging from 32B to 671B parameters) on filesystem, text extraction, CSV analysis, and SQL tasks found success rates between 58.5% and 92.2%. The failure archetypes that appeared across all models:

• Premature action without grounding. Guessing data structure rather than inspecting it first.
• Over-helpfulness under uncertainty. Substituting plausible values for missing entities rather than returning null or requesting clarification.
• Context pollution. Distractor data in the context induces incorrect reasoning paths.
• Fragile execution under cognitive load. Malformed tool calls, generation loops, and inconsistent error recovery as task complexity increases.

The failure that causes the most production damage is usually over-helpfulness under uncertainty. A model that returns null or asks for clarification is easier to handle than a model that confidently fabricates a plausible-looking answer. Characterizing this behavior for your specific use case — how the model responds to incomplete, ambiguous, or adversarial inputs — belongs in your operational documentation alongside accuracy on clean inputs.

Building the Test Battery

The test battery for an operational model card runs in three phases:

Baseline characterization (run before any deployment):

• TTFT and ITL at p50/p95/p99 under representative concurrency
• Cold start timing and warm-up behavior
• Context degradation curve at 25%, 50%, 75%, 100% of the advertised window
• Prompt cache hit/miss latency differential
• Schema compliance rate across simple, nested, and array-heavy schemas
• Semantic correctness rate on schema-valid outputs (sampled manual review if automated validation isn't feasible)

Task-specific reliability (run on your actual production task):

• Accuracy on 200–500 representative samples from your domain
• Hallucination rate on your document types
• Failure mode behavior: what does the model do with incomplete inputs, very long inputs, adversarial inputs?
• Tool call success rate for the tools your agent will actually use

Drift detection (run on every model version update):

• Behavioral baseline comparison against 1,000 reference examples
• Latency regression comparison
• Structured output failure rate comparison

Tools that support this: GuideLLM (Red Hat / vLLM project) simulates configurable production workloads and measures TTFT, ITL, and throughput; Langfuse and Arize Phoenix provide production tracing; DeepEval supports custom task-specific metrics.

Why Teams Skip This (and Why That Changes Later)

The typical pattern documented across hundreds of production LLM deployments: teams reach 80% quality quickly, then spend the majority of their development time closing the gap from 80% to 95%. The failures that emerge in the second phase — context degradation on long documents, structured output semantic failures, behavioral drift after model updates, latency regressions at scale — are exactly the failures that a pre-production test battery would have characterized.

The teams that skip operational characterization don't fail immediately. They fail six months later, after they've deprecated the previous model, after their users have developed expectations, and after the architecture has been built around assumptions about model behavior that the lab never committed to.

Published model cards answer "is this model safe?" Your operational model card answers "will this model work in my system?" Both are necessary. Only one is your job to produce.

Final Checklist Before Model Commitment

Before deprecating your previous model and committing a new one to production:

• TTFT p95 measured at target concurrency — within SLA budget
• Context degradation curve measured at your representative prompt lengths
• Schema compliance rate at your schema complexity level — understood, with error handling designed accordingly
• Semantic failure rate sampled on domain-representative inputs — acceptable or mitigated
• Behavioral baseline captured for future drift comparison
• Cold start behavior characterized if using serverless or variable-capacity infrastructure
• Cache hit rate and hit/miss latency delta measured for your prompt patterns

The teams that discover a model's production limitations after they've committed to it are the ones who treated the lab's published benchmarks as the specification. They are not wrong to want that documentation — it should exist. It just doesn't yet. Until it does, the test battery is yours to run.