TTFT Is the Only Latency Metric Your Users Actually Feel

Your model generates a 500-word response in 8 seconds. A competing model generates the same response in 12 seconds. Intuitively, yours should feel faster. But if your first token arrives at 2.5 seconds and theirs arrives at 400 milliseconds, your users will describe your product as slow — regardless of total generation time. This is the central paradox of LLM latency: the metric your infrastructure team optimizes for (end-to-end generation time, tokens per second) is not the metric your users experience. Time-to-first-token is.

TTFT is not a detail. It is the primary signal users use to judge whether your AI feature is responsive. Getting it wrong means building fast systems that feel slow.

What TTFT Actually Measures

Time-to-first-token is the elapsed time from when a request is submitted to when the first output token is rendered to the user. It captures everything: network transit, queue wait time, prefill computation (the forward pass over all input tokens), and HTTP response initiation. It does not include the time to generate the remaining tokens.

This distinction matters because prefill and decode are computationally different operations. Prefill processes all input tokens in parallel — it's memory-bandwidth-bound and scales with context length. Decode generates output tokens one at a time — it's latency-bound and scales with output length. When your context is long or your system is under load, prefill gets expensive, and TTFT climbs.

Tokens-per-second (TPS) and total generation time are the other two major metrics. They describe throughput, which matters for batch workloads, cost, and capacity planning. But in interactive settings — chat, copilots, code assistants — users have stopped reading dashboards by the time TPS becomes relevant. They're already forming an opinion at the 500ms mark.

Why Streaming Masks Latency (And Why That's Both Good and Dangerous)

Streaming is a perceptual trick. When a model begins emitting tokens the moment they're generated, users experience the interaction as a conversation rather than a query-response cycle. Studies show users perceive streaming responses as 40–60% faster than equivalent non-streaming responses, even when total generation time is identical. The typing-cursor illusion works.

But streaming only masks latency after the first token arrives. If TTFT is 3 seconds, streaming doesn't help — users see a blank screen for 3 seconds, then a burst. The perceived responsiveness that streaming provides is contingent on TTFT being short. A slow TTFT followed by fast streaming feels worse than a slow non-streaming response, because users are actively waiting for something they've been led to expect will appear imminently.

There's a second failure mode: streaming the wrong thing. When a model is in the process of generating an incorrect or hallucinated response, streaming exposes that error progressively. The user reads the first paragraph of a confidently-worded wrong answer before the model corrects itself — or doesn't. Unlike a buffered response where you can validate before display, a streamed response commits the UI to showing in-progress output. This raises the stakes for what you stream first: an outline, a clarifying question, or a status line is better than the start of a wrong answer.

Practically, streaming also requires careful rendering logic. Partial markdown tokens — an unclosed code block, a half-rendered heading — can corrupt the UI layout mid-stream. Any renderer consuming streamed LLM output needs to buffer incomplete markdown syntax rather than rendering chunk-by-chunk verbatim.

The Numbers You Need to Know

Human perception research provides clear latency thresholds. Below 100ms feels instantaneous for basic interactions. Between 100ms and 1 second, users notice a delay but it doesn't interrupt their flow of thought. Beyond 1 second, users must actively wait, and beyond 10 seconds, they'll typically disengage.

For LLM chat interfaces, the targets translate roughly as:

• TTFT under 500ms: feels responsive; users don't consciously register the wait
• TTFT 500ms–1 second: acceptable, especially if there's a clear loading indicator
• TTFT 1–2 seconds: perceptibly slow; users start question-forming about reliability
• TTFT over 2 seconds: degraded experience; users abandon or retry

These thresholds vary by task type. Code completion demands tighter bounds — TTFT above 100ms breaks the autocomplete illusion. Document summarization tolerates 2–3 seconds because users understand a longer task is underway. The interaction model shapes the tolerance window.

What engineers often miss is the difference between average TTFT and tail TTFT. A system with p50 TTFT of 300ms might have p99 of 3 seconds — meaning 1% of users wait 10× longer than the median. That p99 represents your worst-case impression: the user who hits you during a prefill spike, under high concurrency, with a long context window. Track p50, p95, and p99 separately. Alert on p95. Design for p99.

What Drives TTFT and What You Can Do About It

TTFT is determined by a cascade of factors, most of which are controllable.

Context length has the largest user-visible impact. Each additional input token adds approximately 0.2–0.24ms to TTFT (p95) under typical serving conditions. This is roughly linear in practice, despite attention's theoretical quadratic complexity — GPU hardware masks the gap at moderate sequence lengths. But at 10K, 50K, or 100K tokens, the accumulation becomes significant. Prompt engineering discipline — eliminating redundant context, aggressive truncation of history, subsetting schemas and examples — directly translates into lower TTFT.

Prefill caching is the highest-leverage optimization available to most teams. When the prefix of a request matches a previously computed KV cache, the serving layer can skip recomputation entirely. The savings are dramatic: reusing a cached prefix can reduce TTFT from seconds to hundreds of milliseconds for long-context requests. NVIDIA reports 5× TTFT improvements from KV cache early reuse in TensorRT-LLM. Production serving systems — vLLM, SGLang, NVIDIA NIM — all implement prefix caching. If your serving stack supports it, enable it. If your requests share common prefixes (system prompts, tool schemas, few-shot examples), structure them to maximize cache hit rate.

Queue depth and concurrency create hidden TTFT spikes. Under low load, a request reaches a GPU quickly. Under high concurrency, requests wait in queue while other requests' prefill operations complete. This is the primary driver of p99 TTFT degradation. Chunked prefill — splitting long prompts into smaller chunks interleaved with decode iterations — is one mitigation: it prevents a single long-context request from blocking the queue for all other requests. Disaggregated prefill-decode architectures take this further, dedicating separate GPU pools to prefill and decode, achieving 4× higher goodput in research settings.

Provider selection matters more than most teams assume. Across major providers in 2025–2026, TTFT varies by 3–5× for the same model class under similar conditions. Some providers optimize for throughput (higher TPS, slower TTFT); others optimize for latency (lower TTFT, moderate TPS). Measure your actual production TTFT — not synthetic benchmarks — broken down by percentile and context length, before assuming your provider is performing well.

Designing Around TTFT Constraints

If you can't fully eliminate TTFT, you can design around it.

The most reliable technique is giving users something meaningful before the slow part arrives. A status line — "Searching documentation...", "Analyzing your code..." — sets a processing frame that makes a 1.5-second TTFT feel like forward progress rather than silence. This is why skeleton screens outperform blank loading states: they establish that something is happening and roughly what shape the response will take.

A second pattern: stream the outline or plan before streaming the content. If a model needs to generate a long structured response, outputting the section headings first gives users a navigational frame while the body generates. This requires prompt engineering to elicit that ordering, but it's achievable with chain-of-thought prompting or explicit output format instructions.

A third pattern: expose context consumption as a UI signal rather than hiding it. Users who understand why a request is slower — "processing 45,000 tokens of context" — are more tolerant than users who see unexplained delay. This is especially relevant for document analysis, codebase search, or long conversation history.

The one pattern to avoid is hiding limits and then hard-failing. Interfaces that suppress context window or latency signals feel magical until they don't — and when the failure mode is a sudden 10-second blank screen or an error message after the user has already typed their message, the damage to trust is disproportionate.

Instrumenting for TTFT in Production

Most application-layer LLM SDKs do not expose TTFT as a first-class metric. You need to instrument it yourself.

The measurement is straightforward: record a timestamp when you dispatch the request, record another timestamp when you process the first streaming chunk, and compute the delta. Emit this as a histogram metric with tags for model, context length bucket, user tier, and any other dimensions you care about. Alert when p95 exceeds your threshold for that use case.

What's less obvious: you need to measure TTFT from the user's perspective, not the server's. Network transit between your LLM provider and your application server is not included in provider-side latency reports. If your application server is in US-East and your users are in Europe, you're adding 80–120ms of round-trip. Measure client-side TTFT in your frontend — the delta from request initiation to first rendered character — and compare it to your server-side measurement. The gap tells you where latency is accumulating.

Track TTFT separately for different request archetypes. Short prompts with short context will have very different TTFT distributions than long-context document analysis requests. Aggregating them into a single metric hides degradation in one category behind good performance in another.

The Product Decision Hidden Inside TTFT

TTFT is ultimately a product decision disguised as an infrastructure metric. A team that chooses to use a 70B parameter model for a real-time autocomplete feature has made a TTFT commitment, whether they analyzed it that way or not. A team that sends 50,000 tokens of context for a simple Q&A query has made the same implicit decision.

The right framing: every user interaction has a latency budget. TTFT consumes most of it before a single output character appears. What's left — the tolerance window before users feel friction — determines how long your model can generate before the experience degrades. Set your TTFT target based on the interaction type, then work backwards to model size, context length, and infrastructure tier.

Features that feel fast are not always actually fast. Features that feel slow are not always actually slow. The difference, most of the time, is when the first token arrives.

Measure it. Alert on it. Design for it.