One post tagged with "small-language-models"

Here is a number that should bother you more than it does: the median network round trip to a cloud LLM endpoint costs you somewhere between 48 and 800 milliseconds before the model emits a single token. That is not inference time. That is not queue time. That is the price of leaving the building — DNS, TLS, the haul to the nearest region, the haul back. For a single chat completion, you never notice it. For an agent that makes nine sequential tool calls to answer one question, you just shipped five seconds of pure transit latency, and your users feel every bit of it.

Most teams treat the round trip as a fixed cost of doing business, something you optimize around with streaming and spinners. But for a growing class of workloads, the round trip is the bug. The model you need is small enough to run where the data already lives, the task is repetitive enough that the marginal cloud cost compounds into real money, and the data is sensitive enough that shipping it to a third party is a liability you keep quietly accepting. On-device and edge inference is the option most engineering teams dismiss reflexively — "the local models aren't good enough" — and the dismissal is increasingly wrong.