Notes from building software in production.Notes from building software in production.

Six Billion Requests Later: What a Full Year of LLM Serving Actually Looks Like

By A Covaratech engineer

A new preprint analyzes a full year of unsampled production LLM traffic — 6.12 billion requests across 9,174 models — and the findings reshape how you should think about caching, routing, and capacity planning.