What Is Latency vs. Throughput in AI? Two Different Ways to Measure Speed
A system can be fast for one user and slow overall, or the reverse — latency and throughput measure two genuinely different things, and optimizing one can hurt the other.
Two different kinds of "fast"
Latency measures how long a single request takes from start to finish — how long one user waits for a response. Throughput measures how many total requests a system can handle across a given period of time. A system can have low latency for one user while still having limited throughput overall, or vice versa.
Why the two can pull in opposite directions
Inference batching — grouping multiple requests together to process more efficiently — tends to improve overall throughput, but it can add a small amount of latency to each individual request, since a request might wait briefly for a batch to fill before processing starts.
Why both matter, for different reasons
Latency matters most for interactive use, where a user is actively waiting for a response in real time. Throughput matters most for overall system cost and capacity — how many users a given amount of infrastructure can serve at once. A well-tuned system usually balances both rather than maximizing one at the other's expense.
Frequently Asked Questions
What is the difference between latency and throughput?
Latency is how long one individual request takes to complete; throughput is how many total requests a system can handle over a given time period. A system can optimize one at some cost to the other.
Why would a system trade off some latency for more throughput?
Techniques like inference batching improve overall system throughput and cost efficiency by processing requests together, but can add a small delay to individual requests while a batch fills — a common, deliberate tradeoff.