What p99 hides and how to see it

Averages describe the whole population. They do not show what a slow user experienced, and they rarely identify the path that needs attention.

If you manage a service, this matters because the angry customer is usually not living at the average. If you operate the service, it matters because the fix depends on the shape of the latency distribution, not only the headline number.

This article explains what p99 is good for, what it hides, and why histograms are evidence worth keeping.

Start with the distribution

Percentiles summarize a distribution without requiring someone to read every data point. They are useful summaries, but each one leaves some of the distribution out.

Metric	Plain-English reading	Common trap
Average	Add every request time, then divide by request count.	A few very slow requests can disappear inside a pleasant number.
p50	Half of requests were this fast or faster.	The median user is rarely the user who escalates.
p95	95% of requests were this fast or faster.	It says little about the slowest 5%.
p99	99% of requests were this fast or faster.	It still compresses many causes into one number.
p99.9	99.9% of requests were this fast or faster.	At low volume, it may be one strange request pretending to be a trend.

Higher percentiles get you closer to bad experiences. They do not explain those experiences automatically.

The average can tell the wrong story

Consider a simple distribution: most requests are fast, a few are painfully slow, and the average and p99 are both true. Neither number explains the shape by itself.

Request latency distribution, cache hit vs. miss (illustrative)

Metric	Value (%)
0-25ms	5
25-50ms	40
50-100ms	45
100-200ms	1
200-400ms	1
400-800ms	7
800-1600ms	1

Here, p99 lands around 800ms. That number suggests a slow tail, but the histogram shows its shape: 90% of requests resolve under 100ms, and a distinct 8% cluster sits at 400-800ms, with almost nothing in between.

A single percentile is a summary. A histogram is the evidence.

Why shape changes the fix

A single scalar cannot tell you whether one population has a long tail or two populations have different causes. That distinction changes the fix.

If the 400-800ms cluster is cache misses, then optimizing the already-fast cache-hit path is busywork. The better question is why the miss path is slow: the backend call may be expensive, the cache key may be too specific, or the data may need warming before traffic arrives. Same p99, different remediation.

1. Find the headline

   Use p95 or p99 to see whether the user-facing tail is moving.

2. Open the shape

   Use histograms to see clusters, gaps, and separate populations.

3. Attach a cause

   Slice by cache status, route, tenant, host, region, payload size, or dependency.

4. Fix the right path

   Optimize the population causing pain, not the population that is already healthy.

Keep histograms, not only rollups

Instrumenting the tail means bucketing latencies, not just sampling them. A histogram can merge across hosts and time windows. A precomputed percentile cannot.

Averaging ten hosts' p99s, or rolling twenty-four hourly p99s into a daily one, produces a number with no statistical meaning. Only the underlying histograms can be merged and re-percentiled correctly.

Most metrics systems support histograms out of the box. The practical decision is where to put the bucket boundaries: tighter where the SLO lives, coarser everywhere else.

For a web endpoint with a 300ms target, I would rather have useful buckets around the target than a perfect-looking chart that jumps from 100ms to one second. Good telemetry shows the point where user experience changes.

Keep the histograms, not just the rollups, and look at the shape before you trust the number. The p99 you publish should be a headline, not the whole story.