ES|QL PERCENTILE function

Returns the value at which a certain percentage of observed values occur.

number

percentile

Returns the value at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values and the 50th percentile is the MEDIAN.

FROM employees
| STATS p0 = PERCENTILE(salary,  0)
     , p50 = PERCENTILE(salary, 50)
     , p99 = PERCENTILE(salary, 99)

The expression can use inline functions. For example, to calculate a percentile of the maximum values of a multivalued column, first use MV_MAX to get the maximum value per row, and use the result with the PERCENTILE function

FROM employees
| STATS p80_max_salary_change = PERCENTILE(MV_MAX(salary_change), 80)

PERCENTILE can also operate on exponential_histogram fields, approximating the percentile of the values which were used to construct the histograms.

TS exp_histo_sample
| WHERE instance == "instance-0"
| STATS p99 = ROUND(PERCENTILE(responseTime, 99), 4)

PERCENTILE can also operate on tdigest and casted histogram fields, approximating the percentile of the values which were used to construct the digests.

TS histogram_timeseries_index
| WHERE instance == "instance-0"
| STATS p95 = ROUND(PERCENTILE(responseTime::tdigest, 95), 2)

There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at my_array[count(my_array) * 0.5].

Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, approximate percentiles are calculated.

The algorithm used by the percentile metric is called TDigest (introduced by Ted Dunning in Computing Accurate Quantiles using T-Digests).

When using this metric, there are a few guidelines to keep in mind:

• Accuracy is proportional to q(1-q). This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median
• For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).
• As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated

The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:

It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.