[Paper Review] Trimming and threshold selection in extremes

Martin Bladt, Hansjörg Albrecher+1 more|arXiv (Cornell University)|Mar 19, 2019

Financial Risk and Volatility ModelingEconomics, Econometrics and Finance21 references3 citations

TL;DR

This paper proposes a trimmed Hill estimator that improves threshold selection in extreme value statistics by removing lower-order order statistics and rescaling the remaining terms. It introduces a flatness-based method to identify optimal thresholds, reducing reliance on estimating tail characteristics, and presents a weighted tail index estimator effective for lighter tails, validated through simulations and insurance data.

ABSTRACT

We consider removing lower order statistics from the classical Hill estimator in extreme value statistics, and compensating for it by rescaling the remaining terms. Trajectories of these trimmed statistics as a function of the extent of trimming turn out to be quite flat near the optimal threshold value. For the regularly varying case, the classical threshold selection problem in tail estimation is then revisited, both visually via trimmed Hill plots and, for the Hall class, also mathematically via minimizing the expected empirical variance. This leads to a simple threshold selection procedure for the classical Hill estimator which circumvents the estimation of some of the tail character- istics, a problem which is usually the bottleneck in threshold selection. As a by-product, we derive an alternative estimator of the tail index, which assigns more weight to large observations, and works particularly well for relatively lighter tails. A simple ratio statistic routine is suggested to evaluate the good- ness of the implied selection of the threshold. We illustrate the favourable performance and the potential of the proposed method with simulation studies and real insurance data.

Motivation & Objective

• To address the challenge of threshold selection in extreme value statistics, a major bottleneck in tail index estimation.
• To reduce dependence on estimating complex tail characteristics, which often hinders classical threshold selection methods.
• To develop a visually and mathematically grounded procedure for selecting optimal thresholds using trimmed statistics.
• To propose an alternative tail index estimator that emphasizes large observations, particularly effective for lighter tails.
• To provide a simple, interpretable routine for assessing threshold selection quality via a ratio statistic.

Proposed method

• The classical Hill estimator is modified by removing lower-order order statistics and rescaling the remaining upper-order terms.
• Trimmed Hill plots are used to visualize the behavior of the estimator across different trimming levels, revealing flat trajectories near optimal thresholds.
• For the Hall class of distributions, the method minimizes expected empirical variance to mathematically determine the optimal threshold.
• A new tail index estimator is derived that assigns higher weights to larger observations, improving performance in lighter-tailed scenarios.
• A ratio statistic is proposed to evaluate the quality of selected thresholds, providing a diagnostic tool for practitioners.
• The approach is validated through simulation studies and real-world analysis of insurance loss data.

Experimental results

Research questions

• RQ1 How can threshold selection in the Hill estimator be improved without relying on estimating complex tail characteristics?
• RQ2 What patterns emerge in the behavior of trimmed Hill estimators as a function of trimming extent?
• RQ3 Can the flatness of trimmed statistic trajectories near the optimal threshold be leveraged for automatic threshold selection?
• RQ4 How does the proposed weighted tail index estimator compare to the classical Hill estimator in terms of variance and bias, especially for lighter tails?
• RQ5 Can a simple ratio statistic reliably assess the quality of threshold selection in practice?

Key findings

• Trimmed Hill plots exhibit flat trajectories near the optimal threshold, providing a visual cue for threshold selection.
• The method reduces the need to estimate tail characteristics, which are often a bottleneck in classical threshold selection.
• The proposed tail index estimator assigns more weight to large observations and shows improved performance for relatively lighter tails.
• The ratio statistic routine effectively evaluates the quality of selected thresholds, offering a practical diagnostic tool.
• Simulation studies and insurance data applications demonstrate the method's favorable performance and robustness.
• The approach achieves stable and accurate tail index estimation with reduced sensitivity to threshold choice.