[Paper Review] Trimming the Hill estimator: robustness, optimality and adaptivity

Shrijita Bhattacharya, Michalis Kallitsis+1 more|arXiv (Cornell University)|May 8, 2017

Financial Risk and Volatility Modeling20 references3 citations

TL;DR

This paper introduces a robust, optimally weighted trimmed Hill estimator for heavy-tailed distributions that excludes the most extreme order statistics to mitigate contamination effects. It achieves finite-sample efficiency under the Pareto model, ensures asymptotic normality under second-order conditions, and proposes a data-driven trimming procedure via a trimmed Hill plot and sequential testing, enabling adaptivity to unknown contamination levels while identifying extreme outliers.

ABSTRACT

We introduce a trimmed version of the Hill estimator for the index of a heavy-tailed distribution, which is robust to perturbations in the extreme order statistics. In the ideal Pareto setting, the estimator is essentially finite-sample efficient among all unbiased estimators with a given strict upper break-down point. For general heavy-tailed models, we establish the asymptotic normality of the estimator under second order conditions and discuss its minimax optimal rate in the Hall class. We introduce the so-called trimmed Hill plot, which can be used to select the number of top order statistics to trim. We also develop an automatic, data-driven procedure for the choice of trimming. This results in a new type of robust estimator that can {\em adapt} to the unknown level of contamination in the extremes. As a by-product we also obtain a methodology for identifying extreme outliers in heavy tailed data. The competitive performance of the trimmed Hill and adaptive trimmed Hill estimators is illustrated with simulations.

Motivation & Objective

• To address the lack of robust, optimal, and adaptive tail index estimation in the presence of extreme value contamination.
• To develop a trimmed Hill estimator that maintains finite-sample efficiency under the ideal Pareto model while being robust to perturbations in the top order statistics.
• To establish asymptotic normality and minimax optimality of the trimmed estimator under second-order regular variation conditions.
• To propose a data-driven method for selecting the trimming parameter $k_0$ using a trimmed Hill plot and sequential testing.
• To enable automatic adaptation to unknown levels of contamination in extreme values and detect extreme outliers.

Proposed method

• Proposes a weighted trimmed Hill estimator $\widehat{\xi}^{\rm trim}_{k_0,k}(n) = \sum_{i=k_0+1}^{k} c_{k_0,k}(i) \log\left(\frac{X_{(n-i+1,n)}}{X_{(n-k,n)}}\right)$ with optimal weights $c_{k_0,k}(i)$ derived as the best linear unbiased estimator under the Pareto model.
• Derives the optimal weights $c_{k_0,k}(i)$ using the inverse of the covariance matrix of the log-ratios of extreme order statistics, ensuring finite-sample efficiency among unbiased estimators with fixed upper breakdown point.
• Introduces the trimmed Hill plot as a visual tool to guide the selection of the trimming parameter $k_0$ by plotting the trimmed estimator across different $k_0$ values for a fixed $k$.
• Develops a weighted sequential testing procedure to automatically select $k_0$ by testing for structural breaks in the weighted ratios of the trimmed estimators, leveraging the joint asymptotic distribution of the optimal trimmed estimators.
• Uses the asymptotic joint distribution of the trimmed estimators to construct a test statistic that detects anomalies in the extreme order statistics, enabling outlier detection.
• Establishes minimax rate-optimality of the trimmed estimator in the Hall class and proves asymptotic normality under second-order conditions on the slowly varying function $\ell(x)$.

Experimental results

Research questions

• RQ1 Can a trimmed version of the Hill estimator be constructed to achieve finite-sample efficiency among unbiased estimators with a fixed upper breakdown point under the Pareto model?
• RQ2 How can the trimmed Hill estimator maintain the same rate of convergence as the classic Hill estimator when $k_0 = o(k)$?
• RQ3 What is the minimax optimal rate of convergence for the trimmed Hill estimator in the Hall class of heavy-tailed distributions?
• RQ4 Can a data-driven, automatic procedure be developed to select the trimming parameter $k_0$ without prior knowledge of contamination levels?
• RQ5 Can the trimmed estimator be used to detect and identify extreme outliers in heavy-tailed data?

Key findings

• The proposed trimmed Hill estimator achieves finite-sample efficiency among all unbiased estimators with a fixed strong upper breakdown point under the Pareto model.
• The estimator maintains the same rate of convergence as the classic Hill estimator as long as $k_0 = o(k)$, ensuring no loss in efficiency under mild conditions.
• The trimmed Hill estimator is minimax rate-optimal in the Hall class of heavy-tailed distributions, matching the optimal convergence rate for tail index estimation.
• The trimmed Hill plot effectively visualizes the stability of the estimator across different $k_0$ values, aiding in the selection of the trimming parameter.
• The proposed sequential testing procedure for $k_0$ selection is asymptotically valid and ensures $k^\delta \max_{0 \leq k_0 < h(k)} |T_{k_0,k} - T^*_{k_0,k}| \stackrel{P}{\to} 0$, confirming consistency of the data-driven trimming choice.
• The method successfully identifies extreme outliers in heavy-tailed data by detecting statistically significant deviations in the top order statistics, as demonstrated through simulations.