[Paper Review] Choice by Elimination via Deep Neural Networks

Truyen Tran, Dinh Phung+1 more|arXiv (Cornell University)|Feb 17, 2016

Economic and Environmental Valuation28 references3 citations

TL;DR

This paper proposes Neural Choice by Elimination (NCE), a deep learning framework that models sequential ranking as iterative elimination of the least preferred items, leveraging Gompertz-distributed latent utilities and highway networks for scalable, high-performance learning to rank. Evaluated on the Yahoo! Learning to Rank dataset, NCE outperforms both Plackett-Luce and Rank SVM baselines and achieves competitive performance against gradient boosting trees, demonstrating deep neural networks as a viable alternative in large-scale ranking tasks.

ABSTRACT

We introduce Neural Choice by Elimination, a new framework that integrates deep neural networks into probabilistic sequential choice models for learning to rank. Given a set of items to chose from, the elimination strategy starts with the whole item set and iteratively eliminates the least worthy item in the remaining subset. We prove that the choice by elimination is equivalent to marginalizing out the random Gompertz latent utilities. Coupled with the choice model is the recently introduced Neural Highway Networks for approximating arbitrarily complex rank functions. We evaluate the proposed framework on a large-scale public dataset with over 425K items, drawn from the Yahoo! learning to rank challenge. It is demonstrated that the proposed method is competitive against state-of-the-art learning to rank methods.

Motivation & Objective

• To address limitations in existing learning-to-rank models that prioritize forward selection (e.g., Plackett-Luce), which may over-invest in separating lower-ranked items.
• To propose a novel probabilistic sequential choice model—choice by elimination—where the least worthy item is removed at each step, better reflecting real-world decision-making in competitive or elimination-based contexts.
• To demonstrate that deep neural networks, specifically highway networks, can serve as effective and scalable rank function approximators in large-scale learning-to-rank settings.
• To validate the proposed framework against state-of-the-art methods, including gradient boosting trees, on a large public benchmark dataset.

Proposed method

• The choice by elimination model iteratively removes the least preferred item from the remaining set, with final item ranks being the reverse of the elimination order.
• The model is theoretically grounded in marginalizing out Gompertz-distributed latent utilities, providing a probabilistic foundation for the elimination process.
• Highway networks are used as the rank function approximator, enabling deep, trainable, and gradient-friendly architectures with residual-like information flow.
• Dropout regularization and max-norm constraints are applied to prevent overfitting and stabilize training in deep networks.
• The framework is trained end-to-end using listwise ranking losses, with model optimization via stochastic gradient descent and adaptive learning rate decay.
• The architecture uses shared parameters across elimination steps, enabling parameter efficiency and sequence modeling through recurrent-like structure in the hidden layers.

Experimental results

Research questions

• RQ1 Can a choice-by-elimination model outperform traditional forward-selection models like Plackett-Luce in learning-to-rank tasks?
• RQ2 Does the elimination process, which focuses on removing low-quality items early, lead to better ranking performance, especially for top-ranked items?
• RQ3 Can deep neural networks, particularly highway networks, effectively approximate complex rank functions in large-scale learning-to-rank scenarios?
• RQ4 How does the proposed Neural Choice by Elimination framework compare to state-of-the-art gradient boosting tree methods in terms of ranking performance?

Key findings

• The choice by elimination model achieved a normalized discounted gain (NDCG@5) of 0.680 on the Yahoo!-large dataset, outperforming Plackett-Luce (0.671) and Rank SVM (0.666).
• On the Yahoo!-small dataset, the choice by elimination model achieved an expected reciprocal rank (ERR) of 0.503, surpassing Plackett-Luce (0.495) and Rank SVM (0.488).
• Neural networks with highway architectures achieved an NDCG@1 of 0.719 and NDCG@5 of 0.697 on the Yahoo!-large dataset, outperforming the SGTB gradient boosting baseline (0.705 and 0.681).
• The performance gap between the best-performing models in the Yahoo! L2R challenge is small (1.56% in ERR), yet the NCE model still shows a statistically significant improvement over Plackett-Luce (1.62% difference in ERR).
• The use of dropout and max-norm regularization stabilized training and prevented large gradient jumps, contributing to model robustness.
• The framework demonstrates that deep neural networks can be competitive with tree ensembles in large-scale learning-to-rank, challenging the dominance of gradient boosting in this domain.