[Paper Review] AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction

Bin Liu, Chenxu Zhu+7 more|arXiv (Cornell University)|Mar 25, 2020

Recommender Systems and Techniques30 references36 citations

TL;DR

AutoFIS introduces a two-stage approach to automatically select and prune feature interactions in factorization models for CTR prediction, using continuous architecture parameters and GRDA to learn important interactions during a search stage and re-train with attention-like gating.

ABSTRACT

Learning feature interactions is crucial for click-through rate (CTR) prediction in recommender systems. In most existing deep learning models, feature interactions are either manually designed or simply enumerated. However, enumerating all feature interactions brings large memory and computation cost. Even worse, useless interactions may introduce noise and complicate the training process. In this work, we propose a two-stage algorithm called Automatic Feature Interaction Selection (AutoFIS). AutoFIS can automatically identify important feature interactions for factorization models with computational cost just equivalent to training the target model to convergence. In the \emph{search stage}, instead of searching over a discrete set of candidate feature interactions, we relax the choices to be continuous by introducing the architecture parameters. By implementing a regularized optimizer over the architecture parameters, the model can automatically identify and remove the redundant feature interactions during the training process of the model. In the \emph{re-train stage}, we keep the architecture parameters serving as an attention unit to further boost the performance. Offline experiments on three large-scale datasets (two public benchmarks, one private) demonstrate that AutoFIS can significantly improve various FM based models. AutoFIS has been deployed onto the training platform of Huawei App Store recommendation service, where a 10-day online A/B test demonstrated that AutoFIS improved the DeepFM model by 20.3\% and 20.1\% in terms of CTR and CVR respectively.

Motivation & Objective

• Motivate the importance of selecting meaningful low- and high-order feature interactions for CTR models to improve prediction performance and efficiency.
• Propose AutoFIS, a two-stage method that automatically discovers useful interactions while keeping computational cost close to training to convergence.
• Demonstrate that removing redundant interactions can boost accuracy and reduce inference time across FM-based models.
• Show that learned high-order interactions can improve existing models and deliver online performance gains in a real system.

Proposed method

• Relax the discrete interaction selection problem into a continuous architecture parameter space for each potential interaction.
• Use a gating mechanism where architecture parameters determine whether an interaction is kept during the search stage.
• Apply Batch Normalization on pairwise interaction scores to decouple scale from architecture parameters.
• Optimize architecture parameters and model weights jointly in a one-level optimization setup (not a bi-level approach).
• Use GRDA optimizer to induce sparsity in architecture parameters, effectively pruning unimportant interactions.
• In the re-train stage, fix the kept interactions and re-train with architecture parameters as attention units rather than binary gates.

Experimental results

Research questions

• RQ1 RQ1: Can AutoFIS improve CTR model performance by selecting interactions in factorization models?
• RQ2 RQ2: Can interactions found by AutoFIS transfer to improve other models or reduce inference time?
• RQ3 RQ3: Are the selected interactions truly impactful for model performance?
• RQ4 RQ4: Does AutoFIS provide gains in a live, production recommender system?
• RQ5 RQ5: How do components like Batch Normalization contribute to AutoFIS performance?

Key findings

• AutoFIS can remove a substantial portion of 2nd-order interactions (e.g., 71% on Avazu, 76% on DeepFM for the 2nd order) and still improve AUC compared with baselines.
• AutoFM and AutoDeepFM with selected interactions outperform corresponding baselines on Avazu, Criteo, and private datasets in AUC and/or log loss.
• Online A/B testing in Huawei App Store showed CTR improvement of about 20.3% and CVR improvement of about 20.1% for DeepFM after applying AutoFIS.
• The method enables using a relatively small set of high-quality interactions to boost performance with minimal additional computation.
• Selected 3rd-order interactions (when explored) can yield small but positive gains (e.g., 0.1%–0.2% AUC) without substantial cost.
• Important interactions identified by AutoFIS can also boost the performance of existing state-of-the-art models.