[Paper Review] Field-wise Learning for Multi-field Categorical Data

Zhibin Li, Jian Zhang+3 more|arXiv (Cornell University)|Dec 1, 2020

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes field-wise learning for multi-field categorical data, where individual linear models are trained per category within each field using variance and low-rank constraints to improve generalization and interpretability. The method achieves state-of-the-art performance on large-scale datasets like Avazu and Criteo, with superior logloss and robustness to over-parameterization.

ABSTRACT

We propose a new method for learning with multi-field categorical data. Multi-field categorical data are usually collected over many heterogeneous groups. These groups can reflect in the categories under a field. The existing methods try to learn a universal model that fits all data, which is challenging and inevitably results in learning a complex model. In contrast, we propose a field-wise learning method leveraging the natural structure of data to learn simple yet efficient one-to-one field-focused models with appropriate constraints. In doing this, the models can be fitted to each category and thus can better capture the underlying differences in data. We present a model that utilizes linear models with variance and low-rank constraints, to help it generalize better and reduce the number of parameters. The model is also interpretable in a field-wise manner. As the dimensionality of multi-field categorical data can be very high, the models applied to such data are mostly over-parameterized. Our theoretical analysis can potentially explain the effect of over-parametrization on the generalization of our model. It also supports the variance constraints in the learning objective. The experiment results on two large-scale datasets show the superior performance of our model, the trend of the generalization error bound, and the interpretability of learning outcomes. Our code is available at https://github.com/lzb5600/Field-wise-Learning.

Motivation & Objective

• To address the challenge of learning universal models that struggle with high heterogeneity in multi-field categorical data.
• To improve model generalization and reduce overfitting in high-dimensional, sparse categorical feature spaces.
• To enable interpretable, field-wise model interpretation by learning category-specific models per field.
• To theoretically justify the role of over-parameterization and constraints in generalization performance.
• To develop a scalable and efficient learning framework that avoids complex feature engineering.

Proposed method

• The method trains one-to-one field-focused linear models, where each categorical category in a field gets its own model.
• It applies variance constraints to weight vectors within each field to prevent deviation from the population mean.
• Low-rank constraints are imposed on the weight matrices of each field to reduce parameter count and encourage feature correlation.
• The model uses a regularized objective function combining variance and low-rank penalties to enhance generalization.
• Theoretical analysis derives a generalization error bound that supports the effectiveness of variance constraints and the benefit of mild over-parameterization.
• The framework is extensible to more complex models beyond linear models, maintaining interpretability.

Experimental results

Research questions

• RQ1 Can field-wise learning with category-specific models improve generalization on multi-field categorical data compared to universal models?
• RQ2 How do variance and low-rank constraints affect model performance and generalization in high-dimensional, sparse settings?
• RQ3 Does over-parameterization in field-wise models still lead to better generalization, and can this be theoretically justified?
• RQ4 Can the model provide meaningful, interpretable insights at the field and category level?
• RQ5 How does the model scale in terms of training time and parameter efficiency on large-scale datasets?

Key findings

• The proposed model achieved significantly lower logloss than all baselines on both Avazu and Criteo datasets, with improvements of at least 0.001, which is considered significant.
• The model demonstrated superior performance even with a relatively large number of parameters, indicating that over-parameterization is beneficial in this framework.
• The generalization error bound showed an initial decrease followed by an increase with growing parameter count, supporting the benefit of mild over-parameterization.
• The field-wise interpretation revealed that fields with high variance in category-specific model weights (e.g., Country) represent more distinct data groups, enabling targeted analysis.
• The model achieved fast convergence and efficient training, completing within 2–3 hours on both large-scale datasets.
• The variance and low-rank constraints were empirically and theoretically justified, improving generalization and reducing model complexity.