[Paper Review] FROTE: Feedback Rule-Driven Oversampling for Editing Models

Öznur Alkan, Dennis Wei+4 more|arXiv (Cornell University)|Jan 4, 2022

Data Quality and Management4 citations

TL;DR

FROTE is a novel data augmentation technique that edits machine learning models for tabular data by oversampling synthetic instances based on user-provided feedback rules, enabling model decision boundaries to align with updated policies. It improves model performance with minimal accuracy drop on unaffected data segments, outperforming state-of-the-art methods in handling rule-based edits via model-agnostic pre-processing.

ABSTRACT

Machine learning models may involve decision boundaries that change over time due to updates to rules and regulations, such as in loan approvals or claims management. However, in such scenarios, it may take time for sufficient training data to accumulate in order to retrain the model to reflect the new decision boundaries. While work has been done to reinforce existing decision boundaries, very little has been done to cover these scenarios where decision boundaries of the ML models should change in order to reflect new rules. In this paper, we focus on user-provided feedback rules as a way to expedite the ML models update process, and we formally introduce the problem of pre-processing training data to edit an ML model in response to feedback rules such that once the model is retrained on the pre-processed data, its decision boundaries align more closely with the rules. To solve this problem, we propose a novel data augmentation method, the Feedback Rule-Based Oversampling Technique. Extensive experiments using different ML models and real world datasets demonstrate the effectiveness of the method, in particular the benefit of augmentation and the ability to handle many feedback rules.

Motivation & Objective

• To address the challenge of updating ML models when decision boundaries shift due to new rules, such as in loan approvals or claims management.
• To reduce reliance on costly manual relabeling or slow data collection by incorporating expert feedback directly into model training.
• To propose a model-agnostic, pre-processing solution that uses data augmentation to encode feedback rules into the training data.
• To enable efficient, transparent, and auditable model editing through interpretable Boolean feedback rules.
• To minimize performance degradation on data not affected by feedback rules while improving alignment with new policy boundaries.

Proposed method

• FROTE begins by modifying existing training data according to user feedback rules, including relabeling or removing instances if permitted.
• It then applies a rule-aware oversampling strategy inspired by SMOTE, generating synthetic instances near minority or rule-violating regions.
• The method selects base instances using either random sampling or an informed strategy (IP) that considers feature space proximity and rule constraints.
• Synthetic instances are generated by interpolating between selected base instances and their k-nearest neighbors, ensuring alignment with feedback rules.
• The augmented dataset is retrained with any standard classifier, making FROTE model-agnostic and applicable to black-box models.
• The approach balances data augmentation with model fidelity by minimizing changes to unaffected data segments, preserving overall accuracy.

Experimental results

Research questions

• RQ1 Can feedback rules be effectively encoded into training data via data augmentation to update model decision boundaries without retraining from scratch?
• RQ2 How does FROTE compare to state-of-the-art methods in aligning model predictions with user-specified feedback rules?
• RQ3 To what extent does data augmentation improve model performance when only a small number of instances are available in regions requiring policy updates?
• RQ4 Does FROTE maintain model performance on unaffected data segments while improving alignment with new rules?
• RQ5 How do different base instance selection strategies (random vs. IP) impact the effectiveness and efficiency of the augmentation process?

Key findings

• FROTE significantly improves mean ranking accuracy (MRA) and F-score across multiple datasets and models, with MRA improvements ranging from 0.002 to 0.076 across different configurations.
• The method achieves better performance than state-of-the-art approaches, particularly in handling multiple feedback rules and sparse data regions.
• The IP-based base instance selection strategy generally adds fewer synthetic instances than random selection while maintaining comparable or better performance.
• There is a clear inflection point in augmentation size beyond which model performance begins to degrade, indicating a trade-off between rule alignment and data quality.
• Both random and IP selection strategies improve MRA without significantly reducing F-score, suggesting robustness to base instance selection under varying conditions.
• The approach supports transparent, auditable model updates through interpretable Boolean feedback rules, enabling governance and lineage tracking.