[Paper Review] Systematic Attack Surface Reduction For Deployed Sentiment Analysis Models

Josh Kalin, David Noever+1 more|arXiv (Cornell University)|Jun 19, 2020

Adversarial Robustness in Machine Learning21 references4 citations

TL;DR

This paper introduces the BAD (Build, Attack, and Defend) Architecture, a systematic framework for reducing attack surfaces in deployed sentiment analysis models. By establishing baselines, probing vulnerabilities using the Jigsaw Toxic Bias dataset, and applying targeted defenses, the approach demonstrates that adversarial threats in production ML systems are detectable and preventable through structured, repeatable hardening procedures.

ABSTRACT

This work proposes a structured approach to baselining a model, identifying attack vectors, and securing the machine learning models after deployment. This method for securing each model post deployment is called the BAD (Build, Attack, and Defend) Architecture. Two implementations of the BAD architecture are evaluated to quantify the adversarial life cycle for a black box Sentiment Analysis system. As a challenging diagnostic, the Jigsaw Toxic Bias dataset is selected as the baseline in our performance tool. Each implementation of the architecture will build a baseline performance report, attack a common weakness, and defend the incoming attack. As an important note: each attack surface demonstrated in this work is detectable and preventable. The goal is to demonstrate a viable methodology for securing a machine learning model in a production setting.

Motivation & Objective

• To address the growing risk of adversarial attacks on deployed machine learning models, particularly in real-world NLP systems like sentiment analysis.
• To develop a repeatable, systematic methodology for identifying and mitigating attack vectors post-deployment.
• To demonstrate that adversarial vulnerabilities in production models are detectable and preventable through structured hardening.
• To evaluate the effectiveness of the proposed framework using a real-world benchmark dataset, the Jigsaw Toxic Bias dataset.
• To provide a practical, production-ready framework for securing machine learning models beyond initial training.

Proposed method

• The BAD Architecture is structured into three phases: Build (establishing a performance baseline), Attack (systematically probing for vulnerabilities), and Defend (applying countermeasures to harden the model).
• The approach uses the Jigsaw Toxic Bias dataset as a diagnostic benchmark to evaluate model robustness under adversarial conditions.
• Attack vectors are identified by analyzing model behavior under perturbed inputs, simulating real-world adversarial examples.
• Defensive mechanisms are applied based on identified weaknesses, including input sanitization and model-level hardening techniques.
• The framework is evaluated on two distinct implementations to quantify the adversarial lifecycle and effectiveness of mitigation strategies.
• Performance is measured via baseline reports, attack success rates, and post-defense resilience metrics.

Experimental results

Research questions

• RQ1 How can a systematic, repeatable process be established to identify and reduce attack surfaces in deployed sentiment analysis models?
• RQ2 What are the most prevalent and exploitable vulnerabilities in black-box sentiment analysis systems under real-world data distributions?
• RQ3 To what extent can adversarial attacks on deployed models be detected and prevented using structured defense mechanisms?
• RQ4 How does the BAD Architecture improve model resilience across different implementation variants in a production-like setting?
• RQ5 Can the proposed methodology be generalized to other NLP models beyond sentiment analysis?

Key findings

• The BAD Architecture successfully identifies and mitigates multiple exploitable attack vectors in deployed sentiment analysis models through systematic probing and defense.
• Adversarial attacks on the Jigsaw Toxic Bias dataset were consistently detectable and preventable using the proposed defense mechanisms.
• The framework demonstrated measurable improvements in model robustness after defense deployment, with reduced attack success rates across both implementations.
• The study confirms that post-deployment hardening is feasible and effective, even for black-box models with limited introspection.
• The approach provides a repeatable, auditable methodology for securing ML models in production, reducing the risk of adversarial exploitation.
• The results validate that systematic attack surface reduction is a viable and necessary practice for securing real-world machine learning systems.