[Paper Review] Adversarial Machine Learning -- Industry Perspectives

Ram Shankar Siva Kumar, Magnus Nyström+6 more|arXiv (Cornell University)|Feb 4, 2020

Adversarial Robustness in Machine LearningComputer Science31 references50 citations

TL;DR

The paper reports on a 28-organization industry survey showing widespread gaps in securing ML systems against adversarial attacks and proposes SDL-aligned research directions for industrial ML security.

ABSTRACT

Based on interviews with 28 organizations, we found that industry practitioners are not equipped with tactical and strategic tools to protect, detect and respond to attacks on their Machine Learning (ML) systems. We leverage the insights from the interviews and we enumerate the gaps in perspective in securing machine learning systems when viewed in the context of traditional software security development. We write this paper from the perspective of two personas: developers/ML engineers and security incident responders who are tasked with securing ML systems as they are designed, developed and deployed ML systems. The goal of this paper is to engage researchers to revise and amend the Security Development Lifecycle for industrial-grade software in the adversarial ML era.

Motivation & Objective

• Assess how organizations secure ML systems against adversarial attacks across industries.
• Identify gaps in tooling, processes, and personnel for ML security in production.
• Bridge traditional software SDL with adversarial ML-specific security needs.
• Propose a research agenda to amend Security Development Lifecycle for industrial ML.
• Provide actionable insights for ML developers and security incident responders.

Proposed method

• Conduct interviews with two personas (developers/ML engineers and security incident responders) across 28 organizations.
• Map findings to SDL phases (design/development, deployment, under attack).
• Enumerate security gaps using a structured SDL framework tailored to ML.
• Suggest future research directions and practical tooling alignments with industry practices.

Experimental results

Research questions

• RQ1 What is the current state of securing ML systems against adversarial attacks in industry?
• RQ2 What are the major gaps in tools, practices, and knowledge for ML security across organizations?
• RQ3 How can the Security Development Lifecycle be amended to cover adversarial ML in industry-scale software?
• RQ4 What research directions can enable better design, deployment, monitoring, and incident response for ML systems?

Key findings

• 25 of 28 organizations lack the right tools to secure ML systems and seek guidance.
• Only 6 organizations have headcount dedicated to adversarial ML, all among large organizations or government.
• Most security engineers cannot detect or respond to ML attacks as currently practiced.
• Respondents ranked Poisoning as the top attack concern for their org (10 votes).
• Model Stealing (6 votes) and Model Inversion (4 votes) were also notable concerns.
• There is a mismatch between expectations of ML security in platforms and reality among security analysts and developers.