[Paper Review] Modular Learning Component Attacks: Today's Reality, Tomorrow's Challenge

Xinyang Zhang, Yujie Ji+1 more|arXiv (Cornell University)|Aug 25, 2017

Advanced Malware Detection Techniques21 references3 citations

TL;DR

This paper introduces modular learning component (MLC) attacks—specifically logic-bomb attacks—where maliciously modified MLCs cause ML systems to fail predictably under specific conditions. By altering less than 0.33% of parameters with sub-10⁻³ distortion, attackers achieve 100% misdiagnosis success in healthcare ML systems, exploiting high dimensionality and non-convexity inherent in modern models.

ABSTRACT

Many of today's machine learning (ML) systems are not built from scratch, but are compositions of an array of {\em modular learning components} (MLCs). The increasing use of MLCs significantly simplifies the ML system development cycles. However, as most MLCs are contributed and maintained by third parties, their lack of standardization and regulation entails profound security implications. In this paper, for the first time, we demonstrate that potentially harmful MLCs pose immense threats to the security of ML systems. We present a broad class of {\em logic-bomb} attacks in which maliciously crafted MLCs trigger host systems to malfunction in a predictable manner. By empirically studying two state-of-the-art ML systems in the healthcare domain, we explore the feasibility of such attacks. For example, we show that, without prior knowledge about the host ML system, by modifying only 3.3{ extperthousand} of the MLC's parameters, each with distortion below $10^{-3}$, the adversary is able to force the misdiagnosis of target victims' skin cancers with 100\% success rate. We provide analytical justification for the success of such attacks, which points to the fundamental characteristics of today's ML models: high dimensionality, non-linearity, and non-convexity. The issue thus seems fundamental to many ML systems. We further discuss potential countermeasures to mitigate MLC-based attacks and their potential technical challenges.

Motivation & Objective

• To investigate the security risks posed by third-party modular learning components (MLCs) in machine learning systems.
• To demonstrate that maliciously crafted MLCs can trigger predictable system failures through logic-bomb attacks.
• To analyze the feasibility and stealthiness of such attacks in real-world healthcare ML systems.
• To provide analytical justification for the success of MLC-based attacks based on fundamental ML model properties.
• To explore potential countermeasures and their technical challenges.

Proposed method

• The authors implement logic-bomb attacks by subtly modifying parameters in pre-trained MLCs such as word embeddings and deep neural networks.
• They inject malicious behavior into MLCs so that when triggered by specific inputs, the host ML system misclassifies with 100% success rate.
• The modifications are minimal—less than 3.3‰ of parameters altered, with individual changes below 10⁻³ in magnitude.
• The attacks are designed to be evasive: benign and malicious MLCs are nearly indistinguishable, with less than 2.5% difference in performance on non-target inputs.
• The method requires only limited access (e.g., <22% of training data) to the host system’s data, not prior knowledge of the system architecture.
• Theoretical analysis attributes attack success to high dimensionality, non-linearity, and non-convexity in modern ML models.

Experimental results

Research questions

• RQ1 Can maliciously modified modular learning components (MLCs) be used to trigger predictable failures in machine learning systems?
• RQ2 How stealthy can such attacks be, in terms of parameter modification and performance impact on non-target inputs?
• RQ3 What level of access to training data is required for an adversary to successfully craft such attacks?
• RQ4 Why are these attacks effective despite minimal changes to MLC parameters?
• RQ5 What are the potential countermeasures, and what technical challenges do they face?

Key findings

• The authors successfully triggered 100% misdiagnosis in a skin cancer screening system using only 3.3‰ parameter modifications in a neural network MLC.
• The same attack achieved 100% success in a disease early prognosis system, demonstrating broad applicability across healthcare ML systems.
• Malicious MLCs differ from benign ones by less than 3.3‰ of parameters, with each change below 10⁻³, making them highly evasive.
• The performance of the MLC on non-target inputs differs by less than 2.5%, indicating strong generalization and stealth.
• Attackers need access to less than 22% of the host system’s training data, which is often publicly available.
• Theoretical analysis confirms that high dimensionality, non-linearity, and non-convexity of modern ML models underlie the feasibility of such attacks.