[Paper Review] How a simple bug in ML compiler could be exploited for backdoors?

Baptiste David|arXiv (Cornell University)|Nov 27, 2018

Security and Verification in Computing1 references4 citations

TL;DR

This paper exposes a zero-day bug in Microsoft's MASM compiler that silently miscompiles conditional logic involving the NOT operator, enabling attackers to insert stealthy backdoors in compiled binaries without altering source code. The flaw allows malicious code to execute under specific conditions while preserving correct behavior in normal cases, demonstrating how compiler vulnerabilities can bypass traditional code audits and enable undetectable, long-term backdoors in critical software.

ABSTRACT

Whenever a bug occurs in a program, software developers assume that the code is flawed, not the compiler. In fact, if compilers should be correct, they are just normal software with their own bugs. Hard to find, errors in them have significant impact, since it could result to vulnerabilities, especially when they silently miscompile a critical application. Using assembly language to write such software is quite common, especially when time constraint is involved in such program. This paper exposes a bug found in Microsoft Macro Assembler (ml for short) compiler, developed by Microsoft since 1981. This assembly has the characteristics to get high level-like constructs and high level-like records which help the developer to write assembly code. It is in the management of one of this level-like construct the bug has been found. This study aims to show how a compiler-bug can be audited and possibly corrected. For application developers, it shows that even old and mature compilers can present bugs. For security researcher, it shows possibilities to hide some unexpected behavior in software with a clear and officially non-bogus code. It highlights opportunities for including stealth backdoors even in open-source software.

Motivation & Objective

• To expose a previously unknown, silent compiler bug in Microsoft's MASM (ml) compiler that miscompiles conditional logic.
• To demonstrate how such a bug can be exploited by an attacker to insert stealthy backdoors in compiled software without modifying source code.
• To highlight the risk of relying solely on source code audits for security, especially in critical systems compiled with mature but flawed compilers.
• To urge the security community to consider compilers as potential attack surfaces and to audit both source code and compiler behavior.
• To illustrate the feasibility of creating undetectable, long-term backdoors through compiler-level vulnerabilities, akin to Ken Thompson's infamous compiler Trojan.

Proposed method

• Identified a syntax-level bug in MASM’s macro assembly handling of the NOT operator in conditional expressions.
• Constructed test cases using MASM syntax that should evaluate to logical negation but were compiled into incorrect x86 machine code.
• Disassembled the compiled output using tools like WinDbg to verify that the generated assembly code did not match the intended logic.
• Demonstrated that the miscompiled code could result in unintended control flow, such as skipping expected jumps or executing unintended branches.
• Reported the issue to Microsoft, leading to a fix that now emits a syntax error for such constructs to prevent silent miscompilation.
• Used reverse engineering and binary analysis to confirm the presence of the logic error and its exploitable nature in real compiled binaries.

Experimental results

Research questions

• RQ1 How can a compiler bug in a widely used, long-standing assembler like MASM lead to silent miscompilation of conditional logic?
• RQ2 In what ways can such a miscompilation be exploited to insert a stealthy, undetectable backdoor in compiled software?
• RQ3 Why is it difficult for developers and auditors to detect such backdoors when source code appears correct and open-source?
• RQ4 What are the implications of relying on compiler correctness when evaluating the security of critical software, especially in firmware and system-level code?
• RQ5 How can a compiler bug be leveraged to maintain persistence across software updates without modifying the source code?

Key findings

• A silent miscompilation occurs in Microsoft's MASM compiler when using the NOT operator in complex conditional constructs, leading to incorrect control flow in the generated binary.
• The bug causes the compiler to generate incorrect x86 assembly instructions that do not match the intended logical behavior of the source code, even though the syntax is valid.
• The miscompilation is undetectable through static source code analysis, as the code appears correct and follows standard MASM conventions.
• The vulnerability allows an attacker to insert a backdoor that executes only under specific, controlled conditions, making it extremely stealthy and hard to detect.
• Microsoft patched the issue by introducing a syntax error (A2154) for such constructs, preventing further silent miscompilation without breaking backward compatibility.
• The attack surface is particularly dangerous in critical systems such as firmware, kernels, and security tools, where source code is often audited but compiler behavior is not.