[Paper Review] Translingual Obfuscation

Pei Wang, Shuai Wang+3 more|arXiv (Cornell University)|Jan 5, 2016

Advanced Malware Detection Techniques67 references4 citations

TL;DR

This paper introduces translingual obfuscation, a novel software protection technique that translates C programs into Prolog to obscure data layout and control flow using Prolog’s unique features—unification and backtracking. The authors implement this in a tool called Babel, demonstrating effective, stealthy obfuscation with only modest performance overhead compared to commercial tools.

ABSTRACT

Program obfuscation is an important software protection technique that prevents attackers from revealing the programming logic and design of the software. We introduce translingual obfuscation, a new software obfuscation scheme which makes programs obscure by "misusing" the unique features of certain programming languages. Translingual obfuscation translates part of a program from its original language to another language which has a different programming paradigm and execution model, thus increasing program complexity and impeding reverse engineering. In this paper, we investigate the feasibility and effectiveness of translingual obfuscation with Prolog, a logic programming language. We implement translingual obfuscation in a tool called BABEL, which can selectively translate C functions into Prolog predicates. By leveraging two important features of the Prolog language, i.e., unification and backtracking, BABEL obfuscates both the data layout and control flow of C programs, making them much more difficult to reverse engineer. Our experiments show that BABEL provides effective and stealthy software obfuscation, while the cost is only modest compared to one of the most popular commercial obfuscators on the market. With BABEL, we verified the feasibility of translingual obfuscation, which we consider to be a promising new direction for software obfuscation.

Motivation & Objective

• To address the growing need for robust software obfuscation techniques that resist reverse engineering and deobfuscation attacks.
• To explore the feasibility of leveraging inherent language features from one programming paradigm to obscure programs written in another.
• To develop a practical obfuscation method that combines language translation with semantic obfuscation effects without relying on virtualization or complex encryption.
• To evaluate the resilience of translingual obfuscation against state-of-the-art reverse engineering and binary differencing tools.
• To demonstrate that obfuscation via cross-paradigm translation can achieve strong security with minimal performance cost.

Proposed method

• Translating selected C functions into Prolog predicates to exploit Prolog’s logic programming paradigm and execution model.
• Leveraging Prolog’s unification and backtracking mechanisms to obscure control flow and data layout in a way that is difficult to analyze statically or dynamically.
• Designing an 'obfuscating' translation scheme that preserves parts of the C memory model while introducing Prolog-specific semantics, creating a hybrid execution model.
• Implementing the approach in a tool called Babel, which selectively translates C code into Prolog while maintaining program semantics.
• Using symbolic execution and binary differencing tools (BinDiff, CoP) to evaluate resilience against reverse engineering and similarity detection.
• Evaluating performance cost by comparing Babel’s output against a leading commercial obfuscator in terms of execution time and binary size.

Experimental results

Research questions

• RQ1 Can translating imperative C code into a logic programming language like Prolog significantly increase the difficulty of reverse engineering?
• RQ2 To what extent does translingual obfuscation resist modern deobfuscation techniques such as inside-out and outside-in analysis?
• RQ3 How effective is translingual obfuscation in evading binary differencing tools like BinDiff and CoP?
• RQ4 What is the performance overhead of translingual obfuscation compared to existing commercial obfuscators?
• RQ5 Can the combination of language-translation and intrinsic language features produce stronger obfuscation than traditional methods?

Key findings

• Babel successfully obfuscates C programs by transforming control flow and data layout using Prolog’s unification and backtracking, making reverse engineering significantly more complex.
• The obfuscated binaries generated by Babel resist detection by state-of-the-art binary differencing tools such as BinDiff and CoP, indicating strong stealth and resilience.
• Deobfuscation techniques targeting virtualization-based obfuscation, such as decode-and-dispatch analysis, are ineffective against Babel’s translingual approach due to the fundamental paradigm shift.
• The performance overhead introduced by Babel is modest and comparable to that of a leading commercial obfuscator, making it practical for real-world use.
• The hybrid execution model created by mixing C memory semantics with Prolog’s logic execution model enhances obfuscation strength beyond what either language alone can achieve.
• Translingual obfuscation demonstrates that leveraging unique language features for obfuscation is a viable and promising new direction in software protection.