[Paper Review] Execution Integrity with In-Place Encryption

Dean Sullivan, Orlando Arias+4 more|arXiv (Cornell University)|Mar 8, 2017

Security and Verification in Computing19 references3 citations

TL;DR

Scylla introduces a novel in-place encryption scheme that enhances Instruction Set Randomization (ISR) to defend against code-reuse attacks by applying per-basic-block encryption and code diversification, achieving execution integrity without requiring control-flow graph analysis. It resists advanced attacks like JIT-ROP and CROP while maintaining feasible performance overhead of 20% on x86_64 systems.

ABSTRACT

Instruction set randomization (ISR) was initially proposed with the main goal of countering code-injection attacks. However, ISR seems to have lost its appeal since code-injection attacks became less attractive because protection mechanisms such as data execution prevention (DEP) as well as code-reuse attacks became more prevalent. In this paper, we show that ISR can be extended to also protect against code-reuse attacks while at the same time offering security guarantees similar to those of software diversity, control-flow integrity, and information hiding. We present Scylla, a scheme that deploys a new technique for in-place code encryption to hide the code layout of a randomized binary, and restricts the control flow to a benign execution path. This allows us to i) implicitly restrict control-flow targets to basic block entries without requiring the extraction of a control-flow graph, ii) achieve execution integrity within legitimate basic blocks, and iii) hide the underlying code layout under malicious read access to the program. Our analysis demonstrates that Scylla is capable of preventing state-of-the-art attacks such as just-in-time return-oriented programming (JIT-ROP) and crash-resistant oriented programming (CROP). We extensively evaluate our prototype implementation of Scylla and show feasible performance overhead. We also provide details on how this overhead can be significantly reduced with dedicated hardware support.

Motivation & Objective

• Address the growing threat of code-reuse attacks (CRAs), such as ROP and CROP, which bypass traditional defenses like DEP and ASLR.
• Overcome the limitations of existing defenses—software diversity, control-flow integrity, and information hiding—by unifying their strengths into a single, cohesive framework.
• Enable secure execution in the presence of full memory disclosure, including malicious read access to program code, without exposing layout or control flow.
• Provide practical execution integrity within legitimate basic blocks without relying on static or dynamic control-flow graph construction.
• Demonstrate feasibility and performance efficiency of the defense in real-world systems, including support for shared libraries and multitasking environments.

Proposed method

• Apply per-basic-block encryption to randomize and hide the layout of executable code at the binary level, ensuring that only correctly decrypted instructions execute.
• Introduce a hybrid execution model where control flow is restricted to benign entry points of basic blocks, implicitly enforcing control-flow integrity without CFG analysis.
• Use fine-grained code diversification to permute the order and structure of basic blocks, making it infeasible for attackers to reliably predict or reconstruct code layout.
• Implement an execution wrapper that handles runtime decryption and validation of instructions, ensuring only authorized control transfers occur.
• Leverage hardware-assisted decryption support to minimize performance overhead, with prototype evaluation on x86_64 systems including shared library support.
• Combine information-hiding techniques such as execute-no-read (XnR) and code-pointer isolation to prevent leakage of sensitive control structures.

Experimental results

Research questions

• RQ1 Can Instruction Set Randomization be extended to provide strong protection against modern code-reuse attacks like JIT-ROP and CROP?
• RQ2 To what extent can per-basic-block encryption and code diversification jointly prevent disclosure of code layout under full memory read access?
• RQ3 Is it possible to achieve execution integrity within basic blocks without relying on static or dynamic control-flow graph construction?
• RQ4 How does the performance overhead of Scylla compare to existing defenses, and can hardware support significantly reduce this overhead?
• RQ5 Can Scylla withstand advanced bypass techniques such as crash-resistant memory oracles and timing side-channel leaks used in prior attacks?

Key findings

• Scylla successfully resists state-of-the-art code-reuse attacks, including JIT-ROP, CROP, and CFB, by combining per-basic-block encryption with code diversification.
• The prototype implementation incurs only 20% average performance overhead on the SPEC CPU2006 benchmark suite, demonstrating practical feasibility.
• Hardware-assisted decryption support can significantly reduce the performance cost, with the main overhead shifting to code diversification rather than decryption logic.
• Scylla provides complete code layout hiding under full memory disclosure, making it infeasible for attackers to reconstruct the original control flow even with knowledge of code locations.
• The defense is resilient to timing side-channel attacks and memory oracles used in prior breaches, such as those targeting CPI or XnR mechanisms.
• Scylla supports complex runtime environments, including shared libraries and multitasking systems, without compromising security or performance.