[Paper Review] Algorithmic Diversity for Software Security

Michael Stewart|arXiv (Cornell University)|Dec 13, 2013

Security and Verification in Computing5 references3 citations

TL;DR

This paper proposes a deterministic compile-time algorithm for algorithmic diversity in software security, using pattern-based NOP insertion to generate diverse, secure variants of executables. It achieves high entropy (15,010) with minimal performance cost (4% overhead) and low memory bloat, outperforming random or padding-only methods while enabling tunable trade-offs between security, performance, and resource use.

ABSTRACT

Software diversity protects against a modern-day exploits such as code-reuse attacks. When an attacker designs a code-reuse attack on an example executable, it relies on replicating the target environment. With software diversity, the attacker cannot reliably replicate their target. This is a security benefit which can be applied to massive-scale software distribution. When applied to large-scale communities, an invested attacker may perform analysis of samples to improve the chances of a successful attack (M. Franz). We present a general NOP-insertion algorithm which can be expanded and customized for security, performance, or other costs. We demonstrate an improvement in security so that a code-reuse attack based on any one variant has minimal chances of success on another and analyse the costs of this method. Alternately, the variants may be customized to meet performance or memory overhead constraints. Deterministic diversification allows for the flexibility to balance these needs in a way that doesn't exist in a random online method.

Motivation & Objective

• To address the limitations of online, random diversification in large-scale software distribution by introducing a deterministic compile-time approach.
• To reduce the attacker's ability to reliably predict gadget locations across diverse software variants.
• To enable tunable trade-offs between security, performance, and memory overhead in diversified software communities.
• To evaluate diversity using entropy metrics that reflect attacker uncertainty and reduce successful code-reuse attack probabilities.
• To demonstrate that deterministic, pattern-based NOP insertion outperforms random or padding-only methods in diversity and efficiency.

Proposed method

• A deterministic diversification algorithm is applied at compile time, generating a fixed set of diverse variants before distribution.
• The method uses pattern-based NOP insertion—inserting NOPs in structured, non-uniform patterns—to increase entropy and reduce gadget overlap across variants.
• It separates diversification from distribution, allowing consistent, repeatable variant generation independent of runtime conditions.
• Entropy is computed as a measure of attacker uncertainty, with P(c) representing the probability that a functionally equivalent gadget exists at the same location in another variant.
• The framework compares variants using gadget overlap, file size, and performance overhead across multiple transformation types (e.g., NOP05, NOP50, NOP100).
• Performance and file size are measured using 50 diversified SQLite 3.6.9 binaries under standardized benchmarks.

Experimental results

Research questions

• RQ1 Can deterministic, compile-time diversification outperform random, online diversification in generating secure software variants?
• RQ2 To what extent does pattern-based NOP insertion increase entropy and reduce gadget overlap compared to padding-only or random insertion?
• RQ3 How does the performance overhead and memory cost scale with increasing diversity levels in compile-time diversified software?
• RQ4 Can the method be tuned to balance security, performance, and resource constraints without sacrificing diversity?
• RQ5 What is the relationship between diversity metrics (e.g., entropy) and the probability of successful code-reuse attacks?

Key findings

• Pattern-based NOP insertion achieved an entropy score of 15,010 with only 4% performance overhead and 15% file size increase, outperforming padding-only methods.
• The padding-only method achieved the highest entropy (15,019) but with a 2% performance improvement and slightly less disk space than the original.
• Compile-time NOP100 insertion caused 32.18% file size overhead and 6.069% performance overhead, indicating diminishing returns at high insertion rates.
• The pattern-based method produced 600 unique gadgets in a 50-variant community, with minimal performance cost and high diversity.
• Noise-based NOP insertion had 11.31% file size overhead and 8.256% performance cost, indicating high cost for marginal diversity gains.
• The method enables tunable diversity: reducing pad size or noise rate could maintain high entropy at lower cost, suggesting optimization potential.