[Paper Review] On the effectiveness of virtualization-based security

Francesco Gadaleta, Raoul Strackx+3 more|Open Repository and Bibliography (University of Liège)|May 22, 2014

Security and Verification in Computing14 references3 citations

TL;DR

This paper evaluates virtualization-based security by presenting Hello Rootkitty, a hypervisor-level invariance-enforcing framework that detects kernel-level rootkits without modifying guest OS or applications, and explores hardware-assisted provable security via software-based TPMs (µTPMs) in TrustVisor. It demonstrates low overhead and strong isolation, showing virtualization enables effective, lightweight malware detection and formally provable protection for sensitive code and data.

ABSTRACT

Protecting commodity operating systems and applications against malware and targeted attacks has proven to be difficult. In recent years, virtualization has received attention from security researchers who utilize it to harden existing systems and provide strong security guarantees. This has lead to interesting use cases such as cloud computing where possibly sensitive data is processed on remote, third party systems. The migration and processing of data in remote servers, poses new technical and legal questions, such as which security measures should be taken to protect this data or how can it be proven that execution of code wasn't tampered with. In this paper we focus on technological aspects. We discuss the various possibilities of security within the virtualization layer and we use as a case study \HelloRootkitty{}, a lightweight invariance-enforcing framework which allows an operating system to recover from kernel-level attacks. In addition to \HelloRootkitty{}, we also explore the use of special hardware chips as a way of further protecting and guaranteeing the integrity of a virtualized system.

Motivation & Objective

• To address the challenge of protecting commodity operating systems from kernel-level malware and targeted attacks in cloud and enterprise environments.
• To explore how virtualization can provide strong isolation and security guarantees without requiring modifications to existing operating systems or applications.
• To evaluate the effectiveness of hypervisor-based invariance enforcement for detecting rootkits and the feasibility of hardware-assisted, formally provable security for sensitive software components.
• To compare lightweight, non-invasive security mechanisms like Hello Rootkitty with more rigorous, code-modification-dependent approaches such as TrustVisor for provable integrity.

Proposed method

• Hello Rootkitty operates as an in-hypervisor framework that continuously monitors kernel data structures for unauthorized modifications, enforcing invariants to detect rootkit activity.
• The framework performs runtime checks on kernel memory and control structures, triggering alerts or repair actions upon detecting deviations from expected states.
• TrustVisor uses a hypervisor to isolate security-sensitive code and data, enforcing strict access controls to prevent malware from reading or writing protected memory regions.
• It implements software-based TPMs (µTPMs) that emulate hardware TPM functionality on the main CPU, reducing reliance on slow hardware TPMs and improving performance.
• The system uses a hardware TPM to measure and seal long-term secrets during boot, ensuring that only correctly loaded security modules can access protected data.
• Security-sensitive applications are partitioned into trusted (protected) and untrusted (non-protected) components, with the trusted part running in a hardened, isolated environment.

Experimental results

Research questions

• RQ1 Can a hypervisor-based invariance-enforcing framework detect kernel-level rootkits without modifying the guest OS or applications?
• RQ2 What is the performance overhead of running such a detection mechanism in a virtualized environment?
• RQ3 Can software-based TPMs (µTPMs) provide performance improvements over hardware TPMs while maintaining strong security guarantees?
• RQ4 To what extent can virtualization and hardware support (e.g., TPM) be combined to achieve formally provable isolation of sensitive code and data?
• RQ5 How do lightweight, non-invasive security mechanisms compare to more complex, code-modification-based approaches in terms of security coverage and practical deployment?

Key findings

• Hello Rootkitty successfully detects kernel-level rootkits by enforcing invariants on kernel data structures, with minimal performance overhead and no need for guest OS modifications.
• The framework alerts administrators upon detecting anomalies and can, in some cases, initiate kernel repair, significantly improving resilience against rootkit attacks.
• TrustVisor achieves strong isolation of sensitive code and data by leveraging a hypervisor to control access and prevent unauthorized memory operations from malware.
• The use of µTPMs reduces the performance cost of cryptographic operations by eliminating repeated calls to the slow hardware TPM, resulting in measurable speedups over systems like Flicker.
• Hardware support via TPM enables formal verification of boot integrity and secure sealing of long-term secrets, ensuring that only correctly booted security modules can access protected data.
• The two approaches—Hello Rootkitty for detection and TrustVisor for provable isolation—are complementary, enabling both malware detection and strong data protection in multi-tenant cloud environments.