[Paper Review] AIQL: Enabling Efficient Attack Investigation from System Monitoring Data

Peng Gao, Xusheng Xiao+5 more|arXiv (Cornell University)|Jun 6, 2018

Network Security and Intrusion Detection28 references40 citations

TL;DR

AIQL introduces a domain-specific query language and optimized storage/execution engine to enable fast, concise attack investigations over large-scale system monitoring data, outperforming SQL-based and graph-based systems by orders of magnitude.

ABSTRACT

The need for countering Advanced Persistent Threat (APT) attacks has led to the solutions that ubiquitously monitor system activities in each host, and perform timely attack investigation over the monitoring data for analyzing attack provenance. However, existing query systems based on relational databases and graph databases lack language constructs to express key properties of major attack behaviors, and often execute queries inefficiently since their semantics-agnostic design cannot exploit the properties of system monitoring data to speed up query execution. To address this problem, we propose a novel query system built on top of existing monitoring tools and databases, which is designed with novel types of optimizations to support timely attack investigation. Our system provides (1) domain-specific data model and storage for scaling the storage, (2) a domain-specific query language, Attack Investigation Query Language (AIQL) that integrates critical primitives for attack investigation, and (3) an optimized query engine based on the characteristics of the data and the semantics of the queries to efficiently schedule the query execution. We deployed our system in NEC Labs America comprising 150 hosts and evaluated it using 857 GB of real system monitoring data (containing 2.5 billion events). Our evaluations on a real-world APT attack and a broad set of attack behaviors show that our system surpasses existing systems in both efficiency (124x over PostgreSQL, 157x over Neo4j, and 16x over Greenplum) and conciseness (SQL, Neo4j Cypher, and Splunk SPL contain at least 2.4x more constraints than AIQL).

Motivation & Objective

• Motivate timely attack investigation over large-scale system monitoring data to counter Advanced Persistent Threats (APTs).
• Propose a domain-specific data model and storage strategy to scale monitoring data.
• Design Aiql, a domain-specific query language for attack behaviors.
• Develop a specialized query engine with optimizations tailored to system monitoring data.
• Evaluate Aiql against traditional SQL/Graph systems on real-world data."],
• method1
• method2

Proposed method

• Propose a domain-specific data model and storage that partitions data temporally and spatially to enable parallel processing.
• Introduce Aiql with constructs for multievent patterns, dependency tracking, and anomaly queries.
• Develop a query engine that synthesizes SQL queries per event pattern and uses relationship-aware scheduling.
• Implement time-window partitioning and distribution across MPP databases (e.g., Greenplum).
• Use a two-layer optimization: (a) event-pattern pruning power, (b) temporal/spatial parallelization to execute sub-queries in parallel.
• Provide context-aware syntax shortcuts and attribute inference to simplify query authoring.

Experimental results

Research questions

• RQ1 Can Aiql express key attack behaviors (multi-step attacks, dependencies, anomalies) succinctly and effectively?
• RQ2 Do domain-specific optimizations in storage and query scheduling substantially improve attack-investigation performance on large monitoring datasets?
• RQ3 How does Aiql compare to SQL, Cypher, and Splunk SPL in efficiency and conciseness for typical APT-related queries?
• RQ4 Can the system scale to enterprise-scale deployments with hundreds of hosts and billions of events?

Key findings

• Aiql achieves significant end-to-end efficiency gains, surpassing PostgreSQL by 124x and Neo4j by 157x on real attack scenarios.
• Scheduling with relationship-based pruning and temporal/spatial parallelization yields 40x faster scheduling over PostgreSQL and 16x over Greenplum.
• Aiql queries are more concise than SQL, Neo4j Cypher, and Splunk SPL, requiring at least 2.4x fewer constraints, 3.1x fewer words, and 4.7x fewer characters.
• The evaluation used 857 GB of real system monitoring data (16 days, 2.5 billion events) across 150 hosts, demonstrating practicality for enterprise-scale deployments.
• Aiql supports three attack-behavior primitives (multi-step, dependency tracking, and anomaly-based queries) and integrates domain-specific data modeling to optimize storage and query execution.