[Paper Review] Forming IDEAS Interactive Data Exploration & Analysis System.

Robert A. Bridges, Maria A. Vincent+4 more|arXiv (Cornell University)|May 24, 2018

Scientific Computing and Data Management4 citations

TL;DR

IDEAS is a visual analytics platform that empowers cyber security analysts to perform data science tasks without algorithmic expertise by enabling interactive configuration of pre-defined data science operations on cyber threat data. The system integrates with existing data stores, applies algorithms like clustering and anomaly detection under the hood, and returns interactive visualizations—demonstrated through real-world use cases such as identifying malicious hosts and detecting advanced persistent threats—thereby reducing manual effort and replacing intuition with evidence-based analysis.

ABSTRACT

Modern cyber security operations collect an enormous amount of logging and alerting data. While analysts have the ability to query and compute simple statistics and plots from their data, current analytical tools are too simple to admit deep understanding. To detect advanced and novel attacks, analysts turn to manual investigations. While commonplace, current investigations are time-consuming, intuition-based, and proving insufficient. Our hypothesis is that arming the analyst with easy-to-use data science tools will increase their work efficiency, provide them with the ability to resolve hypotheses with scientific inquiry of their data, and support their decisions with evidence over intuition. To this end, we present our work to build IDEAS (Interactive Data Exploration and Analysis System). We present three real-world use-cases that drive the system design from the algorithmic capabilities to the user interface. Finally, a modular and scalable software architecture is discussed along with plans for our pilot deployment with a security operation command.

Motivation & Objective

• To address the growing challenge of analyzing massive volumes of cyber security logs that current tools cannot handle effectively.
• To reduce reliance on time-consuming, manual investigations by security analysts that are based on intuition rather than data-driven insights.
• To bridge the gap between data science capabilities and the practical needs of cyber security operations through a configurable, user-friendly platform.
• To enable analysts to test hypotheses scientifically using machine learning and statistical operations without requiring algorithmic expertise.
• To support intra- and inter-organizational intelligence sharing by allowing safe sharing of analytic configurations without exposing sensitive data.

Proposed method

• The system connects to existing cyber data stores, such as those using Elasticsearch, to ingest network flow, host log, and DNS data.
• Analysts select data subsets and apply pre-configured data science operations (e.g., clustering, anomaly detection) via a web-based interface.
• Underlying algorithms—such as k-means clustering and statistical anomaly detection—are executed automatically, with results visualized interactively.
• A microservices-based software architecture using Protocol Buffers and gRPC enables modular, scalable, and language-agnostic component communication.
• The system is containerized using Docker for easy deployment and supports extensibility for future data types and analytic operations.
• The user interface is designed based on real analyst workflows and use cases, with input and output visualizations tailored to security-relevant patterns.

Experimental results

Research questions

• RQ1 How can data science techniques be made accessible to cyber security analysts without requiring algorithmic expertise?
• RQ2 Can interactive, configurable data science operations reduce the time and effort required for manual incident investigations?
• RQ3 How can visual analytics systems support hypothesis testing and evidence-based decision-making in cyber operations?
• RQ4 What architectural patterns enable scalable, modular, and extensible deployment of data science tools in real-world security environments?
• RQ5 Can analytic configurations be shared across organizations without exposing sensitive data?

Key findings

• Analysts reported that manual analysis of ransomware infections required approximately 80 man-hours, highlighting the need for automation.
• The system enables analysts to detect anomalous lateral movement patterns by clustering hosts based on DNS query ratios and network traffic patterns.
• Clustering operations successfully differentiated between web servers (inbound traffic on ports 80/443) and workstations (outbound traffic), providing actionable insights.
• Anomaly detection operations can be configured to baseline normal host behavior and flag deviations, offering a principled alternative to custom scripting.
• The microservices architecture with gRPC and Protocol Buffers enables secure, scalable, and extensible deployment of analytic components.
• Preliminary deployment plans are underway with a large security operations center, with iterative refinement expected post-deployment.