[Paper Review] Ask Me Anything: A Conversational Interface to Augment Information Security Workers

Bobby Filar, Richard J. Seymour+1 more|arXiv (Cornell University)|Jul 18, 2017

Persona Design and Applications7 references3 citations

TL;DR

This paper introduces Artemis, a conversational AI interface for endpoint detection and response (EDR) tools that enables security analysts—especially junior or inexperienced users—to query and analyze threat data using natural language. By leveraging NLP to map diverse user terminology to canonical search terms and guiding users through triage and hunting workflows, Artemis reduces reliance on complex query languages and improves accessibility, ultimately decreasing cognitive load and accelerating threat response.

ABSTRACT

Security products often create more problems than they solve, drowning users in alerts without providing the context required to remediate threats. This challenge is compounded by a lack of experienced personnel and security tools with complex interfaces. These interfaces require users to become domain experts or rely on repetitive, time consuming tasks to turn this data deluge into actionable intelligence. In this paper we present Artemis, a conversational interface to endpoint detection and response (EDR) event data. Artemis leverages dialog to drive the automation of complex tasks and reduce the need to learn a structured query language. Designed to empower inexperienced and junior security workers to better understand their security environment, Artemis provides an intuitive platform to ask questions of alert data as users are guided through triage and hunt workflows. In this paper, we will discuss our user-centric design methodology, feedback from user interviews, and the design requirements generated upon completion of our study. We will also present core functionality, findings from scenario-based testing, and future research for the Artemis platform.

Motivation & Objective

• To address the growing shortage of skilled information security professionals by creating a more accessible interface for EDR tools.
• To reduce alert fatigue and cognitive overload caused by data deluge in security operations centers (SOCs).
• To lower the barrier to entry for inexperienced analysts by replacing complex query languages with natural language interactions.
• To support both novice and expert users through adaptive, context-aware dialogue that guides threat triage and hunting workflows.
• To develop a user-centered design approach that captures real-world analyst workflows and integrates feedback into the platform’s core functionality.

Proposed method

• Conducting user interviews and observational studies across diverse security teams, including red/blue teams and traditional SOCs, to understand workflows and pain points.
• Employing a user-centered design methodology with three phases: discovery, concepting, and prototyping/user testing.
• Implementing a conversational interface that maps natural language inputs to canonical search terms using NLP techniques and entity classification.
• Designing the system to support context switching and maintain analytic flow by preserving conversation state and intent.
• Applying the Robustness Principle (Postel’s Law) to accept diverse user terminology (e.g., 'box', 'machine', 'endpoint') and map them to a shared canonical vocabulary.
• Exposing an API to enable future integration with chat platforms like Slack and Cortana, and supporting active learning for continuous model improvement.

Experimental results

Research questions

• RQ1 How can a conversational interface reduce the cognitive and technical burden of EDR data analysis for inexperienced security analysts?
• RQ2 What design patterns and NLP strategies enable natural language queries to effectively map to complex security data operations?
• RQ3 How can a dialog-based system support both novice users and experienced analysts in triage and threat-hunting workflows?
• RQ4 What collaboration mechanisms are most effective for sharing investigations and maintaining team alignment in security operations?
• RQ5 How can feedback mechanisms improve the accuracy and adaptability of natural language understanding in security-specific contexts?

Key findings

• The conversational interface significantly reduced the need for users to learn complex query languages, enabling faster and more intuitive access to EDR data.
• Users were able to express themselves using diverse terminology (e.g., 'box', 'machine', 'endpoint'), and the system successfully mapped these to a canonical set, improving usability.
• The system maintained analytic flow during context switching, minimizing disruption to the user’s threat investigation process.
• Users expressed strong interest in collaboration features, particularly the ability to @ teammates to share investigations and receive notifications.
• The platform’s API enables integration with external chat systems, paving the way for future chat-based security workflows.
• Active learning feedback mechanisms—such as thumbs up/down or performance ratings—were identified as viable methods to improve NLP model accuracy over time.