[Paper Review] Motion-to-Response Content Generation via Multi-Agent AI System with Real-Time Safety Verification

HyeYoung Lee|arXiv (Cornell University)|Jan 20, 2026

Emotion and Mood Recognition0 citations

TL;DR

The paper presents a four-agent system that converts audio-based emotions into safe, real-time, age-appropriate response content with a safety verification loop and on-device deployment.

ABSTRACT

This paper proposes a multi-agent artificial intelligence system that generates response-oriented media content in real time based on audio-derived emotional signals. Unlike conventional speech emotion recognition studies that focus primarily on classification accuracy, our approach emphasizes the transformation of inferred emotional states into safe, age-appropriate, and controllable response content through a structured pipeline of specialized AI agents. The proposed system comprises four cooperative agents: (1) an Emotion Recognition Agent with CNN-based acoustic feature extraction, (2) a Response Policy Decision Agent for mapping emotions to response modes, (3) a Content Parameter Generation Agent for producing media control parameters, and (4) a Safety Verification Agent enforcing age-appropriateness and stimulation constraints. We introduce an explicit safety verification loop that filters generated content before output, ensuring compliance with predefined rules. Experimental results on public datasets demonstrate that the system achieves 73.2% emotion recognition accuracy, 89.4% response mode consistency, and 100% safety compliance while maintaining sub-100ms inference latency suitable for on-device deployment. The modular architecture enables interpretability and extensibility, making it applicable to child-adjacent media, therapeutic applications, and emotionally responsive smart devices.

Motivation & Objective

• Bridge emotion recognition and content generation with explicit policy and safety layers.
• Enable interpretable, modular, and on-device emotion-to-response content generation.
• Ensure age-appropriateness and controlled stimulation via rule-based safety verification.
• Demonstrate real-time performance and privacy-preserving, edge-focused deployment.

Proposed method

• Four cooperative agents process input: Emotion Recognition, Response Policy Decision, Content Parameter Generation, and Safety Verification.
• CNN-based acoustic feature extraction and softmax-based emotion classification for e* in C emotion categories.
• Policy mapping from emotion and arousal to discrete response modes via a decision tree.
• Content parameter generation predicting multi-modal media controls (audio, visual, text) from the chosen mode.
• Explicit safety verification using rule-based constraints with a regeneration loop if rules are violated.

Figure 1: Overall architecture of the proposed multi-agent system for emotion-to-response content generation. The system comprises four specialized agents operating sequentially: Emotion Recognition Agent, Response Policy Decision Agent, Content Parameter Generation Agent, and Safety Verification Ag

Experimental results

Research questions

• RQ1 How accurately can emotions be recognized from audio in a lightweight on-device setting?
• RQ2 Can recognized emotions be reliably mapped to safe, age-appropriate response modes?
• RQ3 Do generated content parameters satisfy safety constraints with real-time verification?
• RQ4 What is the end-to-end latency of the system across different hardware, and is it suitable for on-device deployment?
• RQ5 What is the impact of the safety verification loop on output quality and reliability?

Key findings

• Emotion recognition accuracy ranges across datasets, with 73.2% on IEMOCAP for 4-class, 78.5% on RAVDESS for 4-class, and 89.3% on synthetic data.
• Response mode accuracy achieved 89.4%, with high precision and recall around 86–87% across macro metrics.
• Safety verification achieved 100% pass rate across rules, with only 1.6% regenerations needed.
• End-to-end inference latency stayed under 100 ms on tested hardware, including edge devices.
• Ablation shows multi-agent structure improves consistency and safety compared to end-to-end setups.

Figure 2: Processing flowchart of the emotion-to-response content generation method. The system processes audio input through sequential stages (S10-S60) with a safety verification loop that triggers content regeneration upon failure.