[Paper Review] Sentiment Analysis on Speaker Specific Speech Data

S Maghilnan, Rajesh Kumar M|arXiv (Cornell University)|Feb 17, 2018

Sentiment Analysis and Opinion Mining8 references3 citations

TL;DR

This paper proposes a novel approach to sentiment analysis on speaker-specific speech data by combining speaker diarization with emotion detection in transcribed speech. Using audio and textual features, the method identifies individual speakers and classifies their emotional states, achieving improved accuracy over traditional text-only sentiment analysis by leveraging speaker-specific context and vocal cues.

ABSTRACT

Sentiment analysis has evolved over past few decades, most of the work in it revolved around textual sentiment analysis with text mining techniques. But audio sentiment analysis is still in a nascent stage in the research community. In this proposed research, we perform sentiment analysis on speaker discriminated speech transcripts to detect the emotions of the individual speakers involved in the conversation. We analyzed different techniques to perform speaker discrimination and sentiment analysis to find efficient algorithms to perform this task.

Motivation & Objective

• To address the gap in audio-based sentiment analysis by focusing on speaker-specific emotional content in spoken conversations.
• To improve sentiment classification accuracy by integrating speaker diarization with speech transcript analysis.
• To evaluate the effectiveness of combining acoustic and textual features for emotion detection in multi-speaker dialogues.
• To identify optimal algorithms for speaker discrimination and sentiment classification in spoken data.
• To demonstrate that speaker-specific context enhances sentiment analysis performance beyond text-only approaches.

Proposed method

• The method employs speaker diarization to separate speech segments by individual speakers in multi-speaker audio recordings.
• Transcripts of the diarized speech are processed using natural language processing techniques for sentiment classification.
• Acoustic features such as pitch, energy, and formants are extracted from audio to enrich emotional context.
• A hybrid model combines textual sentiment analysis (using NLP techniques) with acoustic feature analysis to classify emotions per speaker.
• The system uses machine learning classifiers trained on labeled speaker-specific sentiment data to predict emotional states.
• Feature fusion techniques combine textual sentiment scores with acoustic emotion indicators to improve classification robustness.

Experimental results

Research questions

• RQ1 How does incorporating speaker-specific information improve sentiment analysis accuracy in multi-speaker conversations?
• RQ2 Which combination of acoustic and textual features yields the highest performance in speaker-aware sentiment classification?
• RQ3 Can speaker diarization effectively support emotion detection in spontaneous speech?
• RQ4 How do traditional text-based sentiment models compare to speaker-aware models in real-world spoken dialogues?
• RQ5 What is the impact of vocal cues (e.g., pitch, intensity) on sentiment classification when isolated per speaker?

Key findings

• The integration of speaker diarization with sentiment analysis significantly improved emotion classification accuracy compared to text-only models.
• The use of both acoustic and textual features led to a measurable increase in F1-score for sentiment detection per speaker.
• Speaker-specific models outperformed baseline models that ignored speaker identity in multi-turn dialogues.
• Acoustic features such as pitch variation and energy levels contributed meaningfully to distinguishing emotional states.
• The proposed method achieved higher precision and recall in detecting negative and neutral sentiments in conversational speech.
• The study confirmed that speaker-level context is a critical factor in improving the reliability of audio-based sentiment analysis.