[Paper Review] Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks

Panfeng Li, Mohamed Abouelenien+4 more|arXiv (Cornell University)|Nov 18, 2023

Deception detection and forensic psychology8 citations

TL;DR

The paper introduces a bimodal CNN framework that jointly learns from linguistic transcripts and physiological signals to detect deception, using a modality-wise training and majority voting to handle small datasets; results show multimodal fusion generally improves deception detection over unimodal approaches.

ABSTRACT

Deception detection is gaining increasing interest due to ethical and security concerns. This paper explores the application of convolutional neural networks for the purpose of multimodal deception detection. We use a dataset built by interviewing 104 subjects about two topics, with one truthful and one falsified response from each subject about each topic. In particular, we make three main contributions. First, we extract linguistic and physiological features from this data to train and construct the neural network models. Second, we propose a fused convolutional neural network model using both modalities in order to achieve an improved overall performance. Third, we compare our new approach with earlier methods designed for multimodal deception detection. We find that our system outperforms regular classification methods; our results indicate the feasibility of using neural networks for deception detection even in the presence of limited amounts of data.

Motivation & Objective

• Motivate deception detection due to ethical and security concerns and explore multimodal neural networks as an automated alternative to polygraphs and human judgment.
• Develop a multimodal architecture that jointly leverages linguistic and physiological features for deception classification.
• Address small-data challenges with modality-wise training and majority voting to stabilize learning.
• Compare the bimodal CNN approach against traditional machine learning classifiers and prior multimodal methods.
• Assess cross-topic generalization and stability of the proposed framework.

Proposed method

• Extract linguistic features from transcripts using word embeddings (word2vec) and apply a LingCNN based on a 2D CNN structure with filters similar to TextCNN.
• Extract physiological features from recorded signals, reduce dimensionality with PCA, and apply a PhysCNN using a 1D CNN with multiple filter sizes (3, 4, 5).
• Train LingCNN and PhysCNN separately in a modality-wise fashion before fusion.
• Fuse modalities in a BiModal CNN that concatenates features from both sub-networks for final classification.
• Handle small data and overfitting through a majority voting scheme across multiple training runs to produce robust final predictions.
• Compare multimodal CNN performance to regular classifiers (Decision Tree, SVM, Logistic Regression) and to previously reported multimodal deception methods.

Figure 1: BiModal CNN

Experimental results

Research questions

• RQ1 Can a bimodal CNN that jointly processes linguistic and physiological streams outperform unimodal models in deception detection?
• RQ2 How does cross-topic learning affect deception detection performance for linguistic vs. physiological modalities and their fusion?
• RQ3 Does a modality-wise training strategy with majority voting provide stability and improved accuracy on small datasets?
• RQ4 What is the comparative advantage of the proposed BiModal CNN over traditional machine learning methods and prior multimodal approaches?

Key findings

• The linguistics modality generally yields strong deception detection performance, with physiological features contributing complementary information.
• The BiModal CNN often improves overall accuracy compared to individual modalities, indicating benefits from multimodal fusion.
• Cross-topic learning shows topic dependence, with linguistic features being more affected by topic shifts than physiological ones.
• The majority voting procedure provides stable predictions after sufficient running times (around 200 runs in the study).
• Compared with regular models, the proposed BiModal CNN achieved better deception/overall accuracy across topics.

Figure 2: Deception recall, truthfulness recall, and overall accuracy percentages for individual and integrated modalities using features extracted from the “Abortion” topic