[Paper Review] Combating Human Trafficking with Deep Multimodal Models

Edmund Tong, Amir Zadeh+2 more|arXiv (Cornell University)|May 8, 2017

Sex work and related issuesSocial Sciences17 references20 citations

TL;DR

This paper introduces the Trafficking-10k dataset, a large-scale, multimodal collection of over 10,000 human trafficking-related advertisements annotated with trafficking likelihood. It proposes the Human Trafficking Deep Network (HTDN), a deep multimodal model that fuses text and image features using fine-tuned VGG and contextualized language representations, achieving state-of-the-art performance in detecting trafficking content by outperforming unimodal and non-neural baselines.

ABSTRACT

Human trafficking is a global epidemic affecting millions of people across the planet. Sex trafficking, the dominant form of human trafficking, has seen a significant rise mostly due to the abundance of escort websites, where human traffickers can openly advertise among at-will escort advertisements. In this paper, we take a major step in the automatic detection of advertisements suspected to pertain to human trafficking. We present a novel dataset called Trafficking-10k, with more than 10,000 advertisements annotated for this task. The dataset contains two sources of information per advertisement: text and images. For the accurate detection of trafficking advertisements, we designed and trained a deep multimodal model called the Human Trafficking Deep Network (HTDN).

Motivation & Objective

• To address the growing challenge of online human trafficking, particularly through escort websites that host at-will advertisements.
• To develop a robust, generalizable machine learning system capable of detecting trafficking indicators despite adversarial obfuscation and non-grammatical language.
• To create the first large-scale, rigorously annotated dataset—Trafficking-10k—featuring both textual and visual modalities for trafficking advertisement detection.
• To design and train an end-to-end multimodal deep neural network (HTDN) that jointly models text and image features for improved detection performance.
• To overcome limitations of keyword-based and unimodal approaches by leveraging semantic understanding across language and vision modalities.

Proposed method

• The HTDN model employs a late-fusion architecture that processes textual and visual inputs through separate encoders: a BiLSTM with character-level embeddings for text and a fine-tuned T-VGG network for images.
• Textual features are extracted using a character-level bidirectional LSTM, which enhances robustness to spelling variations and obfuscation common in trafficking ads.
• Visual features are extracted from a fine-tuned VGG-16 network (T-VGG), improving performance over a pre-trained VGG by adapting to the specific visual patterns in escort ads.
• The model fuses late-learned representations from both modalities using a concatenation and feed-forward network, enabling joint inference on trafficking likelihood.
• Hyperparameters are optimized using a validation set, with Adam optimization and Xavier weight initialization for training stability.
• Baseline models include bag-of-words, keyword-based, word vector, and traditional classifiers (SVM, random forest), used for comparative evaluation.

Experimental results

Research questions

• RQ1 Can a deep multimodal model that jointly analyzes text and images significantly improve detection of human trafficking advertisements compared to unimodal or non-neural methods?
• RQ2 How effective is fine-tuning a pre-trained CNN (T-VGG) versus using a pre-trained VGG for visual feature extraction in trafficking detection?
• RQ3 To what extent does character-level language modeling improve robustness to obfuscation and non-standard language in trafficking ads?
• RQ4 How generalizable are the learned features across diverse trafficking ad patterns, especially given the high class imbalance and adversarial obfuscation?
• RQ5 Can multimodal fusion effectively capture contextual cues—such as text describing a hotel room and images showing a hotel setting—that signal trafficking?

Key findings

• The HTDN model achieves the highest F1-score and weighted accuracy among all evaluated models, demonstrating a significant performance gap over non-neural and unimodal baselines.
• Fine-tuning the VGG network (T-VGG) leads to measurable performance gains over the pre-trained VGG, indicating domain-specific visual feature learning is beneficial.
• The language modality alone (using character-level BiLSTM) outperforms the visual modality alone (T-VGG), suggesting text contains more discriminative signals for trafficking detection.
• The combination of both modalities in HTDN yields superior performance, confirming that multimodal fusion captures complementary cues not available in isolation.
• t-SNE visualization shows that no single feature representation (e.g., bag-of-words, keywords, word vectors, visual features) trivializes the classification task, indicating dataset complexity.
• The random baseline (predicting the majority class) and human performance serve as lower and upper bounds, respectively, with HTDN significantly outperforming the former and approaching the latter.