[Paper Review] Semi-supervised classification by reaching consensus among modalities

Zining Zhu, Jekaterina Novikova+1 more|arXiv (Cornell University)|May 23, 2018

Adversarial Robustness in Machine Learning14 references3 citations

TL;DR

This paper introduces Transductive Consensus Networks (TCNs), a semi-supervised learning framework that leverages multi-modal data by enforcing consensus among modality-specific representations through adversarial training. TCNs outperform or match state-of-the-art benchmarks with as few as 20 labeled samples on Bank Marketing and DementiaBank datasets, demonstrating that stable, near-optimal similarity between modality representations—measured via negative relative Jensen-Shannon divergence—yields superior classification performance.

ABSTRACT

Deep learning has demonstrated abilities to learn complex structures, but they can be restricted by available data. Recently, Consensus Networks (CNs) were proposed to alleviate data sparsity by utilizing features from multiple modalities, but they too have been limited by the size of labeled data. In this paper, we extend CN to Transductive Consensus Networks (TCNs), suitable for semi-supervised learning. In TCNs, different modalities of input are compressed into latent representations, which we encourage to become indistinguishable during iterative adversarial training. To understand TCNs two mechanisms, consensus and classification, we put forward its three variants in ablation studies on these mechanisms. To further investigate TCN models, we treat the latent representations as probability distributions and measure their similarities as the negative relative Jensen-Shannon divergences. We show that a consensus state beneficial for classification desires a stable but imperfect similarity between the representations. Overall, TCNs outperform or align with the best benchmark algorithms given 20 to 200 labeled samples on the Bank Marketing and the DementiaBank datasets.

Motivation & Objective

• Address data scarcity in multi-modal classification by extending Consensus Networks to semi-supervised learning.
• Investigate the role of consensus and classification mechanisms in improving model generalization with limited labeled data.
• Understand the relationship between similarity of modality representations and downstream classification performance.
• Evaluate TCNs on real-world datasets with pathological and non-pathological speech, and tabular data, under low-shot learning settings.

Proposed method

• Propose Transductive Consensus Networks (TCNs) that combine a consensus mechanism (adversarial training to align modality representations) and a classification mechanism (supervised training on labeled data).
• Use adversarial training to compress multi-modal inputs into latent representations that are indistinguishable across modalities, using a discriminator to enforce consensus.
• Treat latent representations as probability distributions and measure their similarity using negative relative Jensen-Shannon divergence.
• Train the model iteratively, alternating between consensus and classification objectives to balance representation alignment and predictive accuracy.
• Introduce three ablation variants: TCN-embed (no classifier), TCN-svm (SVM-based classifier), and TCN-AE (autoencoder-based reconstruction) to isolate mechanism contributions.
• Apply early stopping based on loss convergence and reinitialize if training diverges, ensuring stable optimization.

Experimental results

Research questions

• RQ1 How does the consensus mechanism in TCNs improve generalization in low-labeled-data regimes?
• RQ2 What is the optimal level of similarity between modality representations for effective classification?
• RQ3 To what extent do both consensus and classification mechanisms contribute to TCN performance compared to their individual components?
• RQ4 How does TCN compare to established semi-supervised benchmarks like TSVM, Ladder, CatGAN, and tri-training on real-world datasets?
• RQ5 Can negative relative Jensen-Shannon divergence serve as a reliable proxy for measuring consensus quality in multi-modal representation learning?

Key findings

• TCNs outperform or match the best benchmark algorithms on both the Bank Marketing and DementiaBank datasets when only 20 to 200 labeled samples are available.
• The ablation study shows that both consensus and classification mechanisms are essential: removing either significantly degrades performance, and iterative training is crucial.
• TCN-embed and TCN (full model) achieve comparable performance, both significantly outperforming TCN-AE, which fails to reach consensus due to autoencoder interference.
• TCN-svm performs poorly, indicating that SVM-based classification is ineffective in this multi-modal, adversarial setting.
• The similarity between modality representations, measured via negative relative JS divergence, stabilizes over training and peaks at a suboptimal but meaningful level, indicating that perfect alignment is not necessary for good performance.
• T-SNE visualizations show that at step 30, representations form symmetric, mixed clusters—corresponding to the highest similarity and best classification performance—confirming the existence of a stable, beneficial consensus state.