[Paper Review] Learning the Compositional Visual Coherence for Complementary Recommendations

Zhi Li, Bo Wu+4 more|arXiv (Cornell University)|Jun 8, 2020

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper proposes a Content Attentive Neural Network (CANN) to model compositional visual coherence—both global and semantic-focal—for complementary item recommendations. By integrating multi-head attention for global coherence and hierarchical attention for color, texture, and hybrid-focal representations, CANN outperforms state-of-the-art methods, achieving 90.7% accuracy and 95.1% MRR on the FITB_Random benchmark.

ABSTRACT

Complementary recommendations, which aim at providing users product suggestions that are supplementary and compatible with their obtained items, have become a hot topic in both academia and industry in recent years. %However, it is challenging due to its complexity and subjectivity. Existing work mainly focused on modeling the co-purchased relations between two items, but the compositional associations of item collections are largely unexplored. Actually, when a user chooses the complementary items for the purchased products, it is intuitive that she will consider the visual semantic coherence (such as color collocations, texture compatibilities) in addition to global impressions. Towards this end, in this paper, we propose a novel Content Attentive Neural Network (CANN) to model the comprehensive compositional coherence on both global contents and semantic contents. Specifically, we first propose a extit{Global Coherence Learning} (GCL) module based on multi-heads attention to model the global compositional coherence. Then, we generate the semantic-focal representations from different semantic regions and design a extit{Focal Coherence Learning} (FCL) module to learn the focal compositional coherence from different semantic-focal representations. Finally, we optimize the CANN in a novel compositional optimization strategy. Extensive experiments on the large-scale real-world data clearly demonstrate the effectiveness of CANN compared with several state-of-the-art methods.

Motivation & Objective

• Address the gap in modeling compositional visual coherence—beyond co-purchased relations—in complementary recommendations.
• Capture both global visual coherence and fine-grained semantic-focal coherence (e.g., color, texture) in item collections.
• Improve recommendation quality by simulating user decision-making through a novel compositional optimization strategy.
• Enable more accurate and visually coherent recommendations by integrating visual semantics into the recommendation process.

Proposed method

• Propose a Global Coherence Learning (GCL) module using multi-head attention to model holistic compositional coherence across item features.
• Generate semantic-focal representations from distinct visual modalities: color-focal, texture-focal, and hybrid-focal features.
• Design a Focal Coherence Learning (FCL) module with hierarchical attention to learn local coherence from semantic-focal representations.
• Optimize the CANN model via a novel compositional optimization strategy that aligns with user perception of visual compatibility.
• Train and evaluate the model end-to-end on a large-scale real-world dataset using accuracy and MRR as evaluation metrics.
• Visualize attention scores to interpret how the model captures visual coherence between item pairs (e.g., color and pattern alignment).

Experimental results

Research questions

• RQ1 How can visual compositional coherence—beyond co-purchase patterns—be effectively modeled in complementary recommendations?
• RQ2 To what extent do color, texture, and hybrid visual semantics contribute to predicting complementary item compatibility?
• RQ3 Can a unified neural architecture that integrates global and focal visual coherence improve recommendation performance?
• RQ4 How does the proposed CANN model compare to state-of-the-art methods in capturing visual compatibility in real-world settings?

Key findings

• CANN with all semantic-focal components achieved 90.7% accuracy and 95.1% MRR on the FITB_Random benchmark, outperforming all baselines.
• The hybrid-focal coherence learning component yielded the best performance, followed by color-focal and then texture-focal, indicating color’s dominant role in visual compatibility.
• CANN with full semantic-focal modeling significantly outperformed single-modality models, especially on the FITB_Category set, demonstrating its ability to capture complex compositional relationships.
• Visualization of attention scores confirmed that CANN correctly identifies high-coherence pairs (e.g., black t-shirt with dark blue shorts, leopard-print shoes with bracelet), validating its interpretability.
• The GCL module effectively learned global compositional coherence, while the FCL module enhanced local semantic alignment, contributing synergistically to performance.
• The proposed compositional optimization strategy improved model alignment with user perception, as evidenced by higher accuracy and MRR on both testing sets.