[Paper Review] Large-Scale Attribute-Object Compositions

Filip Radenović, Animesh A. Sinha+3 more|arXiv (Cornell University)|May 24, 2021

Graph Theory and Algorithms61 references4 citations

TL;DR

This paper proposes CompNet, a weakly-supervised deep learning framework for large-scale attribute-object composition classification using Instagram hashtags as noisy supervision. By jointly learning object and attribute classifiers and composing them in classifier space, the model generalizes effectively to unseen attribute-object pairs, outperforming late fusion baselines, especially on rare or unobserved combinations in YFCC100m with 78M images, 7.7K objects, and 1.2K attributes.

ABSTRACT

We study the problem of learning how to predict attribute-object compositions from images, and its generalization to unseen compositions missing from the training data. To the best of our knowledge, this is a first large-scale study of this problem, involving hundreds of thousands of compositions. We train our framework with images from Instagram using hashtags as noisy weak supervision. We make careful design choices for data collection and modeling, in order to handle noisy annotations and unseen compositions. Finally, extensive evaluations show that learning to compose classifiers outperforms late fusion of individual attribute and object predictions, especially in the case of unseen attribute-object pairs.

Motivation & Objective

• To address the challenge of learning attribute-object compositions from images with limited, noisy annotations.
• To enable generalization to unseen attribute-object pairs not present in training data.
• To scale attribute-object classification to a large, diverse dataset with hundreds of thousands of compositions.
• To design a scalable, end-to-end framework that composes classifiers in feature space rather than learning each combination explicitly.
• To evaluate the method on real-world applications such as Facebook Marketplace, where coverage and generalization are critical.

Proposed method

• The framework uses a multi-head architecture with three classifier heads: object, attribute-agnostic attribute, and attribute-object classifiers.
• A composition module learns to combine object and attribute classifier weights directly in the classifier space, enabling inference on unseen combinations.
• A novel loss function is designed to better handle noisy, weakly-supervised hashtags and improve robustness during training.
• A candidate selection strategy reduces the number of attribute-object pairs considered during training and inference, improving efficiency and generalization.
• The model is trained on 78M images from YFCC100m using hashtags as weak labels, with a semi-automatic hashtag engineering process to select visually relevant, shared, and interpretable attributes.
• For deployment, the system avoids using the full MLP at inference and instead stores only the top-k compositional scores, reducing storage and computation.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively learn attribute-object compositions from weakly-supervised, noisy hashtags on a large-scale dataset?
• RQ2 How well can such a model generalize to attribute-object pairs not seen during training?
• RQ3 Does end-to-end composition of classifiers outperform late fusion of separate attribute and object predictions?
• RQ4 What design choices are necessary to make the composition module robust in the presence of noisy labels and missing annotations?
• RQ5 Can the framework be effectively deployed in real-world systems like Facebook Marketplace for scalable attribute filtering and ranking?

Key findings

• The proposed CompNet framework significantly outperforms late fusion baselines in zero-shot generalization, particularly on unseen attribute-object compositions.
• The model generalizes effectively to rare or unobserved combinations due to the classifier-space composition mechanism, which enables inductive bias across attributes and objects.
• Extensive evaluation on YFCC100m with 78M images, 7,694 objects, 1,237 attributes, and 280k compositions shows strong performance across seen and unseen cases.
• The framework enables deployment in Facebook Marketplace for attribute filtering and feed ranking, allowing new attributes to be composed for unseen categories without retraining or new annotations.
• By using a candidate selection strategy and storing only top-k compositional scores, the system reduces storage and inference cost while maintaining high coverage and performance.
• The method demonstrates scalability and robustness in real-world settings, with potential for broader adoption in commerce and retrieval applications.