[Paper Review] Learning Graph Embeddings for Open World Compositional Zero-Shot Learning

Massimiliano Mancini, Muhammad Ferjad Naeem+2 more|arXiv (Cornell University)|May 3, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Compositional Cosine Graph Embeddings (Co-CGE), a graph neural network approach that models dependencies between visual primitives (objects and states) and their compositions using a weighted graph with feasibility-aware edge weights. By integrating learned feasibility scores as margins in a cosine similarity loss and as adjacency weights, Co-CGE achieves state-of-the-art performance in both standard and open-world compositional zero-shot learning, significantly outperforming prior methods in the more realistic open-world setting with a vastly expanded composition space.

ABSTRACT

Compositional Zero-Shot learning (CZSL) aims to recognize unseen compositions of state and object visual primitives seen during training. A problem with standard CZSL is the assumption of knowing which unseen compositions will be available at test time. In this work, we overcome this assumption operating on the open world setting, where no limit is imposed on the compositional space at test time, and the search space contains a large number of unseen compositions. To address this problem, we propose a new approach, Compositional Cosine Graph Embeddings (Co-CGE), based on two principles. First, Co-CGE models the dependency between states, objects and their compositions through a graph convolutional neural network. The graph propagates information from seen to unseen concepts, improving their representations. Second, since not all unseen compositions are equally feasible, and less feasible ones may damage the learned representations, Co-CGE estimates a feasibility score for each unseen composition, using the scores as margins in a cosine similarity-based loss and as weights in the adjacency matrix of the graphs. Experiments show that our approach achieves state-of-the-art performances in standard CZSL while outperforming previous methods in the open world scenario.

Motivation & Objective

• To address the limitation of standard compositional zero-shot learning (CZSL), which assumes a closed world with known test-time compositions, by proposing a more realistic open-world setting where all possible compositions are in the output space.
• To improve generalization to unseen compositions by modeling dependencies between objects, states, and their compositions through a structured graph neural network.
• To reduce the negative impact of infeasible or distractor compositions (e.g., 'ripe dog') by estimating their feasibility and using it to guide representation learning.
• To unify visual and semantic embedding spaces through a cosine-based loss with feasibility margins, enabling better alignment and discrimination.
• To demonstrate that the proposed method generalizes effectively beyond closed-world assumptions, achieving strong performance even when the search space expands to nearly 28,000 compositions.

Proposed method

• Co-CGE employs a graph convolutional network (GCN) that connects nodes representing objects, states, and their compositions, enabling message passing across related concepts to improve representation learning.
• Feasibility scores for each unseen composition are computed based on the cosine similarity between visual and semantic embeddings of the constituent primitives, reflecting how semantically and visually plausible the composition is.
• These feasibility scores are used as margins in a cosine-based cross-entropy loss, encouraging the model to push unfeasible compositions further from correct ones in the embedding space.
• The same feasibility scores are used to weight the adjacency matrix of the graph, reducing the influence of infeasible or noisy compositions during message passing.
• The model is trained using a contrastive learning objective that aligns visual features with compositional embeddings while leveraging the graph structure for global consistency.
• The method supports both closed-world and open-world inference, with the open-world variant assuming no prior knowledge of which compositions will appear at test time.

Experimental results

Research questions

• RQ1 Can a graph-based model effectively learn compositional embeddings that generalize to all possible object-state compositions, even when the full composition space is unknown at test time?
• RQ2 How can feasibility scores derived from visual and semantic embeddings be used to improve the robustness of compositional zero-shot learning models against incoherent or implausible compositions?
• RQ3 Does integrating feasibility scores both as loss margins and as graph weights lead to better generalization than using them in isolation?
• RQ4 How does the proposed Co-CGE model compare to existing state-of-the-art methods in both closed-world and open-world compositional zero-shot learning benchmarks?
• RQ5 To what extent do the learned compositional embeddings support reliable image retrieval across seen and unseen compositions?

Key findings

• Co-CGE achieves state-of-the-art performance on standard compositional zero-shot learning benchmarks, outperforming prior methods such as CGE and CompCos, with a 0.78 AUC on C-GQA and 2.3 accuracy on MIT-States.
• In the open-world setting, Co-CGE significantly outperforms previous methods, achieving a 11.9% harmonic mean accuracy on unseen compositions across datasets, surpassing the next best method (11.8%).
• The model with cosine-based classification (Co-CGE) consistently outperforms its non-cosine counterpart (Co-CGE_ff), showing that metric learning via cosine similarity enhances generalization.
• Co-CGE shows a strong trade-off between seen and unseen composition accuracy, achieving 16.0% seen accuracy and 11.9% unseen accuracy on MIT-States, with a harmonic mean of 12.4%.
• The retrieval study reveals that models with higher CZSL accuracy do not always yield better compositional embeddings; Co-CGE and Co-CGE_CW achieve the best balance between seen and unseen retrieval performance.
• The feasibility score mechanism effectively reduces the negative impact of incoherent compositions, as evidenced by improved performance when feasibility scores are used in both loss margins and graph weights.