[Paper Review] The CAT SET on the MAT: Cross Attention for Set Matching in Bipartite Hypergraphs

Gajanand Sharma, Swyam Prakash Singh+2 more|arXiv (Cornell University)|Oct 30, 2021

Bioinformatics and Genomic Networks4 citations

TL;DR

This paper introduces CATSETMAT, a novel cross-attention neural network for bipartite hypergraph learning, framing bipartite hyperedge prediction as a set-matching problem. It outperforms state-of-the-art methods by modeling cross-attention between left and right hyperedges, resolving a critical positive-negative labeling dilemma in existing hyperedge embedding approaches.

ABSTRACT

Usual relations between entities could be captured using graphs; but those of a higher-order -- more so between two different types of entities (which we term "left" and "right") -- calls for a "bipartite hypergraph". For example, given a left set of symptoms and right set of diseases, the relation between a set subset of symptoms (that a patient experiences at a given point of time) and a subset of diseases (that he/she might be diagnosed with) could be well-represented using a bipartite hyperedge. The state-of-the-art in embedding nodes of a hypergraph is based on learning the self-attention structure between node-pairs from a hyperedge. In the present work, given a bipartite hypergraph, we aim at capturing relations between node pairs from the cross-product between the left and right hyperedges, and term it a "cross-attention" (CAT) based model. More precisely, we pose "bipartite hyperedge link prediction" as a set-matching (SETMAT) problem and propose a novel neural network architecture called CATSETMAT for the same. We perform extensive experiments on multiple bipartite hypergraph datasets to show the superior performance of CATSETMAT, which we compare with multiple techniques from the state-of-the-art. Our results also elucidate information flow in self- and cross-attention scenarios.

Motivation & Objective

• Address the lack of machine learning research on bipartite hypergraphs, a higher-order relational structure between two disjoint node sets.
• Identify a critical flaw in existing hyperedge embedding models like Hyper-SAGNN, which suffer from a positive-negative labeling dilemma when applied to bipartite hypergraphs.
• Reframe bipartite hyperedge prediction as a set-matching (SETMAT) problem, where the goal is to match subsets of left and right nodes that co-occur in a hyperedge.
• Propose a new neural architecture, CATSETMAT, that leverages cross-attention to model interactions between left and right hyperedge sets, improving relational reasoning.
• Demonstrate the superiority of cross-attention over self-attention mechanisms in capturing bipartite relational structures, with ablation studies on information flow.

Proposed method

• Formulate bipartite hyperedge prediction as a set-matching task: given two sets of hyperedges (left and right), predict which pairs form a valid bipartite hyperedge.
• Introduce a cross-attention mechanism that computes attention scores between all pairs of left and right hyperedges, enabling interaction across the two sets.
• Design a neural network architecture, CATSETMAT, that uses query, key, and value projections from left and right hyperedges to compute cross-attention features.
• Apply multi-head cross-attention with residual connections and feed-forward layers to aggregate and refine the attention-based representations.
• Use a contrastive learning objective to distinguish positive hyperedge pairs (from ground-truth bipartite hyperedges) from negative pairs (sampled via sized-random sampling).
• Apply a final classifier head to predict the likelihood of a set pair being a valid bipartite hyperedge, enabling end-to-end training.

Experimental results

Research questions

• RQ1 Can a cross-attention mechanism effectively model higher-order relations in bipartite hypergraphs, where relations span two disjoint node sets?
• RQ2 How does the performance of cross-attention compare to self-attention mechanisms (e.g., Hyper-SAGNN) in bipartite hyperedge prediction tasks?
• RQ3 What is the impact of the positive-negative labeling dilemma on hyperedge prediction in bipartite hypergraphs, and can it be mitigated?
• RQ4 Is there a theoretical equivalence between bipartite hyperedge prediction and set-matching, and can this be leveraged to improve model design?
• RQ5 How does information flow differ between self-attention and cross-attention in the context of bipartite hypergraph learning?

Key findings

• CATSETMAT achieves state-of-the-art performance across multiple bipartite hypergraph datasets, outperforming existing models like Hyper-SAGNN and FSPool.
• The positive-negative labeling dilemma in Hyper-SAGNN leads to high variance (up to 13%) and poor generalization, particularly on challenging datasets like mag-acm-ak.
• When trained on a sized-random negative sampling scheme (removing the per-fixed hypergraph constraint), Hyper-SAGNN achieves a test AUC of 98% on mag-acm-ak, revealing the instability of its original setup.
• The ablation study confirms that cross-attention is essential for capturing cross-set interactions, as self-attention mechanisms fail to model the true relational structure of bipartite hyperedges.
• Theoretical equivalence between bipartite hyperedge prediction and set-matching is established, validating the problem reformulation as a sound foundation for the proposed method.
• The model demonstrates robustness and consistent performance across diverse datasets, including symptoms-diseases, movie-cast, and academic co-authorship networks.