[Paper Review] Neural-Logic Human-Object Interaction Detection

Liulei Li, Jianan Wei+2 more|arXiv (Cornell University)|Nov 16, 2023

Explainable Artificial Intelligence (XAI)6 citations

TL;DR

LogicHOI proposes a novel Transformer-based HOI detector that replaces fixed human-object pair inputs with triplet-reasoning attention, enabling dynamic inference over unobserved {human, action, object} triplets. By grounding affordances and proxemics in first-order logic and projecting them into continuous space, the model improves zero-shot generalization and achieves state-of-the-art performance on V-COCO and HICO-DET.

ABSTRACT

The interaction decoder utilized in prevalent Transformer-based HOI detectors typically accepts pre-composed human-object pairs as inputs. Though achieving remarkable performance, such paradigm lacks feasibility and cannot explore novel combinations over entities during decoding. We present L OGIC HOI, a new HOI detector that leverages neural-logic reasoning and Transformer to infer feasible interactions between entities. Specifically, we modify the self-attention mechanism in vanilla Transformer, enabling it to reason over the triplet and constitute novel interactions. Meanwhile, such reasoning process is guided by two crucial properties for understanding HOI: affordances (the potential actions an object can facilitate) and proxemics (the spatial relations between humans and objects). We formulate these two properties in first-order logic and ground them into continuous space to constrain the learning process of our approach, leading to improved performance and zero-shot generalization capabilities. We evaluate L OGIC HOI on V-COCO and HICO-DET under both normal and zero-shot setups, achieving significant improvements over existing methods.

Motivation & Objective

• Address the limitation of existing HOI detectors that rely on pre-composed human-object pairs, restricting their ability to generalize to novel combinations.
• Overcome the lack of reasoning capacity in current methods, which treat interactions as static predictions on fixed pairs rather than dynamic inferences.
• Integrate symbolic knowledge—specifically affordances and proxemics—into deep learning to guide more logical and generalizable interaction prediction.
• Enable zero-shot generalization by constraining predictions to physically and semantically plausible interactions using first-order logic.
• Develop a more efficient and effective HOI detection framework that maintains high performance while reducing computational overhead compared to prior methods.

Proposed method

• Modify the self-attention mechanism in the Transformer decoder to operate on triplets of queries: human, action, and object, enabling joint reasoning over all three elements.
• Introduce triplet-reasoning attention that updates representations by aggregating information across all three components simultaneously, rather than attending to pre-defined pairs.
• Formulate affordances (which actions an object can support) and proxemics (spatial plausibility of interactions) as first-order logic constraints.
• Embed these logical constraints into continuous space using differentiable functions to guide the learning process of the Transformer model.
• Optimize the model with two auxiliary losses: $\mathcal{L}_{v,p}$ for visual-physical plausibility (proxemics) and $\mathcal{L}_{o,p}$ for object-action compatibility (affordances).
• Apply query filtering and minimize the number of decoder layers and queries to maintain efficiency, achieving faster inference than prior methods despite increased reasoning capacity.

Figure 1 : Left : self-attention aggregates information across pre-composed interaction ( ) queries . Middle : in contrast, our proposed triplet-reasoning attention traverses over human ( ), action ( ), and object ( ) queries to propose plausible interactions. Right : logic-induced affordances and p

Experimental results

Research questions

• RQ1 Can a Transformer-based HOI detector generalize to unseen {human, action, object} triplets without requiring them in the training data?
• RQ2 How can symbolic knowledge such as affordances and proxemics be effectively integrated into a neural network to improve reasoning and zero-shot generalization?
• RQ3 Does modifying the attention mechanism to reason over triplets rather than pairs lead to better performance and more interpretable predictions?
• RQ4 Can logic-guided constraints reduce the model's reliance on spurious correlations and improve robustness to rare or unseen interactions?
• RQ5 What is the trade-off between model complexity, inference speed, and performance when incorporating symbolic reasoning into a neural architecture?

Key findings

• LogicHOI achieves a new state-of-the-art performance of 25.97 mAP on the full set of HICO-DET, a 1.96-point improvement over the previous SOTA.
• On the rare category split of HICO-DET, LogicHOI achieves 15.67 mAP, a 2.41-point gain over the prior state-of-the-art, demonstrating strong zero-shot generalization.
• The ablation study confirms that both $\mathcal{L}_{v,p}$ (proxemics) and $\mathcal{L}_{o,p}$ (affordances) contribute significantly to performance, especially under zero-shot settings.
• The model maintains high efficiency: with only 3 decoder layers and 32 queries, it achieves 16.84 FPS on a ResNet-50 backbone, outperforming most one-stage detectors in speed and FLOPs.
• Increasing the number of queries beyond 32 degrades performance, indicating an optimal balance between representational capacity and optimization stability.
• The integration of logic-guided learning leads to faster convergence and more plausible predictions, as evidenced by the model's ability to reject physically implausible interactions during inference.

Figure 2 : Overview of LogicHOI . We first retrieve human, action, and object queries by $\mathcal{D}^{h}$ , $\mathcal{D}^{a}$ , and $\mathcal{D}^{o}$ , respectively. Then $\mathcal{D}^{p}$ take them as input, reasoning over entities and combining potential interaction triplets. Finally, such proces