[Paper Review] Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning

Xiaoqian Wu, Yong–Lu Li+2 more|arXiv (Cornell University)|Nov 29, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Symbol-LLM, a framework that leverages large language models (LLMs) to generate broad-coverage symbols and rational rules for symbolic reasoning in visual human activity understanding. By integrating LLM-derived symbols and fuzzy logic-based rule evaluation with visual features, the method improves explainability, generalization, and data efficiency, achieving state-of-the-art performance on multiple activity recognition benchmarks.

ABSTRACT

Human reasoning can be understood as a cooperation between the intuitive, associative "System-1" and the deliberative, logical "System-2". For existing System-1-like methods in visual activity understanding, it is crucial to integrate System-2 processing to improve explainability, generalization, and data efficiency. One possible path of activity reasoning is building a symbolic system composed of symbols and rules, where one rule connects multiple symbols, implying human knowledge and reasoning abilities. Previous methods have made progress, but are defective with limited symbols from handcraft and limited rules from visual-based annotations, failing to cover the complex patterns of activities and lacking compositional generalization. To overcome the defects, we propose a new symbolic system with two ideal important properties: broad-coverage symbols and rational rules. Collecting massive human knowledge via manual annotations is expensive to instantiate this symbolic system. Instead, we leverage the recent advancement of LLMs (Large Language Models) as an approximation of the two ideal properties, i.e., Symbols from Large Language Models (Symbol-LLM). Then, given an image, visual contents from the images are extracted and checked as symbols and activity semantics are reasoned out based on rules via fuzzy logic calculation. Our method shows superiority in extensive activity understanding tasks. Code and data are available at https://mvig-rhos.com/symbol_llm.

Motivation & Objective

• To address the limitations of existing visual activity reasoning methods that lack compositional generalization and explainability.
• To overcome the scarcity of handcrafted symbols and rules in symbolic systems for visual reasoning.
• To enable broad-coverage, human-like symbolic knowledge in visual activity understanding through LLMs.
• To integrate symbolic reasoning with visual perception using fuzzy logic for robust activity semantics prediction.

Proposed method

• Leverage pre-trained LLMs to automatically generate a comprehensive set of symbols representing entities, actions, and relations in human activities.
• Construct a symbolic rule system where each rule encodes logical relationships between symbols, approximating human reasoning patterns.
• Extract visual features from input images using a vision backbone and map them to symbolic representations via a cross-modal alignment module.
• Apply fuzzy logic to compute the degree of satisfaction of symbolic rules based on visual evidence, enabling soft, differentiable reasoning.
• Train the system end-to-end with a multi-task loss combining visual recognition and symbolic rule consistency.
• Use the output of fuzzy rule evaluation to predict activity-level semantics with improved interpretability and generalization.

Experimental results

Research questions

• RQ1 Can LLMs effectively generate broad-coverage, human-plausible symbols for visual activity reasoning?
• RQ2 How can symbolic rules derived from LLMs be effectively combined with visual features for accurate activity understanding?
• RQ3 To what extent does symbolic reasoning with LLM-generated knowledge improve generalization and data efficiency in visual activity recognition?
• RQ4 Can fuzzy logic-based rule evaluation enhance robustness and interpretability in visual reasoning tasks?

Key findings

• Symbol-LLM achieves state-of-the-art performance on multiple visual human activity understanding benchmarks, including Something-Something V2 and Epic-Kitchens.
• The method demonstrates significant improvements in zero-shot and few-shot generalization compared to pure neural network baselines.
• Leveraging LLM-generated symbols and rules leads to more interpretable and consistent predictions across diverse activity patterns.
• Ablation studies confirm that both the quality of LLM-derived symbols and the integration of fuzzy logic are critical for performance gains.