[Paper Review] GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answering

Qianlong Li, Chen Huang+4 more|arXiv (Cornell University)|Dec 2, 2024

Data Quality and ManagementDecision Sciences3 citations

TL;DR

GraphOTTER proposes a graph-based reasoning framework that transforms complex tables into undirected graphs and uses explicit, step-by-step reasoning actions—such as VisitNode, GetSharedNeighbours, and AnswerQuestion—to guide LLMs in filtering irrelevant information and precisely locating answers. It achieves a 4.77% average improvement over the best baseline on two benchmark datasets using two LLM backbones.

ABSTRACT

Complex Table Question Answering involves providing accurate answers to specific questions based on intricate tables that exhibit complex layouts and flexible header locations. Despite considerable progress having been made in the LLM era, the reasoning processes of existing methods are often implicit, feeding the entire table into prompts, making it difficult to effectively filter out irrelevant information in the table. To this end, we propose GraphOTTER that explicitly establishes the reasoning process to pinpoint the correct answers. In particular, GraphOTTER leverages a graph-based representation, transforming the complex table into an undirected graph. It then conducts step-by-step reasoning on the graph, with each step guided by a set of pre-defined intermediate reasoning actions. As such, it constructs a clear reasoning path and effectively identifies the answer to a given question. Comprehensive experiments on two benchmark datasets and two LLM backbones demonstrate the effectiveness of GraphOTTER. Further analysis indicates that its success may be attributed to the ability to efficiently filter out irrelevant information, thereby focusing the reasoning process on the most pertinent data. Our code and experimental datasets are available at \url{https://github.com/JDing0521/GraphOTTER}.

Motivation & Objective

• To address the challenge of implicit reasoning in existing LLM-based methods for complex table question answering, which often process entire tables without filtering irrelevant content.
• To improve answer accuracy by explicitly modeling reasoning paths through a graph representation of complex tables with merged cells and flexible headers.
• To reduce reliance on full-table prompts by introducing structured, step-by-step reasoning actions that guide the LLM toward relevant cells.
• To enable more transparent, traceable, and efficient reasoning by maintaining a dynamic reasoning trace that tracks visited nodes and shared neighbors.
• To demonstrate the effectiveness of graph-based reasoning in complex table QA through comprehensive evaluation on benchmark datasets.

Proposed method

• Transforms complex tables into an undirected graph where header and data cells are represented as nodes, preserving semantic and spatial relationships.
• Introduces a set of predefined intermediate reasoning actions—VisitNode, GetSharedNeighbours, AnswerQuestion—to guide LLMs through structured navigation of the graph.
• Maintains a dynamic reasoning trace that records visited nodes and shared neighbors, enabling the system to avoid redundant or irrelevant paths.
• Uses a prompt engineering strategy that integrates the current table content, reasoning trace, and interaction history to guide each reasoning step.
• Employs a JSON schema for answer generation that ensures consistency in output format, including cells, operation, explanation, and final answer.
• Supports two LLM backbones and is evaluated on two benchmark datasets to ensure generalization and robustness.

Experimental results

Research questions

• RQ1 Can explicit, step-by-step reasoning on a graph-structured representation of complex tables improve answer accuracy in table question answering?
• RQ2 To what extent does graph-based reasoning reduce the influence of irrelevant table content compared to end-to-end prompting?
• RQ3 How effective is the reasoning trace mechanism in guiding LLMs toward relevant cells while avoiding redundant or incorrect paths?
• RQ4 Does the use of structured reasoning actions (e.g., GetSharedNeighbours) lead to measurable performance gains on complex table QA benchmarks?
• RQ5 Can GraphOTTER generalize across different LLM backbones and complex table layouts with merged cells and non-standard headers?

Key findings

• GraphOTTER achieves an average performance gain of +4.77% over the best baseline on two benchmark datasets, demonstrating significant improvement in complex table question answering.
• The explicit reasoning process enables effective filtering of irrelevant information, which is identified as a key factor in the model’s success.
• The reasoning trace mechanism successfully tracks visited nodes and shared neighbors, supporting coherent and focused navigation through the graph.
• The use of structured reasoning actions leads to more reliable and interpretable reasoning paths compared to implicit prompting strategies.
• The model maintains strong performance across two different LLM backbones, indicating its compatibility and robustness.
• Ablation studies confirm that the graph representation and explicit reasoning components are critical to the model’s performance, with ablation leading to significant accuracy drops.