[Paper Review] PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance

Yang Deng, Wenqiang Lei+3 more|arXiv (Cornell University)|Oct 17, 2022

Topic Modeling4 citations

TL;DR

This paper introduces PACIFIC, a new dataset and task—Proactive Conversational Question Answering (PCQA)—for financial QA over hybrid tabular and textual data. It proposes UniPCQA, a unified sequence-to-sequence framework that reformulates numerical reasoning as code generation and uses multi-task learning with consensus voting to improve performance on clarification question generation, ambiguity detection, and answer generation, demonstrating the challenge of proactive, reasoning-intensive CQA in finance.

ABSTRACT

To facilitate conversational question answering (CQA) over hybrid contexts in finance, we present a new dataset, named PACIFIC. Compared with existing CQA datasets, PACIFIC exhibits three key features: (i) proactivity, (ii) numerical reasoning, and (iii) hybrid context of tables and text. A new task is defined accordingly to study Proactive Conversational Question Answering (PCQA), which combines clarification question generation and CQA. In addition, we propose a novel method, namely UniPCQA, to adapt a hybrid format of input and output content in PCQA into the Seq2Seq problem, including the reformulation of the numerical reasoning process as code generation. UniPCQA performs multi-task learning over all sub-tasks in PCQA and incorporates a simple ensemble strategy to alleviate the error propagation issue in the multi-task learning by cross-validating top-$k$ sampled Seq2Seq outputs. We benchmark the PACIFIC dataset with extensive baselines and provide comprehensive evaluations on each sub-task of PCQA.

Motivation & Objective

• To address the lack of proactive, multi-turn financial QA systems that handle ambiguous queries and require numerical reasoning.
• To develop a new benchmark dataset, PACIFIC, supporting hybrid contexts (tables and text), proactive clarification, and complex numerical reasoning.
• To define and formalize Proactive Conversational Question Answering (PCQA) as a unified task combining clarification question generation and conversational QA.
• To propose UniPCQA, a unified Seq2Seq framework that models ambiguity detection, clarification generation, and answer generation jointly.
• To evaluate the challenges of hybrid input/output formats and numerical reasoning in financial CQA through comprehensive ablation and error analysis.

Proposed method

• Reformulate numerical reasoning in financial QA as code generation, encoding arithmetic operations and entity references into executable Python code.
• Design a unified input representation that integrates tabular data, textual context, and user queries into a single sequence for Seq2Seq modeling.
• Define a unified output format that supports three response types: ambiguity prediction, clarification questions, and direct answers.
• Implement multi-task learning across all PCQA sub-tasks (ambiguity detection, CQG, CQA) within a single Seq2Seq model to share representations.
• Apply a consensus voting ensemble strategy that cross-validates the top-k generated outputs to reduce error propagation in multi-task learning.
• Use a pre-trained language model fine-tuned on the hybrid PACIFIC dataset to handle diverse input modalities and reasoning types.

Experimental results

Research questions

• RQ1 How effective is a unified Seq2Seq framework in handling ambiguity detection, clarification question generation, and answer generation in financial conversational QA?
• RQ2 To what extent does reformulating numerical reasoning as code generation improve performance on financial QA tasks with complex arithmetic?
• RQ3 Can multi-task learning with shared representations improve performance across multiple sub-tasks in proactive CQA?
• RQ4 How does the consensus voting ensemble strategy mitigate error propagation in multi-task Seq2Seq models for PCQA?
• RQ5 What are the primary failure modes and error patterns in current models when handling hybrid tabular and textual financial contexts?

Key findings

• The PACIFIC dataset presents a significant challenge, with state-of-the-art models still falling short of human performance, indicating room for future research.
• Error analysis shows that 25% of errors are related to clarification question generation and 13% to numerical calculation, highlighting key failure modes.
• The percentage of span-extraction-related errors decreased from 84% to 46% compared to prior models, indicating improved reasoning generalization.
• The proposed UniPCQA model achieves strong performance across all sub-tasks, demonstrating the effectiveness of unified modeling and code-based reasoning.
• The consensus voting strategy improves robustness by cross-validating top-k generated outputs, reducing error propagation in multi-task learning.
• The dataset and code are publicly released at https://github.com/dengyang17/PACIFIC/, enabling reproducibility and future benchmarking.