[Paper Review] Traffic Performance GPT (TP-GPT): Real-Time Data Informed Intelligent ChatBot for Transportation Surveillance and Management

Bingzhang Wang, Zhiyu Cai+3 more|arXiv (Cornell University)|May 5, 2024

Big Data Technologies and ApplicationsDecision Sciences3 citations

TL;DR

This paper proposes TP-GPT, an intelligent, privacy-preserving chatbot that leverages pre-trained large language models (LLMs) to enable real-time, natural language-based querying and interpretation of complex traffic databases. By integrating domain-specific prompts, Chain-of-Thought reasoning, few-shot learning, multi-agent collaboration, and chat memory, TP-GPT achieves 80% flawless response accuracy on the TransQuery benchmark—significantly outperforming GPT-4 Turbo, PaLM 2, and SQLCoder in generating correct SQL queries and natural language insights for transportation analytics.

ABSTRACT

The digitization of traffic sensing infrastructure has significantly accumulated an extensive traffic data warehouse, which presents unprecedented challenges for transportation analytics. The complexities associated with querying large-scale multi-table databases require specialized programming expertise and labor-intensive development. Additionally, traditional analysis methods have focused mainly on numerical data, often neglecting the semantic aspects that could enhance interpretability and understanding. Furthermore, real-time traffic data access is typically limited due to privacy concerns. To bridge this gap, the integration of Large Language Models (LLMs) into the domain of traffic management presents a transformative approach to addressing the complexities and challenges inherent in modern transportation systems. This paper proposes an intelligent online chatbot, TP-GPT, for efficient customized transportation surveillance and management empowered by a large real-time traffic database. The innovative framework leverages contextual and generative intelligence of language models to generate accurate SQL queries and natural language interpretations by employing transportation-specialized prompts, Chain-of-Thought prompting, few-shot learning, multi-agent collaboration strategy, and chat memory. Experimental study demonstrates that our approach outperforms state-of-the-art baselines such as GPT-4 and PaLM 2 on a challenging traffic-analysis benchmark TransQuery. TP-GPT would aid researchers and practitioners in real-time transportation surveillance and management in a privacy-preserving, equitable, and customizable manner.

Motivation & Objective

• To address the challenge of limited accessibility and complexity in querying large-scale, multi-table real-time traffic databases.
• To bridge the gap between non-technical users and traffic data analytics by enabling natural language interaction with transportation databases.
• To enhance interpretability and accuracy in traffic data analysis by integrating semantic understanding with structured query generation.
• To develop a privacy-preserving, customizable, and equitable platform for real-time transportation surveillance and management.
• To demonstrate that domain-specific prompting and multi-agent reasoning significantly improve LLM performance on transportation-specific database queries.

Proposed method

• The system uses a customized, transportation-specialized prompt to embed domain context and role definitions for improved query accuracy.
• Chain-of-Thought prompting enables iterative reasoning and self-correction, reducing syntax and logic errors in SQL generation.
• Few-shot learning is applied using template questions and corresponding correct SQL queries to improve performance on similar query patterns.
• A multi-agent collaboration strategy simulates expert review processes, with agents generating thoughts, reviewing outputs, and refining queries across multiple rounds.
• Chat memory maintains context across interactions, enabling coherent, long-form conversations and improved response consistency.
• The framework integrates with a real-time traffic database and generates both executable SQL queries and human-readable natural language interpretations.

Experimental results

Research questions

• RQ1 Can a fine-tuned LLM-based chatbot accurately translate natural language traffic inquiries into correct SQL queries for large-scale, multi-table traffic databases?
• RQ2 How does domain-specific prompting and multi-agent reasoning improve the reliability and accuracy of LLM-generated queries in transportation analytics?
• RQ3 To what extent does few-shot learning enhance performance on similar query patterns compared to zero-shot inference?
• RQ4 How does the integration of Chain-of-Thought prompting reduce errors in complex query generation compared to sequential processing?
• RQ5 Can the system maintain high accuracy and privacy while enabling real-time, user-friendly access to sensitive traffic data?

Key findings

• TP-GPT achieved an 80% flawless response rate on the TransQuery benchmark, significantly outperforming GPT-4 Turbo (26% flawless), PaLM 2, and SQLCoder.
• The removal of the domain-specific prompt reduced the flawless response rate to 26%, demonstrating its critical role in contextual accuracy and role definition.
• The multi-agent strategy had the most substantial impact, reducing the flawless rate to 44% when removed, highlighting the importance of iterative review and reasoning.
• Few-shot learning had a minimal but measurable impact, primarily benefiting queries similar to the exemplars, with improved consistency in generating Microsoft SQL syntax.
• Chain-of-Thought prompting enabled iterative refinement, reducing errors from incorrect date formatting and logic flaws common in sequential models like GPT-4 Turbo.
• An interactive web-based TP-GPT chatbot was successfully deployed, demonstrating real-world usability for researchers and practitioners in transportation management.