[Paper Review] On the Security Vulnerabilities of Text-to-SQL Models

Xutan Peng, Yipeng Zhang+2 more|arXiv (Cornell University)|Nov 28, 2022

Web Application Security Vulnerabilities4 citations

TL;DR

This paper demonstrates that Text-to-SQL models in commercial and open-source NLP systems are vulnerable to malicious exploitation, enabling SQL injection and denial-of-service attacks via crafted natural language inputs. It shows that backdoor attacks on these models achieve 100% success in generating malicious SQL without degrading performance, highlighting a critical security risk in real-world AI deployments.

ABSTRACT

Although it has been demonstrated that Natural Language Processing (NLP) algorithms are vulnerable to deliberate attacks, the question of whether such weaknesses can lead to software security threats is under-explored. To bridge this gap, we conducted vulnerability tests on Text-to-SQL systems that are commonly used to create natural language interfaces to databases. We showed that the Text-to-SQL modules within six commercial applications can be manipulated to produce malicious code, potentially leading to data breaches and Denial of Service attacks. This is the first demonstration that NLP models can be exploited as attack vectors in the wild. In addition, experiments using four open-source language models verified that straightforward backdoor attacks on Text-to-SQL systems achieve a 100% success rate without affecting their performance. The aim of this work is to draw the community's attention to potential software security issues associated with NLP algorithms and encourage exploration of methods to mitigate against them.

Motivation & Objective

• To investigate whether Text-to-SQL models in real-world applications can be exploited as attack vectors for database security breaches.
• To evaluate the feasibility of black-box and backdoor attacks on commercial and open-source Text-to-SQL systems.
• To develop practical protocols for detecting vulnerabilities in deployed NLP systems.
• To raise awareness in the research community about the security risks of NLP models in production environments.
• To propose mitigation strategies and future research directions for securing NLP-based database interfaces.

Proposed method

• Conducted black-box vulnerability testing on six commercial Text-to-SQL applications using adversarial natural language prompts.
• Designed and injected malicious payloads into training data to create backdoors in open-source models, including BART and T5.
• Trained four open-source language models on poisoned datasets to evaluate backdoor trigger effectiveness during inference.
• Performed manual and single-host security tests on production systems to minimize risk while validating exploitability.
• Used coordinated vulnerability disclosure (CVD) to responsibly report findings to stakeholders, ensuring no data access or modification.
• Evaluated model performance on standard benchmarks and unseen schemas to assess generalization and stealth of backdoors.

Experimental results

Research questions

• RQ1 Can real-world commercial Text-to-SQL systems be manipulated via natural language inputs to generate malicious SQL?
• RQ2 To what extent can backdoor attacks be embedded in Text-to-SQL models without affecting their performance or detectability?
• RQ3 What are the practical implications of such vulnerabilities in production environments, particularly regarding data breaches and service disruption?
• RQ4 How effective are existing attack strategies in the context of Text-to-SQL systems compared to other NLP applications?
• RQ5 What defense mechanisms and automated detection tools are needed to secure NLP-driven database interfaces?

Key findings

• Six commercial Text-to-SQL applications, including BAIDU-UNIT and CHATGPT, were successfully exploited to generate malicious SQL, leading to potential data breaches and denial-of-service attacks.
• Black-box attacks on commercial systems achieved full exploitation, with BAIDU-UNIT’s vulnerabilities confirmed by the vendor as 'Highly Dangerous' and financially rewarded.
• Backdoor attacks on four open-source models (including state-of-the-art BART and T5) achieved 100% success in generating malware during inference, with no degradation in standard benchmark performance.
• The poisoned training data enabled the models to generalize across unseen database schemas while remaining stealthy, indicating a high-risk supply chain attack surface.
• All reported vulnerabilities were responsibly disclosed, and most vendors patched the issues following the researchers’ recommendations.
• The study revealed that even seemingly irrelevant or benign prompts could trigger malware generation, suggesting broader risks in LLM-based code generation.