[Paper Review] From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy

Maanak Gupta, Charankumar Akiri+3 more|arXiv (Cornell University)|Jul 3, 2023

Ethics and Social Impacts of AI55 citations

TL;DR

The paper surveys how Generative AI (GenAI) like ChatGPT affects cybersecurity and privacy, detailing attack vectors, jailbreaks, and defense strategies, plus social and legal implications. It also compares ChatGPT and Bard and outlines open challenges.

ABSTRACT

Undoubtedly, the evolution of Generative AI (GenAI) models has been the highlight of digital transformation in the year 2022. As the different GenAI models like ChatGPT and Google Bard continue to foster their complexity and capability, it's critical to understand its consequences from a cybersecurity perspective. Several instances recently have demonstrated the use of GenAI tools in both the defensive and offensive side of cybersecurity, and focusing on the social, ethical and privacy implications this technology possesses. This research paper highlights the limitations, challenges, potential risks, and opportunities of GenAI in the domain of cybersecurity and privacy. The work presents the vulnerabilities of ChatGPT, which can be exploited by malicious users to exfiltrate malicious information bypassing the ethical constraints on the model. This paper demonstrates successful example attacks like Jailbreaks, reverse psychology, and prompt injection attacks on the ChatGPT. The paper also investigates how cyber offenders can use the GenAI tools in developing cyber attacks, and explore the scenarios where ChatGPT can be used by adversaries to create social engineering attacks, phishing attacks, automated hacking, attack payload generation, malware creation, and polymorphic malware. This paper then examines defense techniques and uses GenAI tools to improve security measures, including cyber defense automation, reporting, threat intelligence, secure code generation and detection, attack identification, developing ethical guidelines, incidence response plans, and malware detection. We will also discuss the social, legal, and ethical implications of ChatGPT. In conclusion, the paper highlights open challenges and future directions to make this GenAI secure, safe, trustworthy, and ethical as the community understands its cybersecurity impacts.

Motivation & Objective

• Provide an overview of GenAI evolution and its cybersecurity landscape.
• Identify vulnerabilities and attack techniques targeting ChatGPT (e.g., jailbreaking, prompt injection).
• Explore how GenAI can be used for cyber offense (social engineering, phishing, automated hacking, malware/code generation).
• Discuss defense strategies leveraging GenAI (automation, threat intel, secure coding, incident response).
• Examine social, legal, and ethical implications and compare GenAI systems (ChatGPT vs Bard).

Proposed method

• Review and synthesize existing literature and examples of GenAI usage in cybersecurity.
• Analyze ChatGPT-specific vulnerabilities and jailbreak techniques (DAN, SWITCH, CHARACTER Play).
• Demonstrate attack scenarios including social engineering, phishing, automated hacking, payload and malware/code generation.
• Discuss defense applications of GenAI in threat intelligence, incident response, and secure coding.
• Compare security features of ChatGPT and Google Bard and outline open challenges and future directions.

Experimental results

Research questions

• RQ1 What are the key ways GenAI impacts cybersecurity and privacy from both offensive and defensive perspectives?
• RQ2 What vulnerabilities and jailbreak techniques affect ChatGPT, and how can attackers exploit them?
• RQ3 How can GenAI be used to enhance cyber defense (automation, threat intelligence, secure coding) and incident response?
• RQ4 What are the social, legal, and ethical implications of GenAI in cybersecurity, including privacy concerns?
• RQ5 How do ChatGPT and Google Bard compare in terms of security features and risks?

Key findings

• GenAI enables both offensive and defensive cybersecurity use cases, creating a double-edged impact.
• ChatGPT vulnerabilities include jailbreaking (DAN, SWITCH, CHARACTER Play), reverse psychology, prompt injection, and “escaping” behaviors, which can bypass safeguards.
• Attack scenarios enabled by GenAI encompass social engineering, spear phishing, automated hacking, payload/malware/code generation, and WAF circumvention.
• GenAI can enhance defense through threat intelligence, incident response automation, secure code generation, and ethical guideline development.
• The paper discusses social, legal, and privacy implications and provides a comparative view of ChatGPT and Bard for security features, outlining open challenges and future directions.