[Paper Review] How to Do Things with Deep Learning Code

Minh Hua, Rita Raley|arXiv (Cornell University)|Apr 19, 2023

Ethics and Social Impacts of AI4 citations

TL;DR

This paper applies Critical Code Studies to deep learning code by analyzing OpenAI’s GPT-2 through two classes of code: model-level and application-level. It demonstrates that code, especially in applications like AI Dungeon and This Word Does Not Exist, actively shapes model behavior, revealing how developers engineer outputs through filtering and structure—offering a path to demystify AI and enable critical, creative, and responsible engagement with large language models.

ABSTRACT

The premise of this article is that a basic understanding of the composition and functioning of large language models is critically urgent. To that end, we extract a representational map of OpenAI's GPT-2 with what we articulate as two classes of deep learning code, that which pertains to the model and that which underwrites applications built around the model. We then verify this map through case studies of two popular GPT-2 applications: the text adventure game, AI Dungeon, and the language art project, This Word Does Not Exist. Such an exercise allows us to test the potential of Critical Code Studies when the object of study is deep learning code and to demonstrate the validity of code as an analytical focus for researchers in the subfields of Critical Artificial Intelligence and Critical Machine Learning Studies. More broadly, however, our work draws attention to the means by which ordinary users might interact with, and even direct, the behavior of deep learning systems, and by extension works toward demystifying some of the auratic mystery of "AI." What is at stake is the possibility of achieving an informed sociotechnical consensus about the responsible applications of large language models, as well as a more expansive sense of their creative capabilities-indeed, understanding how and where engagement occurs allows all of us to become more active participants in the development of machine learning systems.

Motivation & Objective

• To investigate whether Critical Code Studies (CCS) can be effectively applied to deep learning systems, particularly those involving statistical and mathematical components beyond symbolic code.
• To demonstrate that code—especially application-level code—plays a central role in shaping the behavior and output of large language models like GPT-2.
• To challenge the perception of AI as an impenetrable 'black box' by showing how open-source code enables scrutiny, modification, and collective responsibility in model development.
• To expand the scope of Critical Artificial Intelligence Studies (CAIS) by centering code as a site of analysis, not just model architecture or data.
• To advocate for a more inclusive, participatory model of AI development where users and developers can actively shape and direct model behavior through code.

Proposed method

• The authors extract a representational map of GPT-2’s codebase, distinguishing between model-level code (defining the architecture and inference) and application-level code (ADLC) that structures user interaction.
• They conduct case studies on two GPT-2 applications: AI Dungeon, a text adventure game, and This Word Does Not Exist (TWDNE), a language art project, to analyze how ADLC shapes model output.
• The analysis focuses on how ADLC uses pattern matching, regular expressions, and filtering mechanisms to regulate and structure the model’s generative behavior.
• The study treats GPT-2 not as a monolithic system but as a distributed artifact composed of open-source code, training data, and developer-imposed constraints.
• It applies CCS principles to both symbolic code and mathematical operations, arguing that code in deep learning systems is analyzable not only for syntax but for its functional and ethical implications.
• The authors use screen captures and code inspection to trace how outputs are engineered—e.g., TWDNE’s 'invented' words are not generated but functionally returned via structured data handling.

Experimental results

Research questions

• RQ1 To what extent can Critical Code Studies be meaningfully applied to deep learning systems that include statistical and mathematical components beyond symbolic code?
• RQ2 How does application-level code (ADLC) in GPT-2-based systems actively shape, regulate, and structure the behavior of large language models?
• RQ3 In what ways do developers use code to mediate the expressive potential and risks of language model outputs, such as hate speech or incoherence?
• RQ4 How does the open-source nature of GPT-2’s code enable collective responsibility, scrutiny, and creative engagement in AI development?
• RQ5 Can code be a site of critical and creative intervention in AI systems, allowing users to move beyond passive consumption toward active participation?

Key findings

• Application-level code (ADLC) in systems like AI Dungeon and This Word Does Not Exist is not a passive interface but an active mechanism that filters, structures, and shapes the model’s output, often more than the model’s own generation process.
• The project This Word Does Not Exist demonstrates that GPT-2 is not generating novel words but returning structured data—exposing the model as a function rather than a sentient entity.
• The open-source availability of GPT-2’s code enabled developers to identify and implement safeguards against harmful outputs, such as filtering hate speech, through pattern matching and regular expressions.
• The development of GPT-2 as a system is not centralized to OpenAI; rather, it is a collaborative, distributed effort involving amateur, academic, and professional developers who collectively shape its behavior.
• The study reveals that the model’s 'wild expressivity' can be channeled through code, meaning that responsible AI development is not only possible but already underway through community-driven code interventions.
• By analyzing code as a site of critical inquiry, the paper shows that deep learning systems can be understood not as monolithic black boxes but as malleable, analyzable, and modifiable artifacts.