[Paper Review] Generative AI has lowered the barriers to computational social sciences

Yongjun Zhang|arXiv (Cornell University)|Nov 17, 2023

Computational and Text Analysis Methods4 citations

TL;DR

This paper demonstrates how generative AI, particularly GPT-4 and related tools, lowers barriers in computational social science by enabling non-expert researchers to generate, annotate, and debug complex code for multimodal data analysis. It shows that with minimal programming knowledge, scholars can fine-tune Swin Transformer models for protest image classification using automated prompt engineering, achieving accurate predictions on large-scale global image datasets.

ABSTRACT

Generative artificial intelligence (AI) has revolutionized the field of computational social science (CSS), unleashing new possibilities for collecting and analyzing multimodal data, especially for scholars who may not have extensive programming expertise. This breakthrough carries profound implications for social scientists. First, generative AI can significantly enhance the productivity of social scientists by automating the generation, annotation, and debugging of code. Second, it empowers researchers to delve into sophisticated data analysis through the innovative use of prompt engineering. Last, the educational sphere of CSS stands to benefit immensely from these tools, given their exceptional ability to annotate and elucidate complex codes for learners, thereby simplifying the learning process and making the technology more accessible.

Motivation & Objective

• To reduce the programming barrier for social scientists engaging in computational research.
• To enable non-expert researchers to generate, annotate, and debug complex code for multimodal data analysis.
• To demonstrate the feasibility of using generative AI for fine-tuning vision models on protest image datasets.
• To streamline the analysis of large-scale, globally distributed image datasets using automated inference pipelines.
• To enhance accessibility and productivity in computational social science through AI-assisted coding and data processing.

Proposed method

• Using GPT-4’s code interpreter to generate Python scripts for fine-tuning a Swin Transformer V2 model on protest image classification.
• Designing a custom dataset class to load images and corresponding labels from TSV annotation files.
• Implementing a custom data collator to batch image tensors and labels efficiently for training.
• Employing Hugging Face’s Transformers library to load and fine-tune the pre-trained SwinV2 model with specified hyperparameters.
• Applying the trained model to classify 1.6 million protest-related images stored in country-specific subfolders using parallelized inference.
• Validating image files before prediction using PIL and handling errors gracefully through exception handling.

Experimental results

Research questions

• RQ1 Can generative AI tools like GPT-4 enable non-programming experts to generate functional code for training vision models in computational social science?
• RQ2 How effective is prompt engineering in generating accurate and efficient code for fine-tuning Swin Transformer models on protest image datasets?
• RQ3 To what extent can generative AI streamline the end-to-end pipeline of data loading, model training, evaluation, and inference for large-scale image analysis?
• RQ4 Can AI-generated code reliably handle real-world data challenges such as file validation and error recovery in multimodal datasets?
• RQ5 How does the integration of generative AI tools impact the accessibility and scalability of computational social science research?

Key findings

• Generative AI successfully generated a complete, functional Python script for fine-tuning a Swin Transformer V2 model on protest image classification using only natural language prompts.
• The model was trained and evaluated using standard metrics, with accuracy computed during evaluation, demonstrating reliable performance on the test set.
• The trained model was successfully applied to classify 1.6 million images across country-specific folders, with predictions saved to a CSV file.
• The inference pipeline used parallelized execution via ThreadPoolExecutor, significantly improving processing speed for large-scale image datasets.
• The system correctly handled file validation and error recovery, logging issues for invalid or unreadable image files without crashing.
• The entire workflow—from code generation to inference—was executed with minimal human intervention, highlighting the potential for democratizing access to advanced computational methods.