[Paper Review] Augmenting the Author: Exploring the Potential of AI Collaboration in Academic Writing

Joseph Tu, Hilda Hadan+4 more|arXiv (Cornell University)|Apr 23, 2024

Artificial Intelligence in Healthcare and Education6 citations

TL;DR

The paper compares Gemini and ChatGPT as AI writing assistants in research-outline development, emphasizing prompt design, output analysis, and responsible human–AI collaboration.

ABSTRACT

This workshop paper presents a critical examination of the integration of Generative AI (Gen AI) into the academic writing process, focusing on the use of AI as a collaborative tool. It contrasts the performance and interaction of two AI models, Gemini and ChatGPT, through a collaborative inquiry approach where researchers engage in facilitated sessions to design prompts that elicit specific AI responses for crafting research outlines. This case study highlights the importance of prompt design, output analysis, and recognizing the AI's limitations to ensure responsible and effective AI integration in scholarly work. Preliminary findings suggest that prompt variation significantly affects output quality and reveals distinct capabilities and constraints of each model. The paper contributes to the field of Human-Computer Interaction by exploring effective prompt strategies and providing a comparative analysis of Gen AI models, ultimately aiming to enhance AI-assisted academic writing and prompt a deeper dialogue within the HCI community.

Motivation & Objective

• Investigate how generative AI can assist in the early drafting stages of academic writing.
• Examine prompt design strategies that elicit useful and reliable AI outputs.
• Provide a comparative analysis of two Gen AI models (ChatGPT and Gemini) in drafting research outlines.
• Highlight limitations, challenges, and best practices for responsible AI-assisted writing in HCI.
• Promote transparency and dialogue within the scholarly community about AI-enabled writing workflows.

Proposed method

• Facilitated collaborative inquiry sessions with 4–6 researchers to design prompts and discuss AI outputs.
• Use of Otter.Ai for audio transcription of weekly discussions to analyze prompts and responses.
• Structured prompts to elicit outlines from both Gemini and ChatGPT for the same research topic.
• Qualitative comparison of model strengths, limitations, and suggested improvements based on group deliberations.
• Critical reflection on reliability, transparency, and the role of human authors in AI-assisted writing.

Experimental results

Research questions

• RQ1 How do prompt design and prompt variation affect the quality and coherence of AI-generated research outlines?
• RQ2 What are the strengths, limitations, and distinct capabilities of ChatGPT versus Gemini in generating research outlines?
• RQ3 What are the reliability, transparency, and ethical considerations when using Gen AI for academic writing?
• RQ4 How can human–AI collaboration be optimized to enhance the outline development process in research?”
• RQ5 What guidelines emerge for responsible and effective AI-assisted writing in HCI contexts?

Key findings

• Prompt variation significantly influences output quality across models.
• Gemini provides dynamic suggestions and rationales and can guide reference searches.
• ChatGPT adheres to explicit instructions but may fabricate responses and struggles with sentence restructuring and some LaTeX formatting.
• Both models rely on explicit prompts for tasks like paragraph improvement based on references, highlighting the need for precise instruction.
• Transparency about AI involvement and handling of model limitations are crucial for credible AI-assisted writing.
• Human–AI collaboration requires ongoing reflection and dialogue to maximize benefits and mitigate evolving limitations.