[Paper Review] If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History

Giselle González García, Christian Weilbach|arXiv (Cornell University)|Oct 16, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes augmenting large language models (LLMs) with vector embeddings from specialized historical sources—such as primary and secondary scholarly texts—to enable accurate, conversational research assistance in history. By prompting the LLM with curated, peer-reviewed corpora, the method achieves high-fidelity extraction of structured data (e.g., genealogical records) and effective question-answering, significantly reducing hallucination and improving source traceability compared to standard LLM use.

ABSTRACT

The recent advent of powerful Large-Language Models (LLM) provides a new conversational form of inquiry into historical memory (or, training data, in this case). We show that by augmenting such LLMs with vector embeddings from highly specialized academic sources, a conversational methodology can be made accessible to historians and other researchers in the Humanities. Concretely, we evaluate and demonstrate how LLMs have the ability of assisting researchers while they examine a customized corpora of different types of documents, including, but not exclusive to: (1). primary sources, (2). secondary sources written by experts, and (3). the combination of these two. Compared to established search interfaces for digital catalogues, such as metadata and full-text search, we evaluate the richer conversational style of LLMs on the performance of two main types of tasks: (1). question-answering, and (2). extraction and organization of data. We demonstrate that LLMs semantic retrieval and reasoning abilities on problem-specific tasks can be applied to large textual archives that have not been part of the its training data. Therefore, LLMs can be augmented with sources relevant to specific research projects, and can be queried privately by researchers.

Motivation & Objective

• To investigate whether LLMs can be effectively used as conversational research assistants in history by integrating domain-specific, peer-reviewed sources.
• To reduce hallucination and improve factual reliability in LLM outputs by grounding responses in vetted historical corpora.
• To evaluate the performance of LLMs in structured data extraction tasks—such as genealogical record tabulation—using primary and secondary sources.
• To demonstrate that private, source-augmented LLMs can support sensitive or confidential research without relying on proprietary APIs.
• To explore the feasibility of using open-source LLMs for academic research in the humanities, especially on local or non-GPU hardware.

Proposed method

• Augmenting pre-trained LLMs (e.g., ChatGPT) with vector embeddings from curated historical sources, including primary documents and scholarly secondary texts.
• Using retrieval-augmented generation (RAG) techniques to inject relevant source chunks into the LLM’s context window during inference.
• Designing structured prompts to extract tabular data from unstructured historical texts, such as genealogical records from *Historia de Familias Cubanas*.
• Inferring gender from Spanish name conventions using the LLM’s pre-trained knowledge, with validation against known data.
• Evaluating performance on question-answering and data extraction tasks using manually verified reference data.
• Comparing results across LLMs, including GPT-3.5 and Claude 2, with a focus on context window size and reasoning accuracy.

Figure 1 : Interaction between a Human and an LLM in form of a conversation. The human optionally supplies a single source text document for the LLM to read first. The LLM can also retrieve text chunks from a vector database filled from a corpus of source documents. Dashed lines denote optional depe

Experimental results

Research questions

• RQ1 Can LLMs provide reliable, source-grounded research assistance in history when augmented with domain-specific corpora?
• RQ2 How accurately can LLMs extract structured data (e.g., birth, death, marriage dates) from unstructured historical texts?
• RQ3 To what extent can LLMs infer gender from Spanish names in historical documents without explicit gender labels?
• RQ4 How does source-augmentation reduce hallucination and improve factual reliability in LLM-generated responses?
• RQ5 Can open-source LLMs match the performance of proprietary models like ChatGPT in academic research tasks when fine-tuned with scholarly sources?

Key findings

• ChatGPT successfully extracted structured genealogical data from 9 volumes of *Historia de Familias Cubanas* into a table with 95%+ accuracy, missing only a few entries with complex name constructions (e.g., 'y' conjunctions).
• The model correctly inferred gender for 17 out of 18 individuals based on name conventions, with only one incorrect inference out of 18.
• Only one birth date was incorrectly inferred among all extracted dates, indicating a low error rate in temporal data extraction.
• Despite being trained on only 7% non-English data, GPT-3.5 demonstrated strong performance in Spanish, accurately translating and interpreting Spanish historical texts.
• Source-augmented prompting significantly reduced hallucination by grounding responses in vetted, peer-reviewed corpora, improving traceability and reliability.
• Open-source LLMs show promise for private, on-device research use, especially with growing context window support in models like Claude 2.

(a) Word cloud of the 9 volumes of Historia de Familias Cubanas by Francisco Javier de Santa Cruz [ 9 ] showing the word frequency and the regularity of names and expressions. Made by the authors with VoyantTools.