[Paper Review] Design Space and Implementation of RAG-Based Avatars for Virtual Archaeology

Wilhelm Kerle-Malcharek, Giulio Biondi+9 more|arXiv (Cornell University)|Mar 24, 2026

3D Surveying and Cultural Heritage0 citations

TL;DR

The paper defines a design space for RAG-driven avatars in VR for cultural heritage, builds a VR prototype around the Maxentius mausoleum, and evaluates RAG configurations and user workload in a VR context.

ABSTRACT

Immersive technologies, such as virtual and augmented reality, are transforming digital heritage by enabling users to explore and interact with culturally significant sites. It is now possible to view and augment digital twins, or digitally reconstructed versions of them, and to enable access to previously unreachable locations for a broader audience. Here, we investigate retrieval-augmented generation (RAG)-based avatars as an interface for accessing further information about digital cultural heritage objects while immersed in dedicated virtual environments. We present a requirement design space that spans the application realm, avatar personality, and I/O modalities. We instantiate it with a RAG system coupled to a conversational avatar in a virtual reality (VR) environment, using the Maxentius mausoleum from the 4th century AD as a case study, through which users gain access to curated on-demand information of the digitised heritage object. Our workflow utilises scholarly texts and enriches them with metadata. We evaluate various RAG configurations in terms of answer quality on a small expert-crafted question-answer set, as well as the perceived workload of users of a VR setup using such a RAG avatar. We demonstrate evidence that users perceive the overall workload for interacting with such an avatar as below average and that such avatars help to gain topical engagement. Overall, our work demonstrates how to utilise RAG-driven VR avatars for archaeological purposes and provides evidence that they can offer a pathway for immersive, AI-enhanced digital heritage applications.

Motivation & Objective

• Propose a requirement design space for AI-driven avatars in VR for cultural heritage applications.
• Instantiate a RAG-based avatar prototype in a VR environment using the Maxentius mausoleum as a case study.
• Evaluate how different RAG configurations affect answer quality and user workload in VR interactions.
• Demonstrate feasibility of immersive, AI-enhanced digital heritage applications and outline design considerations for practitioners.

Proposed method

• Develop a three-block conceptual design (application realm, avatar personality, avatar I/O) to guide system requirements.
• Design a logical VR–RAG architecture with local data storage, embedding-based retrieval, and TTS for dialogue.
• Implement a physical Unity-based VR prototype connected to a local FlowiseAI RAG stack using a Qdrant vector store and a CIDOC-CRM-informed metadata approach.
• Curate a literature-based knowledge base focusing on the Maxentius mausoleum and enrich metadata with author, title, publication type, and relevance.
• Evaluate multiple RAG configurations (seven setups) using expert-generated QA pairs and an LLM-as-judge metric, plus NASA-TLX workload assessment for VR users.

Experimental results

Research questions

• RQ1 How can a design space for AI-driven avatars in VR CH contexts be structured to cover application, personality, and I/O considerations?
• RQ2 What RAG configurations provide the best balance between answer quality and computational practicality for on-premise VR deployment in cultural heritage?
• RQ3 How does interacting with a RAG-based avatar in VR affect user workload and perceived engagement?
• RQ4 What practical workflow and metadata strategies best support domain-specific reasoning in RAG-RV deployments?
• RQ5 Can a RAG-driven VR avatar effectively deliver expert-level information about a specific archaeological object (Maxentius mausoleum) while maintaining controlled creativity and accuracy?

Key findings

• RAG-driven VR avatars can provide on-demand, expert-level information within a VR cultural heritage context.
• Different RAG configurations influence answer quality, with domain-aware metadata and knowledge graphs improving retrieval guidance.
• Users report below-average workload when interacting with a RAG avatar in VR, suggesting good usability for immersive heritage exploration.
• A pure on-premise, low-temperature RAG setup with curated literature supports responsible, non-creativity-heavy responses suitable for academic use.
• The Maxentius mausoleum case study demonstrates a feasible workflow from OCR'd literature to a CIDOC-CRM-informed knowledge graph powering a GraphRAG system.
• The study provides evidence that AI-driven avatars may facilitate immersive, AI-enhanced digital heritage applications.