[Paper Review] Automatic Image Content Extraction: Operationalizing Machine Learning in Humanistic Photographic Studies of Large Visual Archives

Anssi Männistö, Mert Seker+2 more|arXiv (Cornell University)|Apr 5, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper introduces the Automatic Image Content Extraction (AICE) framework, a machine learning-based system that operationalizes large-scale visual content analysis in humanities research by reformulating traditional visual content analysis for compatibility with state-of-the-art ML tools. It enables automated, scalable, and structured analysis of vast photographic archives, significantly reducing manual effort and expanding research scope beyond traditional limits.

ABSTRACT

Applying machine learning tools to digitized image archives has a potential to revolutionize quantitative research of visual studies in humanities and social sciences. The ability to process a hundredfold greater number of photos than has been traditionally possible and to analyze them with an extensive set of variables will contribute to deeper insight into the material. Overall, these changes will help to shift the workflow from simple manual tasks to more demanding stages. In this paper, we introduce Automatic Image Content Extraction (AICE) framework for machine learning-based search and analysis of large image archives. We developed the framework in a multidisciplinary research project as framework for future photographic studies by reformulating and expanding the traditional visual content analysis methodologies to be compatible with the current and emerging state-of-the-art machine learning tools and to cover the novel machine learning opportunities for automatic content analysis. The proposed framework can be applied in several domains in humanities and social sciences, and it can be adjusted and scaled into various research settings. We also provide information on the current state of different machine learning techniques and show that there are already various publicly available methods that are suitable to a wide-scale of visual content analysis tasks.

Motivation & Objective

• To address the limitations of manual, text-based analysis in handling large-scale visual archives in humanities and social sciences.
• To bridge the gap between traditional visual content analysis methodologies and modern machine learning techniques.
• To enable researchers to analyze hundreds of thousands of images with detailed, content-aware annotations previously unattainable through manual coding.
• To promote interdisciplinary collaboration between humanistic researchers and machine learning experts to ensure methodological rigor and ethical responsibility.
• To advocate for responsible data annotation practices to prevent bias and ensure inclusivity in machine learning applications to visual archives.

Proposed method

• Reformulating traditional visual content analysis by introducing a structured set of variables suitable for machine learning-based extraction.
• Mapping each visual content variable to existing machine learning techniques, including object detection, scene recognition, and attribute classification.
• Designing the AICE framework as a modular system that supports incremental learning and adaptation to new research categories.
• Leveraging publicly available pre-trained models and transfer learning to reduce the need for large-scale custom training.
• Integrating multi-modal analysis potential by linking image content with textual metadata and contextual information.
• Emphasizing open publication of models and data to ensure reproducibility and critical evaluation of results.

Experimental results

Research questions

• RQ1 How can machine learning techniques be systematically integrated into humanistic photographic studies of large visual archives?
• RQ2 What are the key variables in visual content analysis that can be reliably extracted using current machine learning models?
• RQ3 In what ways can the AICE framework enhance the scope and depth of traditional photographic research beyond manual coding limits?
• RQ4 What are the main challenges in adopting automated image analysis in humanities research, and how can they be mitigated?
• RQ5 How can interdisciplinary collaboration between humanists and machine learning researchers ensure methodological soundness and ethical responsibility?

Key findings

• The AICE framework enables the analysis of image collections at scale—potentially a hundredfold larger than traditional manual methods—by automating content extraction.
• Current machine learning techniques already support a wide range of visual content analysis tasks, including object detection, scene classification, and attribute recognition.
• Automated content annotation reduces human error and labor intensity, shifting research workflows from manual coding to deeper analytical and interpretive phases.
• The framework supports incremental learning, allowing researchers to add new categories to trained models without retraining from scratch.
• Collaboration between humanistic and machine learning researchers is essential to ensure that training data reflect diverse cultural and social contexts and avoid bias.
• Open publication of models and data enhances reproducibility and enables critical evaluation of results, supporting cumulative and integrative research in the humanities.