[Paper Review] AI Art Curation: Re-imagining the city of Helsinki in occasion of its Biennial

Ludovica Schaerf, Pepe Ballesteros+3 more|arXiv (Cornell University)|Jun 6, 2023

3D Surveying and Cultural HeritageEarth and Planetary Sciences3 citations

TL;DR

This paper presents an AI-driven curation system that reimagines Helsinki by placing artworks from the Helsinki Art Museum into fictional public spaces using visual-textual models and diffusion-based 360° panorama generation. By estimating depth and generating machine prompts, the project creates an interactive web-based installation that blurs artistic, spatial, and perceptual boundaries, offering a novel form of digital cultural engagement for the 2023 Helsinki Art Biennial.

ABSTRACT

Art curatorial practice is characterized by the presentation of an art collection in a knowledgeable way. Machine processes are characterized by their capacity to manage and analyze large amounts of data. This paper envisages AI curation and audience interaction to explore the implications of contemporary machine learning models for the curatorial world. This project was developed for the occasion of the 2023 Helsinki Art Biennial, entitled New Directions May Emerge. We use the Helsinki Art Museum (HAM) collection to re-imagine the city of Helsinki through the lens of machine perception. We use visual-textual models to place indoor artworks in public spaces, assigning fictional coordinates based on similarity scores. We transform the space that each artwork inhabits in the city by generating synthetic 360 art panoramas. We guide the generation estimating depth values from 360 panoramas at each artwork location, and machine-generated prompts of the artworks. The result of this project is an AI curation that places the artworks in their imagined physical space, blurring the lines of artwork, context, and machine perception. The work is virtually presented as a web-based installation on this link http://newlyformedcity.net/, where users can navigate an alternative version of the city while exploring and interacting with its cultural heritage at scale.

Motivation & Objective

• To investigate the potential of generative AI in redefining curatorial practices by integrating museum collections into imagined urban environments.
• To explore how machine perception can reinterpret the relationship between artworks, their context, and public space.
• To develop a scalable, interactive web-based installation that enables large-scale audience engagement with cultural heritage through AI-generated spatial reconstructions.
• To examine the implications of synthetic spatialization and prompt engineering in shaping artistic perception and urban imagination.

Proposed method

• Utilizes pre-trained visual-textual models to match indoor artworks from the Helsinki Art Museum to public spaces based on semantic and visual similarity scores.
• Assigns fictional geographic coordinates to artworks by computing similarity between artwork embeddings and urban scene embeddings.
• Generates synthetic 360° panoramic views of each artwork’s imagined urban location using diffusion models conditioned on depth estimation and text prompts.
• Estimates depth values from 360° panoramas to enhance spatial realism and guide consistent generation of synthetic urban environments.
• Employs machine-generated prompts derived from artwork embeddings to guide the diffusion model in creating contextually coherent virtual scenes.
• Deploys the final output as an interactive web-based installation accessible via a public URL, enabling virtual navigation of the re-imagined city.

Experimental results

Research questions

• RQ1 How can generative AI models be used to re-contextualize museum artworks within fictional urban environments?
• RQ2 What role does visual-textual similarity play in determining plausible spatial placements of artworks in public space?
• RQ3 To what extent can AI-generated depth and panoramic rendering create convincing and immersive virtual urban experiences?
• RQ4 How does audience interaction with AI-curated virtual spaces affect perception of cultural heritage and spatial meaning?
• RQ5 What are the ethical and curatorial implications of using synthetic data and machine perception to reconstruct urban cultural landscapes?

Key findings

• The AI curation system successfully generated coherent and visually plausible 360° panoramas that integrate artworks into fictional urban settings based on semantic and visual similarity.
• Depth estimation from generated panoramas improved spatial consistency and realism in the synthetic urban reconstructions.
• The web-based installation enabled scalable, interactive exploration of the re-imagined Helsinki, allowing users to navigate a dynamic, AI-curated version of the city.
• The project demonstrated that machine perception can generate contextually meaningful spatial arrangements of artworks without direct human intervention.
• The system achieved recognition as a best paper at the CVPR 2023 EC3V workshop, validating its innovative approach to AI-driven cultural curation.