[Paper Review] Nostalgin: Extracting 3D City Models from Historical Image Data

Amol Kapoor, Hunter Larco+1 more|arXiv (Cornell University)|May 6, 2019

Advanced Vision and Imaging22 references4 citations

TL;DR

Nostalgin is a deep learning pipeline that reconstructs 3D city models from sparse, low-quality historical images lacking metadata. By combining semantic segmentation, novel image rectification, and efficient inpainting, it generates photorealistic 3D reconstructions from single images per facade, even with occlusions or unknown camera parameters, successfully reconstructing two 1940s Manhattan blocks.

ABSTRACT

What did it feel like to walk through a city from the past? In this work, we describe Nostalgin (Nostalgia Engine), a method that can faithfully reconstruct cities from historical images. Unlike existing work in city reconstruction, we focus on the task of reconstructing 3D cities from historical images. Working with historical image data is substantially more difficult, as there are significantly fewer buildings available and the details of the camera parameters which captured the images are unknown. Nostalgin can generate a city model even if there is only a single image per facade, regardless of viewpoint or occlusions. To achieve this, our novel architecture combines image segmentation, rectification, and inpainting. We motivate our design decisions with experimental analysis of individual components of our pipeline, and show that we can improve on baselines in both speed and visual realism. We demonstrate the efficacy of our pipeline by recreating two 1940s Manhattan city blocks. We aim to deploy Nostalgin as an open source platform where users can generate immersive historical experiences from their own photos.

Motivation & Objective

• To enable automatic 3D reconstruction of historical cities from sparse, low-quality, metadata-poor historical images.
• To overcome challenges in historical image data such as missing camera parameters, occlusions, and low image density.
• To develop a scalable, modular pipeline that works with minimal input—even a single image per facade—without requiring multiple views or manual annotation.
• To create a platform for users to generate immersive, walkable 3D historical environments from personal or archival photos.
• To improve upon existing methods in speed, visual realism, and robustness to image sparsity and degradation.

Proposed method

• A hybrid deep learning and algorithmic approach for tight semantic segmentation of building facades from historical images.
• A novel image rectification method that handles unknown camera parameters and non-uniform distortions in historical photos.
• An efficient deep image inpainting technique optimized for large-scale, high-resolution images, enabling reconstruction of occluded or missing facade details.
• A 3D modeling system that places rectified and inpainted facade images into a coherent 3D urban layout using geometric constraints and spatial reasoning.
• A distributed processing pipeline enabling end-to-end reconstruction of entire city blocks from archival image collections.
• Integration of contextual attention mechanisms and multi-scale feature extraction to improve inpainting quality and scalability.

Experimental results

Research questions

• RQ1 Can a 3D city model be reconstructed from a single historical image per facade, even with unknown camera parameters and occlusions?
• RQ2 How can image segmentation and rectification be made robust to the poor quality and lack of metadata in historical images?
• RQ3 What architectural choices enable efficient and high-fidelity image inpainting on large historical images without significant computational cost?
• RQ4 To what extent can a fully automated pipeline outperform existing baselines in visual realism and reconstruction speed?
• RQ5 Can the resulting 3D models support fully walkable, interactive environments that reflect historical urban layouts?

Key findings

• Nostalgin reduced wall time by 90.7% and heap allocation by 79.6% compared to the FreeForm inpainting baseline, demonstrating superior scalability.
• The method achieved a 27.8% reduction in heap allocation and 54.7% reduction in wall time when processing full-resolution images, indicating strong memory and runtime efficiency.
• The pipeline successfully reconstructed two 1940s Manhattan blocks using only archival images from the New York Municipal Archives, with no geotagging or EXIF data.
• The 3D reconstructions are fully walkable and support dynamic viewing angles not present in the original images, confirming the model's geometric coherence.
• Qualitative comparisons with modern Google Street View images show that Nostalgin accurately captures buildings that have since been demolished or significantly altered.
• The combination of contextual attention and multi-scale features improved inpainting quality with only slight increases in $l1$ and $l2$ loss, indicating high visual fidelity.