[Paper Review] Deep Demosaicing for Polarimetric Filter Array Cameras

Mara Pistellato, Filippo Bergamasco+2 more|PubMed|Nov 24, 2022

Optical Polarization and Ellipsometry4 citations

TL;DR

This paper proposes PFADN, a novel CNN-based demosaicing network for Polarimetric Filter Array (PFA) cameras that uses mosaiced convolutions to preserve the 2×2 filter pattern and a real-world data acquisition method via consumer LCD screens to train on camera-specific non-linearities. It achieves state-of-the-art performance with a mean absolute error of 0.46° in polarisation angle estimation, outperforming both algorithmic and learning-based baselines.

ABSTRACT

Polarisation Filter Array (PFA) cameras allow the analysis of light polarisation state in a simple and cost-effective manner. Such filter arrays work as the Bayer pattern for colour cameras, sharing similar advantages and drawbacks. Among the others, the raw image must be demosaiced considering the local variations of the PFA and the characteristics of the imaged scene. Non-linear effects, like the cross-talk among neighbouring pixels, are difficult to explicitly model and suggest the potential advantage of a data-driven learning approach. However, the PFA cannot be removed from the sensor, making it difficult to acquire the ground-truth polarization state for training. In this work we propose a novel CNN-based model which directly demosaics the raw camera image to a per-pixel Stokes vector. Our contribution is twofold. First, we propose a network architecture composed by a sequence of Mosaiced Convolutions operating coherently with the local arrangement of the different filters. Second, we introduce a new method, employing a consumer LCD screen, to effectively acquire real-world data for training. The process is designed to be invariant by monitor gamma and external lighting conditions. We extensively compared our method against algorithmic and learning-based demosaicing techniques, obtaining a consistently lower error especially in terms of polarisation angle.

Motivation & Objective

• To address the challenge of demosaicing raw PFA camera images into full-resolution Stokes vectors, accounting for sensor-specific non-linearities like cross-talk and noise.
• To develop a data-driven approach that bypasses the limitations of synthetic or surrogate training data by using real-world measurements from a consumer LCD screen.
• To design a CNN architecture that explicitly models the local 2×2 filter pattern of PFA sensors through a novel mosaiced convolution operation.
• To improve demosaicing accuracy—especially in polarisation angle estimation—beyond existing algorithmic and deep learning methods.
• To enable camera-specific fine-tuning of the model using accessible, low-cost data collection, making it practical for real-world deployment.

Proposed method

• A novel Mosaiced Convolution block is introduced, which applies different convolutional kernels based on the local 2×2 arrangement of PFA filters (0°, 45°, 90°, 135°), preserving spatial and polarization structure.
• The PFADN network processes raw PFA images end-to-end to output both full-resolution intensity and polarisation angle (AoLP) maps.
• Real-world training data is acquired using a consumer LCD monitor, with a calibration method that compensates for monitor gamma and viewing angle non-linearities.
• The data acquisition pipeline ensures that the polarization state of the monitor's pixels is accurately mapped to the PFA sensor, enabling realistic training without removing the filter array.
• The network is trained to minimize both PSNR for intensity and MAE for polarisation angle, with loss functions tailored to the dual-output regression task.
• The method is invariant to monitor gamma and lighting conditions by leveraging the LCD's known polarization properties and controlled viewing geometry.

Figure 1: Our method demosaics a PFA camera image to produce a high-resolution intensity image (center) and angle of polarisation (right).

Experimental results

Research questions

• RQ1 Can a data-driven deep learning model achieve superior demosaicing performance on PFA cameras compared to traditional algorithmic methods?
• RQ2 Can real-world training data be effectively acquired using a consumer LCD screen without requiring specialized hardware or ground-truth polarization sensors?
• RQ3 Does a CNN architecture that explicitly models the PFA’s 2×2 filter pattern through mosaiced convolutions outperform standard convolutional layers in demosaicing accuracy?
• RQ4 To what extent can a model trained on real-world data capture sensor-specific non-linearities like cross-talk and noise, improving polarisation angle estimation?
• RQ5 Can the proposed method generalize across different scenes and camera instances while maintaining high accuracy in both intensity and polarisation angle reconstruction?

Key findings

• PFADN achieves a mean absolute error (MAE) of 0.46° in polarisation angle estimation on the test set, significantly outperforming all competing methods.
• The model reduces intensity image PSNR error to 32.31 dB, comparable to state-of-the-art upscaling methods, while achieving substantially lower angle error.
• Compared to the FSRCNN-based method, which uses separate models for intensity and angle, PFADN achieves better angle accuracy with a single unified architecture.
• In noisy input conditions (Gaussian noise σN up to 0.01), PFADN maintains superior performance in both PSNR and angle MAE, indicating robustness to degradation.
• Qualitative results show that PFADN produces sharper intensity images with cleaner, less noisy polarisation angle maps, especially near edges and in low-DOLP regions.
• The method effectively captures camera-specific non-linearities, as demonstrated by its consistent superiority over algorithmic methods like Bicubic and Intensity Correlation, even in complex scenes.

Figure 2: Overview of the proposed data aquisition procedure. An LCD screen is observed in several different poses. Three images are acquired for each pose: (i) a Charuco board $I_{C_{i}}$ , (ii) a random sample from the DIV2K dataset $I_{R_{i}}$ and (iii) a black image. The black image is subtracte