[Paper Review] The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection

Geoffroi Côté, Fahim Mannan+3 more|arXiv (Cornell University)|Dec 8, 2022

Industrial Vision Systems and Defect Detection6 citations

TL;DR

This paper introduces a differentiable spherical lens simulation model that enables end-to-end optimization of compound lens designs, including glass materials, by introducing quantized continuous glass variables. It achieves improved object detection performance in automotive applications—even with simplified two- or three-element lenses—despite reduced image quality, by jointly optimizing lens geometry, materials, and downstream neural networks under manufacturable constraints.

ABSTRACT

Most camera lens systems are designed in isolation, separately from downstream computer vision methods. Recently, joint optimization approaches that design lenses alongside other components of the image acquisition and processing pipeline -- notably, downstream neural networks -- have achieved improved imaging quality or better performance on vision tasks. However, these existing methods optimize only a subset of lens parameters and cannot optimize glass materials given their categorical nature. In this work, we develop a differentiable spherical lens simulation model that accurately captures geometrical aberrations. We propose an optimization strategy to address the challenges of lens design -- notorious for non-convex loss function landscapes and many manufacturing constraints -- that are exacerbated in joint optimization tasks. Specifically, we introduce quantized continuous glass variables to facilitate the optimization and selection of glass materials in an end-to-end design context, and couple this with carefully designed constraints to support manufacturability. In automotive object detection, we report improved detection performance over existing designs even when simplifying designs to two- or three-element lenses, despite significantly degrading the image quality.

Motivation & Objective

• Address the limitation of existing lens design methods that optimize only geometric parameters and ignore glass material selection due to their categorical nature.
• Enable joint optimization of lens design and downstream vision networks by making glass material selection differentiable through quantized continuous variables.
• Overcome challenges in lens optimization, such as non-convex loss landscapes and manufacturing constraints, via a differentiable simulation framework.
• Improve object detection performance in real-world automotive scenarios by co-designing lenses and neural networks, even with simplified lens configurations.
• Ensure manufacturability by incorporating practical constraints into the optimization pipeline for real-world deployability.

Proposed method

• Develop a differentiable spherical lens simulation model that accurately captures geometrical aberrations using ray-tracing principles.
• Introduce quantized continuous glass variables to represent discrete glass materials in a differentiable manner, enabling gradient-based optimization over material choices.
• Formulate the lens design optimization problem with constraints on surface curvature, thickness, and material selection to ensure manufacturability.
• Integrate the differentiable lens model into an end-to-end training pipeline that jointly optimizes lens parameters and the downstream object detection network.
• Apply gradient-based optimization techniques to navigate the non-convex loss landscape of lens design while respecting physical and manufacturing bounds.
• Use a differentiable image quality metric to guide lens optimization, balancing image fidelity and downstream detection performance.

Experimental results

Research questions

• RQ1 Can differentiable lens simulation enable end-to-end optimization of both lens geometry and glass material selection?
• RQ2 How does joint optimization of lens design and deep learning-based object detection improve performance compared to isolated lens design?
• RQ3 To what extent can simplified lens configurations (e.g., two- or three-element lenses) achieve superior detection performance when optimized jointly with the vision pipeline?
• RQ4 Can quantized continuous glass variables effectively enable gradient-based optimization over discrete glass materials?
• RQ5 How do manufacturability constraints affect the optimization process and final lens design quality?

Key findings

• The proposed method achieves improved object detection performance in automotive scenarios even when using simplified two- or three-element lens designs.
• Despite a significant degradation in image quality, the joint optimization of lens and network leads to better detection accuracy than conventional lens designs.
• The use of quantized continuous glass variables enables effective differentiable optimization over discrete glass materials, overcoming a major limitation in prior work.
• The differentiable lens simulation accurately models geometrical aberrations, enabling reliable optimization of complex lens systems.
• Manufacturability constraints are successfully integrated into the optimization process, resulting in designs that are feasible for real-world fabrication.
• The end-to-end optimization framework demonstrates that lens design and vision network training can be co-optimized to yield superior system-level performance.