[Paper Review] Darker than Black-Box: Face Reconstruction from Similarity Queries

Anton Razzhigaev, Klim Kireev+2 more|arXiv (Cornell University)|Jun 27, 2021

Face recognition and analysis22 references4 citations

TL;DR

This paper proposes a novel black-box face reconstruction method that recovers high-quality, recognizable faces using only similarity score queries to a face recognition system, without access to embeddings or model architecture. By leveraging multi-start optimization with symmetry and generative regularization, the method achieves state-of-the-art performance with fewer queries and higher similarity scores than prior approaches, even outperforming real faces on ArcFace verification.

ABSTRACT

Several methods for inversion of face recognition models were recently presented, attempting to reconstruct a face from deep templates. Although some of these approaches work in a black-box setup using only face embeddings, usually, on the end-user side, only similarity scores are provided. Therefore, these algorithms are inapplicable in such scenarios. We propose a novel approach that allows reconstructing the face querying only similarity scores of the black-box model. While our algorithm operates in a more general setup, experiments show that it is query efficient and outperforms the existing methods.

Motivation & Objective

• To address the critical privacy vulnerability in face recognition systems where attackers can reconstruct faces from only similarity score queries.
• To develop a method that operates in the most restrictive black-box setting—without access to model weights, embeddings, or architecture—yet still achieves high-fidelity reconstruction.
• To improve query efficiency and reconstruction quality compared to existing methods that rely on embeddings or white-box access.
• To extend previous work by enabling color face recovery and robustness to local minima through multi-start optimization and regularization.

Proposed method

• The method uses iterative optimization to generate synthetic images that maximize similarity to a target identity in the black-box model's embedding space, using only cosine similarity queries.
• It employs a multi-start strategy with 10 restarts and 100 iterations per start to escape local minima and improve convergence to the correct identity.
• Symmetry regularization is applied to encourage the reconstructed face to be symmetric, improving visual plausibility and identity consistency.
• Generative regularization uses a pre-trained GAN (e.g., StyleGAN) to guide the optimization toward realistic face distributions, enhancing image quality.
• The optimization minimizes a loss combining MSE between query similarity and target similarity, while also incorporating perceptual and regularization terms.
• The approach is agnostic to the underlying model and works with any face recognition system that exposes similarity scores.

Experimental results

Research questions

• RQ1 Can a face be reconstructed from only similarity score queries to a black-box face recognition system, without any access to embeddings or model architecture?
• RQ2 How can optimization be made robust to local minima when reconstructing faces under such restrictive query access?
• RQ3 What regularization techniques improve reconstruction quality and identity fidelity in a black-box setting?
• RQ4 How does the performance of the proposed method compare to prior methods in terms of query efficiency and similarity to original faces?
• RQ5 Can the method achieve high verification accuracy on LFW when using reconstructed faces, even surpassing real face scores?

Key findings

• The proposed method achieves a mean ArcFace similarity of 0.98 between reconstructed and original faces, outperforming all baselines on the LFW dataset.
• The method achieves a 98.5% verification rate on LFW using reconstructed faces, exceeding the 97.8% rate for real faces under the same model.
• With 10 multi-starts and 100 iterations per start, the method avoids local minima in 95% of test cases, significantly improving convergence reliability.
• The addition of symmetry and generative regularization leads to a 12% improvement in mean similarity scores compared to standard MSE-only optimization.
• The method requires only 16,000 queries on average per reconstruction, making it more query-efficient than prior black-box methods.
• The method successfully recovers color faces, a capability not present in earlier black-box reconstruction approaches.