[논문 리뷰] Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning

Nicolas Papernot, Patrick McDaniel|arXiv (Cornell University)|2018. 03. 13.

Adversarial Robustness in Machine Learning참고 문헌 88인용 수 368

한 줄 요약

본 논문은 Deep k-Nearest Neighbors (DkNN)를 소개한다. 이는 DNN에서 계층별 최근접 이웃을 활용하여 컨피던스, 해석가능성, 그리고 로버스트성을 (adversarial 입력에 대해서도 포함) conformal prediction을 통해 제공하는 하이브리드 분류기이다.

ABSTRACT

Deep neural networks (DNNs) enable innovative applications of machine learning like image recognition, machine translation, or malware detection. However, deep learning is often criticized for its lack of robustness in adversarial settings (e.g., vulnerability to adversarial inputs) and general inability to rationalize its predictions. In this work, we exploit the structure of deep learning to enable new learning-based inference and decision strategies that achieve desirable properties such as robustness and interpretability. We take a first step in this direction and introduce the Deep k-Nearest Neighbors (DkNN). This hybrid classifier combines the k-nearest neighbors algorithm with representations of the data learned by each layer of the DNN: a test input is compared to its neighboring training points according to the distance that separates them in the representations. We show the labels of these neighboring points afford confidence estimates for inputs outside the model's training manifold, including on malicious inputs like adversarial examples--and therein provides protections against inputs that are outside the models understanding. This is because the nearest neighbors can be used to estimate the nonconformity of, i.e., the lack of support for, a prediction in the training data. The neighbors also constitute human-interpretable explanations of predictions. We evaluate the DkNN algorithm on several datasets, and show the confidence estimates accurately identify inputs outside the model, and that the explanations provided by nearest neighbors are intuitive and useful in understanding model failures.

연구 동기 및 목표

• Leverage the modular representations of DNNs to assess prediction conformity with training data across all layers.
• Provide reliable confidence estimates that reflect nonconformity to the training manifold.
• Improve interpretability by exposing training exemplars that explain predictions.
• Enhance robustness to adversarial inputs by detecting nonconformal predictions across layers.

제안 방법

• For a test input, compute the l-layer representations produced by the trained DNN.
• At each layer, find the k nearest training representations using locality-sensitive hashing (LSH).
• Collect the labels of the k nearest neighbors at each layer into multisets Ωλ.
• Use conformal prediction to compute nonconformity α(x,j) based on Ωλ and calibration data.
• Compute p-values p_j(z) for each class j and output the prediction with the highest p-value, plus associated confidence and credibility.

실험 결과

연구 질문

• RQ1 How can layer-wise representations in a DNN be used to assess the conformity of a prediction with training data?
• RQ2 Can a calibrated confidence measure be produced that reflects nonconformity to the training manifold?
• RQ3 Do layer-wise nearest-neighbor explanations improve interpretability and help detect adversarial or out-of-distribution inputs?
• RQ4 Does the approach enhance robustness by ensuring predictions are supported across multiple representations within the network.

주요 결과

• DkNN yields credibility estimates that identify inputs far from the training manifold better than standard DNN confidence.
• On out-of-distribution or geometrically transformed inputs, DkNN credibility is below 10% versus 20%–50% for a DNN.
• Nearest-neighbor explanations provide intuitive, human-understandable predictions across layers.
• DkNN identifies adversarial examples via low credibility, and adaptive attacks often require perturbing input semantics to change the prediction.
• Predictions maintain integrity when supported by the training manifold across layers, indicating robustness and interpretability.