[论文解读] Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration

Xiao Wang, Hongrui Liu+2 more|arXiv (Cornell University)|Sep 29, 2021

Advanced Graph Neural Networks参考文献 39被引用 36

一句话总结

本文表明 GNNs 通常表现出低于信心水平的置信度，并提出 CaGCN，这是一个考虑拓扑结构的事后校准模型，以及用于经过校准自训练以提升校准和准确性的 CaGCN-st。

ABSTRACT

Despite Graph Neural Networks (GNNs) have achieved remarkable accuracy, whether the results are trustworthy is still unexplored. Previous studies suggest that many modern neural networks are over-confident on the predictions, however, surprisingly, we discover that GNNs are primarily in the opposite direction, i.e., GNNs are under-confident. Therefore, the confidence calibration for GNNs is highly desired. In this paper, we propose a novel trustworthy GNN model by designing a topology-aware post-hoc calibration function. Specifically, we first verify that the confidence distribution in a graph has homophily property, and this finding inspires us to design a calibration GNN model (CaGCN) to learn the calibration function. CaGCN is able to obtain a unique transformation from logits of GNNs to the calibrated confidence for each node, meanwhile, such transformation is able to preserve the order between classes, satisfying the accuracy-preserving property. Moreover, we apply the calibration GNN to self-training framework, showing that more trustworthy pseudo labels can be obtained with the calibrated confidence and further improve the performance. Extensive experiments demonstrate the effectiveness of our proposed model in terms of both calibration and accuracy.

研究动机与目标

• 识别在半监督节点分类中 GNNs 是否经过良好校准。
• 证明现有 GNNs 存在低置信度（欠自信），及其对可信预测的影响。
• 提出 CaGCN，一种考虑拓扑结构的非线性、保持准确性的 GCN 校准函数。
• 展示经过校准的置信度可以提升 GNN 的自训练性能。
• 在标准数据集上展示在校准指标和预测准确度方面的实证提升。

提出的方法

• 使用可靠性图和置信度分布，在 Cora、Citeseer、Pubmed 以及 CoraFull 上证明 GCN 和 GAT 存在欠自信。
• 提出 CaGCN，是一个校准的 GCN，它接收来自分类 GCN 的 logits，并沿着图的拓扑传播经过校准的置信度，以产生节点级别的校准概率。
• 通过设计类温度缩放的机制来确保准确性保持，使用另一个 GCN 学习节点级温度，保持类别顺序；提供关于等单调性的证明。
• 将负对数似然（NLL）与置信校准正则化器相结合的优化公式，以在正确预测上提高置信度、在错误预测上降低置信度。
• 将 CaGCN 扩展为 CaGCN-st，一种自训练框架，在该框架中经过校准的置信度生成伪标签，从而实现分阶段再训练并提升性能。

实验结果

研究问题

• RQ1 What is the calibration state of common GNNs (GCN, GAT) on semi-supervised node classification?
• RQ2 Can topology-aware calibration improve confidence estimates without sacrificing accuracy?
• RQ3 How can a calibration function be designed to preserve the original classifier's accuracy while improving calibration?
• RQ4 Does calibrating confidence benefit self-training for unlabeled nodes in graph settings?
• RQ5 What empirical gains in calibration metrics and accuracy can CaGCN and CaGCN-st achieve on standard datasets?

主要发现

• GNNs（GCN、GAT）存在欠自信，可靠性图显示在各数据集上准确度高于置信度。
• CaGCN 提升了校准（ECE 较低），并在与未校准模型、TS 和 MS 基线相比时保持或提升分类准确性。
• CaGCN-st 在 Cora、Citeseer、Pubmed、CoraFull 的多种标签率下，节点分类准确度持续超越基线。
• 通过 CaGCN 校准置信度能够实现有效的自训练，获得高于传统自训练方法的准确性。
• 所提方法利用对置信度的拓扑感知平滑，实现保持准确性的校准。