[論文レビュー] Defining 'Good': Evaluation Framework for Synthetic Smart Meter Data

Sheng Chai, Gus Chadney+4 more|arXiv (Cornell University)|Jul 16, 2024

Smart Grid Energy Management被引用数 5

ひとこと要約

本論文は忠実度、有用性、プライバシー評価フレームワークを合成スマートメータデータに適用し、新規指標とプライバシーテストを提案するとともに、Low Carbon Londonデータセット上でFaradayモデルを用い、差分プライバシー設定でそれらを実証している。

ABSTRACT

Access to granular demand data is essential for the net zero transition; it allows for accurate profiling and active demand management as our reliance on variable renewable generation increases. However, public release of this data is often impossible due to privacy concerns. Good quality synthetic data can circumnavigate this issue. Despite significant research on generating synthetic smart meter data, there is still insufficient work on creating a consistent evaluation framework. In this paper, we investigate how common frameworks used by other industries leveraging synthetic data, can be applied to synthetic smart meter data, such as fidelity, utility and privacy. We also recommend specific metrics to ensure that defining aspects of smart meter data are preserved and test the extent to which privacy can be protected using differential privacy. We show that standard privacy attack methods like reconstruction or membership inference attacks are inadequate for assessing privacy risks of smart meter datasets. We propose an improved method by injecting training data with implausible outliers, then launching privacy attacks directly on these outliers. The choice of $ε$ (a metric of privacy loss) significantly impacts privacy risk, highlighting the necessity of performing these explicit privacy tests when making trade-offs between fidelity and privacy.

研究の動機と目的

• Translate established synthetic data evaluation concepts (fidelity, utility, privacy) to the energy sector and personal smart meter data.
• Propose tailored metrics that capture smart meter characteristics such as spikiness and spatio-temporal hierarchies.
• Examine how privacy can be improved and measured using differential privacy and explicit privacy attacks.
• Assess trade-offs between data fidelity, utility, and privacy in synthetic smart meter generation.

提案手法

• Adopt fidelity, utility, and privacy as core evaluation concepts for synthetic smart meter data.
• Apply DP-SGD with PyTorch Opacus to control privacy loss during model training.
• Develop privacy attack experiments (reconstruction and membership inference) to complement differential privacy guarantees.
• Introduce novel distance-based and time-series-specific metrics to measure fidelity, including ACF distribution, peak timing, cluster distributions, and aggregated-level similarity.
• Use the Faraday generative model trained on the Low Carbon London dataset to generate synthetic data for evaluation.
• Explore the impact of varying privacy levels (epsilon) on fidelity and utility.

実験結果

リサーチクエスチョン

• RQ1 How can fidelity, utility, and privacy concepts be effectively defined and measured for synthetic smart meter data?
• RQ2 What metrics capture the unique time-series and hierarchical characteristics of smart meter data (e.g., spikiness, peaks, clustering)?
• RQ3 How do differential privacy and explicit privacy attacks complement each other in assessing privacy risk for synthetic smart meter outputs?
• RQ4 What are the trade-offs between fidelity, utility, and privacy when generating synthetic smart meter data?

主な発見

• Privacy attack tests reveal that standard privacy metrics may be insufficient for smart meter data and require explicit attack testing.
• Differential privacy level (epsilon) significantly affects privacy risk, necessitating explicit testing when balancing fidelity and privacy.
• Novel reconstruction-attack method using implausible outliers provides a practical measure of memorisation risk in time-series data.
• Fidelity and utility metrics tailored to smart meter data (ACF distribution, peak timing, cluster distributions, aggregated-level similarity) effectively compare real and synthetic data.
• The Faraday model with DP settings demonstrates measurable trade-offs between data fidelity, predictive utility, and privacy protection across dataset sizes and privacy levels.