[论文解读] Reconsidering utility: unveiling the limitations of synthetic mobility data generation algorithms in real-life scenarios

Alexandra Kapp, Helena Mihaljević|arXiv (Cornell University)|Jul 3, 2024

Human Mobility and Location-Based Analysis被引用 1

一句话总结

本文评估了五种最先进的合成出行数据生成模型在行程数据上的表现，通过将合成行程与OpenStreetMap道路网络进行地图匹配，并与一种隐私保护路由基线进行比较，评估其在现实世界中的实用性。尽管在空间分布方面表现良好，但所有模型均未能生成真实的行程长度、交叉口交通流，或保持时间与轨迹特征，仅有AdaTrace和PrivTrace提供了可用的差分隐私输出。

ABSTRACT

In recent years, there has been a surge in the development of models for the generation of synthetic mobility data. These models aim to facilitate the sharing of data while safeguarding privacy, all while ensuring high utility and flexibility regarding potential applications. However, current utility evaluation methods fail to fully account for real-life requirements. We evaluate the utility of five state-of-the-art synthesis approaches, each with and without the incorporation of differential privacy (DP) guarantees, in terms of real-world applicability. Specifically, we focus on so-called trip data that encode fine granular urban movements such as GPS-tracked taxi rides. Such data prove particularly valuable for downstream tasks at the road network level. Thus, our initial step involves appropriately map matching the synthetic data and subsequently comparing the resulting trips with those generated by the routing algorithm implemented in OpenStreetMap, which serves as an efficient and privacy-friendly baseline. Out of the five evaluated models, one fails to produce data within reasonable computation time and another generates too many jumps to meet the requirements for map matching. The remaining three models succeed to a certain degree in maintaining spatial distribution, one even with DP guarantees. However, all models struggle to produce meaningful sequences of geo-locations with reasonable trip lengths and to model traffic flow at intersections accurately. It is important to note that trip data encompasses various relevant characteristics beyond spatial distribution, such as temporal information, all of which are discarded by these models. Consequently, our results imply that current synthesis models fall short in their promise of high utility and flexibility.

研究动机与目标

• 评估合成出行数据生成模型在细粒度行程数据上的现实世界实用性，特别是在城市交通背景下的表现。
• 识别当前模型的缺陷，这些缺陷导致其无法达到或超越一种隐私友好型路由基线（OpenStreetMap路由引擎）的性能。
• 评估是否可以有意义地将差分隐私集成到合成行程生成中，而不会牺牲实用性。
• 挑战一种假设，即合成数据在灵活性和实用性方面具有优势，尤其是在道路网络层面的分析中，如交通量和速度估算。
• 倡导针对具体应用场景的建模方法，而非采用‘一刀切’的合成数据生成方式。

提出的方法

• 使用基于OSRM的路由方法，将五种最先进的生成模型生成的合成行程与OpenStreetMap道路网络进行地图匹配，以验证其合理性。
• 将合成行程的特征（如道路偏好、行程长度、交叉口流量）与OSRM路由引擎生成的特征进行对比，后者作为隐私保护基线。
• 通过人工评估（调查参与者）评估道路偏好的感知真实性，AdaTrace在该任务中达到90%准确率和F1 ≥ 0.7。
• 使用空间分布度量（包括6×6和28×28的网格化空间分辨率）测量统计相似性，以评估其对真实世界热点区域的保真度。
• 从多个维度评估实用性：行程长度与直线距离之比、交叉口交通流量，以及道路使用情况的统计相似性。
• 通过项目级差分隐私（item-level DP）评估差分隐私的集成，分析其对实用性与隐私权衡的影响。

实验结果

研究问题

• RQ1 什么是行程数据的高实用性？在真实交通场景中，如何衡量其表现？
• RQ2 最先进的合成数据生成模型在实用性度量上与隐私保护路由基线相比表现如何？
• RQ3 具有差分隐私保证的合成行程数据是否仍能实现足够实用性的实际应用？

主要发现

• TrajGAIL在城市规模场景下无法在合理计算时间内生成数据，导致其在现实世界中不可行。
• DP-Loc生成了过多的跳跃，使得地图匹配变得不可行，违反了道路网络对齐的基本要求。
• AdaTrace实现了最高实用性，使人工参与者能够以90%的准确率和F1 ≥ 0.7的得分识别出偏好道路。
• 仅有AdaTrace和PrivTrace生成了可用的差分隐私数据，且AdaTrace的DP版本在多项评估中优于无DP的PrivTrace。
• 所有模型在行程长度与直线距离之比指标上均表现低于基线，表明其行程几何形状缺乏真实性。
• 在交叉口交通流量建模方面，无一模型显著优于基线，AdaTrace仅略胜一筹，表明其实际应用价值有限。