Differentially-Private Synthetic Visit Infilling with Transformers

Modeling user mobility patterns from trajectory data is valuable for a broad range of applications, e.g., urban planning, route optimization, and public health analysis. However, real-world trajectories are often incomplete, missing places visited by the user. Transformer-based architectures have recently demonstrated strong performance on the visit infilling task, which consists of reconstructing missing visits in a user trajectory by capturing complex spatial and temporal dependencies. Since trajectory data are inherently sensitive, protecting the privacy of individuals who contribute their data to training the model is essential. Differential Privacy (DP) is a prominent protection mechanism which provides strong protection guarantees. However, directly applying DP to train transformers on trajectory data results in severe accuracy degradation, due to the high-dimensional nature of the inputs. We identify the root causes of accuracy loss in DP-compliant transformer training for visit infilling, and propose a set of techniques to improve the privacy-utility trade-off without weakening the formal (ϵ,δ) protection guarantees. We show experimentally that our approach substantially improves accuracy over private baselines, especially in high-privacy regimes, and may achieve performance comparable to non-private training.