Differentially-Private Neural Network Training with Private Features and Public Labels

Training neural networks (NN) with differential privacy (DP) protection has been extensively studied in the past decade, with the DP-SGD (stochastic gradient descent) mechanism representing the benchmark approach. Conventional DP-SGD assumes that both the features and the labels of training samples must be protected. A recent variation of DP-SGD considers training when the input sample features are non-private, and only labels must be protected, which improves accuracy by reducing the amount of noise injected by DP. We argue that in some scenarios, the converse holds, namely the labels may be publicly known, while the features themselves are sensitive. We provide a customized technique for this setting, we identify several design trade-offs, and we show how one can factor in such trade-offs to revise the architecture of the NN in order to improve accuracy. Extensive experiments on real data show that our approach significantly outperforms the DP-SGD baseline.