[Paper Review] A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression

Kai Yang, Tao Fan+3 more|arXiv (Cornell University)|Dec 1, 2019

Privacy-Preserving Technologies in Data12 references57 citations

TL;DR

The paper proposes a quasi-Newton vertical federated learning framework for logistic regression using additively homomorphic encryption to reduce communication rounds, achieving faster convergence with modest per-round cost compared to a first-order SGD baseline.

ABSTRACT

Data privacy and security becomes a major concern in building machine learning models from different data providers. Federated learning shows promise by leaving data at providers locally and exchanging encrypted information. This paper studies the vertical federated learning structure for logistic regression where the data sets at two parties have the same sample IDs but own disjoint subsets of features. Existing frameworks adopt the first-order stochastic gradient descent algorithm, which requires large number of communication rounds. To address the communication challenge, we propose a quasi-Newton method based vertical federated learning framework for logistic regression under the additively homomorphic encryption scheme. Our approach can considerably reduce the number of communication rounds with a little additional communication cost per round. Numerical results demonstrate the advantages of our approach over the first-order method.

Motivation & Objective

• Motivate privacy-preserving collaborative learning when data is vertically partitioned across two parties.
• Develop a quasi-Newton method to accelerate convergence in vertical FL for logistic regression.
• Reduce communication rounds while controlling additional per-round communication cost.
• Leverage additively homomorphic encryption to enable secure gradient and loss computations.
• Provide empirical evidence comparing against a first-order SGD-based vertical FL approach.

Proposed method

• Adopt Taylor approximation of the logistic loss to enable encrypted computations under additively homomorphic encryption.
• Use sub-sampled Hessian information and update curvature every L iterations to stabilize the quasi-Newton update.
• Compute encrypted loss and gradient via exchanging encrypted intermediate values between parties A and B.
• Aggregate encrypted gradient and loss at a coordinator to compute a descent direction g̃ = H g with learning rate η.
• Update the inverse Hessian H using a limited-memory BFGS-style scheme with a memory M and batches S and SH.
• Show that communication costs per round are bounded and compare against SGD-based vertical FL.

Experimental results

Research questions

• RQ1 Can a stochastic quasi-Newton approach reduce the number of communication rounds in vertical federated learning for logistic regression?
• RQ2 What is the trade-off between communication cost per round and total rounds when using sub-sampled Hessians in encrypted settings?
• RQ3 How does the proposed framework compare to a first-order SGD-based vertical FL method in terms of convergence and performance on real datasets?
• RQ4 Is secure computation maintained via additively homomorphic encryption while achieving faster convergence?

Key findings

• The quasi-Newton vertical FL framework reduces the number of communication rounds compared with the SGD-based baseline.
• With S H and L chosen as in experiments, the method achieves faster convergence (fewer epochs) while maintaining similar loss and AUC on test data.
• The encryption-based framework incurs slightly higher per-round cost but reduces total communication rounds by leveraging curvature information.
• The reported experiments on credit scoring datasets show lower epochs and comparable or better performance metrics (Loss, AUC) for the proposed method in several settings.
• Overall, the method demonstrates communication efficiency improvements over the first-order approach in the studied vertical partitioning setup.