[Paper Review] Distributed Picard Iteration: Application to Distributed EM and Distributed PCA.

Francisco L. Andrade, Mário A. T. Figueiredo+1 more|arXiv (Cornell University)|Jun 20, 2021

Sparse and Compressive Sensing Techniques18 references4 citations

TL;DR

This paper introduces distributed algorithms for Expectation-Maximization (EM) and Principal Component Analysis (PCA) using Distributed Picard Iteration (DPI), a method that enables agents in a network to converge to a fixed point of a locally contractive map through iterative communication. The key contribution is proving local linear convergence for both distributed EM and PCA by showing their update rules correspond to locally contractive maps that are averages of local maps, with convergence guarantees inherited from DPI.

ABSTRACT

In recent work, we proposed a distributed Picard iteration (DPI) that allows a set of agents, linked by a communication network, to find a fixed point of a locally contractive (LC) map that is the average of individual maps held by said agents. In this work, we build upon the DPI and its local linear convergence (LLC) guarantees to make several contributions. We show that Sanger's algorithm for principal component analysis (PCA) corresponds to the iteration of an LC map that can be written as the average of local maps, each map known to each agent holding a subset of the data. Similarly, we show that a variant of the expectation-maximization (EM) algorithm for parameter estimation from noisy and faulty measurements in a sensor network can be written as the iteration of an LC map that is the average of local maps, each available at just one node. Consequently, via the DPI, we derive two distributed algorithms - distributed EM and distributed PCA - whose LLC guarantees follow from those that we proved for the DPI. The verification of the LC condition for EM is challenging, as the underlying operator depends on random samples, thus the LC condition is of probabilistic nature.

Motivation & Objective

• To extend the Distributed Picard Iteration (DPI) framework to enable distributed computation of fixed points in networked systems.
• To demonstrate that Sanger's PCA algorithm can be reformulated as an iteration of a locally contractive map that is the average of local maps across agents.
• To show that a variant of the EM algorithm for sensor networks can also be expressed as an iteration of a locally contractive map under noisy, distributed data.
• To establish local linear convergence guarantees for distributed EM and PCA by verifying the locally contractive condition under probabilistic assumptions.

Proposed method

• Formulate Sanger's PCA algorithm as the iteration of a locally contractive map that is the average of local maps, each known to one agent holding a data subset.
• Reformulate a distributed EM variant as the iteration of a locally contractive map derived from local likelihood updates at each agent.
• Apply the Distributed Picard Iteration (DPI) framework to these reformulated maps, leveraging its existing local linear convergence (LLC) guarantees.
• Verify the locally contractive condition for the EM map using probabilistic analysis due to the dependence on random samples in sensor measurements.
• Ensure convergence by modeling the global map as the average of local maps, each accessible only at its respective agent, with communication only between neighbors.
• Use the DPI convergence framework to derive distributed algorithms for EM and PCA with theoretical convergence guarantees.

Experimental results

Research questions

• RQ1 Can Sanger's PCA algorithm be interpreted as an iteration of a locally contractive map that is the average of local maps across agents?
• RQ2 Can a distributed EM algorithm for sensor networks be expressed as the iteration of a locally contractive map under noisy data?
• RQ3 Does the Distributed Picard Iteration (DPI) framework provide local linear convergence guarantees for both distributed EM and PCA?
• RQ4 How can the locally contractive condition be verified for the EM map when it depends on random samples?

Key findings

• Sanger's PCA algorithm corresponds to the iteration of a locally contractive map that is the average of local maps, each known to one agent, enabling distributed implementation.
• A variant of the EM algorithm for distributed parameter estimation in sensor networks can be expressed as the iteration of a locally contractive map, allowing distributed computation.
• The locally contractive condition for the EM map is established in a probabilistic sense due to the dependence on random samples from noisy measurements.
• Both distributed EM and distributed PCA inherit the local linear convergence (LLC) guarantees of the DPI framework, ensuring fast convergence under mild conditions.
• The framework enables scalable, decentralized computation of PCA and EM by leveraging local computation and neighbor-wise communication.