[Paper Review] The rate of linear convergence of the Douglas-Rachford algorithm for subspaces is the cosine of the Friedrichs angle

Heinz H. Bauschke, Yunier Bello-Cruz+3 more|arXiv (Cornell University)|Sep 18, 2013

Optimization and Variational Analysis21 references20 citations

TL;DR

This paper establishes that the Douglas-Rachford algorithm for finding the intersection of two subspaces in a Hilbert space converges linearly to the projection of the initial point onto the intersection, with the rate of convergence being the cosine of the Friedrichs angle between the subspaces. The analysis is conducted in general (possibly infinite-dimensional) Hilbert spaces, providing sharp convergence rates and identifying the limit as the best approximation solution.

ABSTRACT

The Douglas-Rachford splitting algorithm is a classical optimization method that has found many applications. When specialized to two normal cone operators, it yields an algorithm for finding a point in the intersection of two convex sets. This method for solving feasibility problems has attracted a lot of attention due to its good performance even in nonconvex settings. In this paper, we consider the Douglas-Rachford algorithm for finding a point in the intersection of two subspaces. We prove that the method converges strongly to the projection of the starting point onto the intersection. Moreover, if the sum of the two subspaces is closed, then the convergence is linear with the rate being the cosine of the Friedrichs angle between the subspaces. Our results improve upon existing results in three ways: First, we identify the location of the limit and thus reveal the method as a best approximation algorithm; second, we quantify the rate of convergence, and third, we carry out our analysis in general (possibly infinite-dimensional) Hilbert space. We also provide various examples as well as a comparison with the classical method of alternating projections.

Motivation & Objective

• To identify the exact limit of the Douglas-Rachford iterates when applied to two subspaces.
• To quantify the linear convergence rate of the algorithm in the subspace case.
• To extend the analysis beyond finite-dimensional spaces to general Hilbert spaces.
• To compare the Douglas-Rachford method with the method of alternating projections in terms of convergence speed.
• To establish sharp convergence estimates using the Friedrichs angle as the key parameter.

Proposed method

• The analysis employs firmly nonexpansive operators and projection theory in Hilbert spaces.
• The Douglas-Rachford operator is defined as $ T = P_V(2P_U - \text{Id}) + \text{Id} - P_U $, where $ P_U $ and $ P_V $ are orthogonal projections onto subspaces $ U $ and $ V $.
• The limit of the sequence $ (x_n) $ is shown to be $ P_{U\cap V}x_0 $, the best approximation to $ x_0 $ in $ U \cap V $.
• The convergence rate is derived using the Friedrichs angle $ \theta $, with the rate being $ \cos\theta $, and this bound is proven to be sharp.
• The proof relies on properties of firmly nonexpansive mappings and spectral analysis of the Douglas-Rachford operator.
• The analysis is carried out in general Hilbert spaces, including infinite-dimensional cases, and is supported by numerical experiments and counterexamples.

Experimental results

Research questions

• RQ1 What is the exact limit point to which the Douglas-Rachford iterates converge when applied to two subspaces?
• RQ2 What is the precise rate of linear convergence of the Douglas-Rachford algorithm in the subspace case?
• RQ3 How does the convergence rate depend on the geometric relationship between the subspaces?
• RQ4 Is the convergence rate sharp, and can it be expressed in terms of the Friedrichs angle?
• RQ5 How does the Douglas-Rachford method compare to the method of alternating projections in terms of convergence speed?

Key findings

• The Douglas-Rachford iterates converge linearly to $ P_{U\cap V}x_0 $, the best approximation of the initial point in the intersection of the subspaces.
• The linear convergence rate is exactly $ \cos\theta $, where $ \theta $ is the Friedrichs angle between the subspaces, and this rate is sharp.
• The convergence analysis holds in general (possibly infinite-dimensional) Hilbert spaces, not just finite-dimensional ones.
• When the sum of the subspaces is closed, the convergence is linear with the rate $ \cos\theta $, and this condition is necessary for linear convergence.
• Numerical experiments show that the Douglas-Rachford method outperforms the method of alternating projections when the Friedrichs angle is small (i.e., $ \theta < 0.1 $).
• The method exhibits a 'rippling' behavior in the shadow sequence, which is more pronounced for small angles, but this does not hinder the overall convergence rate.