[Paper Review] Relative Interior Rule in Block-Coordinate Minimization

Tomáš Werner, Daniel Průša|arXiv (Cornell University)|Oct 21, 2019

Advanced Numerical Analysis Techniques10 references4 citations

TL;DR

This paper proposes the relative interior rule for block-coordinate minimization, ensuring that at each iteration, the minimizer is selected from the relative interior of the optimal set over the current block. The key contribution is proving that methods adhering to this rule are not worse than any other block-coordinate method: they either maintain or improve convergence behavior, avoiding stagnation at suboptimal points and guaranteeing convergence to an interior local minimum under mild conditions.

ABSTRACT

(Block-)coordinate minimization is an iterative optimization method which in every iteration finds a global minimum of the objective over a variable or a subset of variables, while keeping the remaining variables constant. While for some problems, coordinate minimization converges to a global minimum (e.g., convex differentiable objective), for general (non-differentiable) convex problems this may not be the case. Despite this drawback, (block-)coordinate minimization can be an acceptable option for large-scale non-differentiable convex problems; an example is methods to solve the linear programming relaxation of the discrete energy minimization problem (MAP inference in graphical models). When block-coordinate minimization is applied to a general convex problem, in every iteration the minimizer over the current coordinate block need not be unique and therefore a single minimizer must be chosen. We propose that this minimizer be chosen from the relative interior of the set of all minimizers over the current block. We show that this rule is not worse, in a certain precise sense, than any other rule.

Motivation & Objective

• To address the instability and suboptimal convergence of block-coordinate minimization methods due to non-unique minimizers over coordinate blocks.
• To formalize a principled selection rule for minimizers that enhances convergence quality and robustness.
• To prove that selecting minimizers from the relative interior ensures superior or equal performance compared to arbitrary minimizer choices.
• To establish theoretical guarantees that such methods avoid getting trapped in poor local minima and converge to better-quality solutions.

Proposed method

• Introduces the concept of an interior local minimum, defined as a point where the minimizer over each block lies in the relative interior of the optimal set.
• Replaces standard block-coordinate minimization (which selects any minimizer) with a rule that always selects from the relative interior of the minimizer set.
• Uses the epigraph representation of the objective function and lifts the search space to include the function value, enabling analysis via convex geometry.
• Defines pre-interior local minima as points that reach an interior local minimum in finitely many steps under the relative interior rule.
• Applies the relative interior rule via the update: $ x_{n+1} riangleq ext{ri}(M(X igcap (x_n + I_n), f)) $, ensuring robustness and convergence to better solutions.
• Proves equivalence between local minima in the original space and in the epigraph space under the lifted subspace $ \bar{I} = I \times \mathbb{R} $.

Experimental results

Research questions

• RQ1 Can a principled rule for selecting minimizers in block-coordinate minimization improve convergence quality and avoid poor local minima?
• RQ2 Does selecting minimizers from the relative interior of the optimal set lead to better or at least no worse convergence behavior than arbitrary selection?
• RQ3 What is the relationship between pre-interior local minima and interior local minima under the relative interior rule?
• RQ4 How does the relative interior rule compare to standard block-coordinate methods in terms of convergence and objective value improvement?

Key findings

• Any sequence generated by the relative interior rule maintains or improves the objective value, and if the initial point is a pre-interior local minimum, the method reaches an interior local minimum in finite steps.
• If the initial point is not a pre-interior local minimum, the objective value strictly decreases after finitely many steps, ensuring progress.
• The relative interior rule guarantees that once an interior local minimum is reached, it is preserved in all subsequent iterates.
• The method is not worse than any other block-coordinate minimization method: it either maintains or improves upon the best possible outcome achievable by arbitrary minimizer selection.
• The epigraph reformulation preserves the structure of local minima, and the relative interior rule corresponds exactly to selecting points in the relative interior of the epigraph's minimizer set.
• Max-sum diffusion satisfies the relative interior rule, and the proposed method generalizes and justifies its favorable empirical performance.