[Paper Review] Provable Non-Convex Optimization and Algorithm Validation via Submodularity

Yatao Bian|arXiv (Cornell University)|Jan 1, 2019

Complexity and Algorithms in Graphs87 references2 citations

TL;DR

This thesis introduces continuous submodularity as a framework for provable non-convex optimization, enabling efficient approximation algorithms with theoretical guarantees for maximizing non-convex, non-concave functions. It further applies information-theoretic algorithm validation to analyze MaxCut algorithms, revealing robustness differences through algorithmic information content.

ABSTRACT

Submodularity is one of the most well-studied properties of problem classes in combinatorial optimization and many applications of machine learning and data mining, with strong implications for guaranteed optimization. In this thesis, we investigate the role of submodularity in provable non-convex optimization and validation of algorithms. A profound understanding which classes of functions can be tractably optimized remains a central challenge for non-convex optimization. By advancing the notion of submodularity to continuous domains (termed "continuous submodularity"), we characterize a class of generally non-convex and non-concave functions -- continuous submodular functions, and derive algorithms for approximately maximizing them with strong approximation guarantees. Meanwhile, continuous submodularity captures a wide spectrum of applications, ranging from revenue maximization with general marketing strategies, MAP inference for DPPs to mean field inference for probabilistic log-submodular models, which renders it as a valuable domain knowledge in optimizing this class of objectives. Validation of algorithms is an information-theoretic framework to investigate the robustness of algorithms to fluctuations in the input/observations and their generalization ability. We investigate various algorithms for one of the paradigmatic unconstrained submodular maximization problem: MaxCut. Due to submodularity of the MaxCut objective, we are able to present efficient approaches to calculate the algorithmic information content of MaxCut algorithms. The results provide insights into the robustness of different algorithmic techniques for MaxCut.

Motivation & Objective

• To develop a theoretical framework for optimizing non-convex functions using continuous submodularity.
• To provide approximation guarantees for maximizing continuous submodular functions.
• To validate algorithms for submodular maximization problems using information-theoretic robustness analysis.
• To connect continuous submodularity to real-world applications such as revenue maximization, MAP inference, and mean field approximation.

Proposed method

• Introduces continuous submodularity as an extension of discrete submodularity to continuous domains.
• Defines continuous DR-submodular functions and establishes their structural properties.
• Proposes gradient-based and conditional gradient algorithms for maximizing continuous submodular functions with approximation guarantees.
• Derives efficient computation of algorithmic information content for MaxCut using submodularity.
• Applies information-theoretic measures to quantify robustness of MaxCut algorithms under input fluctuations.
• Uses submodular structure to enable tractable computation of algorithmic complexity and generalization insights.

Experimental results

Research questions

• RQ1 Can continuous submodularity enable provable approximation guarantees for non-convex optimization?
• RQ2 How can submodularity be leveraged to validate the robustness of algorithms for unconstrained submodular maximization?
• RQ3 What is the algorithmic information content of MaxCut algorithms, and how does it relate to their robustness?
• RQ4 To what extent can continuous submodular optimization model real-world machine learning and data mining problems?
• RQ5 How do different algorithmic techniques for MaxCut compare in terms of robustness and information content?

Key findings

• The paper establishes that continuous DR-submodular functions admit approximation algorithms with constant-factor guarantees, even though they are generally non-convex and non-concave.
• The proposed algorithms achieve an approximation ratio of 1/2 for continuous DR-submodular maximization under box constraints.
• Algorithmic information content for MaxCut is efficiently computable due to the submodularity of the objective, enabling robustness analysis.
• Different MaxCut algorithms exhibit varying robustness, with greedy and randomized methods showing distinct information-theoretic profiles.
• The framework reveals that submodular structure enables tractable validation of algorithmic robustness and generalization.
• Applications such as revenue maximization, DPP inference, and mean field approximation are shown to be naturally modeled within the continuous submodular framework.