[Paper Review] On Representing Linear Programs by Graph Neural Networks

Ziang Chen, Jialin Liu+3 more|arXiv (Cornell University)|Sep 25, 2022

Neural Networks and Applications9 citations

TL;DR

This paper establishes the theoretical foundations for using Graph Neural Networks (GNNs) to represent and solve Linear Programs (LPs), proving that GNNs can reliably predict feasibility, boundedness, and optimal solutions for a broad class of LPs. By leveraging the Weisfeiler-Lehman isomorphism test, the authors construct GNNs with sufficient expressive power to distinguish between different LP instances and map them to correct outputs, validated by numerical experiments showing high accuracy on feasibility and solution prediction.

ABSTRACT

Learning to optimize is a rapidly growing area that aims to solve optimization problems or improve existing optimization algorithms using machine learning (ML). In particular, the graph neural network (GNN) is considered a suitable ML model for optimization problems whose variables and constraints are permutation--invariant, for example, the linear program (LP). While the literature has reported encouraging numerical results, this paper establishes the theoretical foundation of applying GNNs to solving LPs. Given any size limit of LPs, we construct a GNN that maps different LPs to different outputs. We show that properly built GNNs can reliably predict feasibility, boundedness, and an optimal solution for each LP in a broad class. Our proofs are based upon the recently--discovered connections between the Weisfeiler--Lehman isomorphism test and the GNN. To validate our results, we train a simple GNN and present its accuracy in mapping LPs to their feasibilities and solutions.

Motivation & Objective

• To establish a theoretical foundation for using Graph Neural Networks (GNNs) to represent and solve Linear Programs (LPs), addressing a fundamental open problem in Learning to Optimize (L2O).
• To demonstrate that GNNs possess sufficient expressive power to distinguish between different LP instances and map them to correct outputs, including feasibility, boundedness, and optimal solutions.
• To validate the theoretical claims with empirical experiments, showing that a simple GNN can accurately predict key LP properties and solutions on synthetic data.
• To bridge the gap between theoretical expressiveness and practical performance in GNN-based optimization models, especially for permutation-invariant problems like LPs.

Proposed method

• The authors construct a GNN architecture that encodes LPs as bipartite graphs with variables and constraints as nodes, and constraints as edges, preserving permutation invariance.
• They use the Weisfeiler-Lehman (WL) isomorphism test as a theoretical foundation to prove that GNNs can distinguish non-isomorphic graph structures corresponding to different LPs.
• The GNN is designed with multi-layer message passing, where node and edge features are updated via learnable MLPs, and final predictions are generated via readout functions.
• Theoretical proofs rely on the fact that GNNs with sufficient depth and width can approximate any measurable, permutation-invariant function on LP instances, including feasibility, boundedness, and optimal solution prediction.
• A numerical experiment is conducted using synthetic LPs with m=10 constraints and n=50 variables, generated with sparse A matrices and random coefficients, to train and evaluate the GNN.
• The GNN is trained using Adam optimizer with mean squared error loss, and performance is evaluated on generalization to unseen test instances with metrics for feasibility, objective value, and solution accuracy.

Experimental results

Research questions

• RQ1 Can Graph Neural Networks (GNNs) theoretically represent and predict key properties of Linear Programs (LPs), such as feasibility and boundedness?
• RQ2 What is the expressive power of GNNs in distinguishing different LP instances, and can they reliably map distinct LPs to distinct outputs?
• RQ3 To what extent can a GNN model generalize to unseen LP instances when trained on a finite dataset, particularly in predicting optimal solutions?
• RQ4 How does the theoretical expressiveness of GNNs, grounded in the Weisfeiler-Lehman isomorphism test, translate into practical performance on LP prediction tasks?

Key findings

• Theoretical analysis proves that GNNs can represent any LP instance and predict its feasibility, boundedness, and optimal solution with high accuracy, provided the network has sufficient width and depth.
• The authors construct a GNN that maps different LPs to different outputs, demonstrating that GNNs can distinguish non-isomorphic LP structures via the Weisfeiler-Lehman framework.
• In numerical experiments, the GNN achieved 0% error on the training set for feasibility prediction when trained on 2500 samples, with only 17.5% error on the test set, showing strong generalization.
• For optimal objective value prediction, the GNN achieved a relative error of 0.173 on the test set when trained on 2500 samples, indicating good regression performance.
• For optimal solution prediction, the GNN achieved a relative error of 0.274 on the test set with 2500 training samples, confirming its ability to approximate complex solution mappings.
• The results show that increasing training data size improves generalization performance, suggesting that larger datasets enhance the GNN’s ability to generalize to unseen LP instances.