[Paper Review] Multi-Objective GFlowNets

Moksh Jain, Sharath Chandra Raparthy+5 more|arXiv (Cornell University)|Oct 23, 2022

Advanced Multi-Objective Optimization Algorithms4 citations

TL;DR

This paper introduces Multi-Objective GFlowNets (MOGFNs), a novel framework for generating diverse, Pareto-optimal solutions in multi-objective optimization. It proposes two variants: MOGFN-PC, which uses reward-conditional GFlowNets with scalarization to model sub-problems, and MOGFN-AL, which employs an active learning loop with acquisition functions. The method achieves superior Pareto performance and significantly improved candidate diversity on synthetic, molecular, and protein design tasks.

ABSTRACT

We study the problem of generating diverse candidates in the context of Multi-Objective Optimization. In many applications of machine learning such as drug discovery and material design, the goal is to generate candidates which simultaneously optimize a set of potentially conflicting objectives. Moreover, these objectives are often imperfect evaluations of some underlying property of interest, making it important to generate diverse candidates to have multiple options for expensive downstream evaluations. We propose Multi-Objective GFlowNets (MOGFNs), a novel method for generating diverse Pareto optimal solutions, based on GFlowNets. We introduce two variants of MOGFNs: MOGFN-PC, which models a family of independent sub-problems defined by a scalarization function, with reward-conditional GFlowNets, and MOGFN-AL, which solves a sequence of sub-problems defined by an acquisition function in an active learning loop. Our experiments on wide variety of synthetic and benchmark tasks demonstrate advantages of the proposed methods in terms of the Pareto performance and importantly, improved candidate diversity, which is the main contribution of this work.

Motivation & Objective

• To address the lack of diversity in existing multi-objective optimization (MOO) methods, which focus only on Pareto optimality without considering the need for diverse candidates.
• To leverage the inherent diversity of GFlowNets—sampling candidates proportional to reward—for generating varied, high-quality solutions in MOO.
• To enable efficient and effective candidate generation in scientific discovery tasks where objectives are imperfect proxies and downstream evaluations are costly.
• To develop two novel variants: MOGFN-PC for scalarized sub-problems and MOGFN-AL for active learning-based MOO.
• To empirically validate that MOGFNs improve both Pareto performance and diversity across diverse benchmark and real-world tasks.

Proposed method

• MOGFN-PC uses reward-conditional GFlowNets to model a family of single-objective sub-problems derived via scalarization of the multi-objective problem.
• MOGFN-AL formulates MOO as a sequence of single-objective sub-problems within an active learning loop, using NEHVI as the acquisition function to guide candidate generation.
• The policy network in the GFlowNet, implemented as a Transformer encoder, generates mutations by selecting positions and amino acid tokens in sequences, with logits masked to prevent duplicate sequences.
• A reward exponent β is used and annealed over time to stabilize training, with the acquisition function value used as the reward signal.
• The method employs a uniform random policy mix δ to balance exploration and exploitation during training.
• The surrogate model is a multi-task GP with an ICM kernel in the latent space of a 1D convolutional encoder, trained with standard deep learning techniques and early stopping.

Experimental results

Research questions

• RQ1 Can GFlowNets be effectively extended to multi-objective optimization to generate diverse, Pareto-optimal solutions?
• RQ2 Does MOGFN-PC, based on scalarization and reward-conditional GFlowNets, achieve better diversity and Pareto performance than existing MOO baselines?
• RQ3 Can MOGFN-AL, using an active learning loop with NEHVI acquisition, improve sample efficiency and diversity in expensive scientific discovery tasks?
• RQ4 How do key hyperparameters such as β, δ, and maximum mutations affect the performance and diversity of generated candidates?
• RQ5 To what extent does candidate diversity improve the likelihood of success in downstream evaluations when objectives are imperfect proxies?

Key findings

• MOGFN-PC successfully generates diverse, Pareto-optimal candidates on synthetic and molecular generation tasks, marking the first empirical validation of reward-conditional GFlowNets in this context.
• In the active learning setting for fluorescent protein design, MOGFN-AL achieved significant improvements in sample efficiency and diversity compared to baseline methods.
• The method demonstrated superior performance on the Pareto front, with higher hypervolume improvement and better coverage of trade-offs across objectives.
• The ablation study confirmed that the reward exponent β and policy mix δ are critical for balancing exploration and exploitation, directly impacting diversity and convergence.
• The use of a uniform random policy mix δ improved diversity without sacrificing performance, especially in early training phases.
• The framework outperformed strong baselines in both diversity metrics and Pareto front quality across all benchmark tasks, including sequence and molecule generation.