[Paper Review] Composable Text Controls in Latent Space with ODEs

Guangyi Liu, Zeyu Feng+8 more|arXiv (Cornell University)|Aug 1, 2022

Topic Modeling4 citations

TL;DR

This paper proposes LatentOps, a method for composable text control in the continuous latent space of pretrained language models using ordinary differential equations (ODEs). By adapting a VAE-like framework to connect pretrained LMs to a compact latent space, it enables efficient sampling of text with arbitrary combinations of attributes (e.g., sentiment, formality, keywords) via ODE-based samplers, achieving superior generation quality and efficiency over prior search-based or finetuning approaches.

ABSTRACT

Real-world text applications often involve composing a wide range of text control operations, such as editing the text w.r.t. an attribute, manipulating keywords and structure, and generating new text of desired properties. Prior work typically learns/finetunes a language model (LM) to perform individual or specific subsets of operations. Recent research has studied combining operations in a plug-and-play manner, often with costly search or optimization in the complex sequence space. This paper proposes a new efficient approach for composable text operations in the compact latent space of text. The low-dimensionality and differentiability of the text latent vector allow us to develop an efficient sampler based on ordinary differential equations (ODEs) given arbitrary plug-in operators (e.g., attribute classifiers). By connecting pretrained LMs (e.g., GPT2) to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. The flexible approach permits diverse control operators (sentiment, tense, formality, keywords, etc.) acquired using any relevant data from different domains. Experiments show that composing those operators within our approach manages to generate or edit high-quality text, substantially improving over previous methods in terms of generation quality and efficiency.

Motivation & Objective

• To address the scalability and efficiency limitations of existing methods that finetune or search in discrete sequence space for text control operations.
• To enable arbitrary composition of diverse text control operators (e.g., sentiment, formality, keywords) in a single, unified framework.
• To overcome the incompatibility between discrete text sequences and continuous latent spaces in prior latent editing methods.
• To develop a fast, differentiable, and scalable sampling mechanism for generating high-quality text in latent space.
• To demonstrate that a single adapted language model can perform diverse, composable text operations without retraining.

Proposed method

• The method uses a variational autoencoder (VAE) framework to map text sequences to a low-dimensional, continuous latent space, enabling differentiability and efficient optimization.
• Pretrained language models (e.g., GPT-2) are adapted via parameter-efficient fine-tuning to reconstruct text from latent vectors, ensuring high-fidelity decoding.
• Arbitrary text control operators (e.g., sentiment classifiers, keyword detectors) are defined as energy functions in the latent space, forming an energy-based model (EBM).
• An ODE-based sampler is employed to generate latent vectors that satisfy the composed energy constraints, leveraging continuous dynamics for stable and efficient sampling.
• The ODE sampler is trained using score-based generative modeling principles, enabling high-quality, fluent text generation from sampled latent vectors.
• The framework supports both editing existing text and generating new text with specified attributes by composing multiple operators in the latent space.

Experimental results

Research questions

• RQ1 Can a single, unified framework efficiently compose multiple text control operations (e.g., sentiment, formality, keywords) without retraining?
• RQ2 Does sampling in the continuous latent space using ODEs outperform search-based or optimization-based methods in terms of generation quality and inference speed?
• RQ3 Can a single adapted language model generalize across diverse, compositional text control tasks?
• RQ4 How well does the ODE sampler maintain fluency and diversity while satisfying complex, multi-attribute constraints?
• RQ5 To what extent can the method scale to complex, real-world text editing and generation scenarios?

Key findings

• The ODE sampler achieves the best trade-off between generation quality, diversity, and inference speed, with a generation time of 5.5 seconds—significantly faster than SDE (15.6s) and competitive with SGLD (5.1s).
• LatentOps substantially improves generation quality over baselines on both editing and generation tasks, as measured by automatic metrics like BLEU, ROUGE, and human evaluation.
• The method enables high-fidelity text editing with multiple attributes applied simultaneously or sequentially, maintaining semantic coherence and fluency.
• The framework supports plug-and-play integration of diverse operators (e.g., sentiment, formality, keywords) trained on different domains, demonstrating strong composability.
• The VAE-based adaptation enables effective decoding from latent vectors, preserving the capabilities of the original pretrained LM (e.g., GPT-2) in the latent space.
• The ablation study confirms that the ODE sampler is more stable and less sensitive to hyperparameters than Langevin dynamics (SGLD) or SDE-based methods.