[Paper Review] Adversarial Constraint Learning for Structured Prediction

Hongyu Ren, Russell J. Stewart+3 more|arXiv (Cornell University)|May 27, 2018

Time Series Analysis and Forecasting17 references3 citations

TL;DR

This paper proposes adversarial constraint learning (ACL), a novel framework that implicitly learns structural constraints for structured prediction by training a model to generate outputs indistinguishable from a black-box simulator via adversarial training. The method enables high-accuracy semi-supervised structured prediction with minimal labeled data—achieving state-of-the-art performance on pose estimation, tracking, and time series prediction, even with zero labeled inputs when constraints are learned from simulation.

ABSTRACT

Constraint-based learning reduces the burden of collecting labels by having users specify general properties of structured outputs, such as constraints imposed by physical laws. We propose a novel framework for simultaneously learning these constraints and using them for supervision, bypassing the difficulty of using domain expertise to manually specify constraints. Learning requires a black-box simulator of structured outputs, which generates valid labels, but need not model their corresponding inputs or the input-label relationship. At training time, we constrain the model to produce outputs that cannot be distinguished from simulated labels by adversarial training. Providing our framework with a small number of labeled inputs gives rise to a new semi-supervised structured prediction model; we evaluate this model on multiple tasks --- tracking, pose estimation and time series prediction --- and find that it achieves high accuracy with only a small number of labeled inputs. In some cases, no labels are required at all.

Motivation & Objective

• To reduce reliance on manually specified, domain-specific constraints in structured prediction by learning them implicitly from data.
• To address the challenge of high-dimensional, complex output structures where hand-crafted constraints are impractical to define.
• To enable effective semi-supervised learning by combining small labeled datasets with unlabeled data constrained by adversarially learned invariants.
• To develop a framework that generalizes across diverse structured prediction tasks without requiring input-label pairs for constraint specification.

Proposed method

• The framework uses a generative adversarial network (GAN)-style setup, training a primary model and a discriminator to distinguish real simulated outputs from model-generated ones.
• The primary model is optimized to fool the discriminator, ensuring its outputs match the distribution of valid, simulated labels.
• A black-box simulator provides representative label samples (e.g., from physics engines or human-annotated data) without requiring input-label relationships.
• The method integrates supervised learning on a small labeled set with adversarial constraint learning on a large unlabeled set, forming a semi-supervised structured prediction model.
• The discriminator is trained to classify whether a given output comes from the simulator or the model, while the model is trained to maximize the discriminator’s confusion.
• The approach implicitly captures complex structural invariants (e.g., human pose plausibility, time series dynamics) without explicit formulaic constraints.

Experimental results

Research questions

• RQ1 Can adversarial training be used to implicitly learn structural constraints for structured prediction without requiring hand-crafted formulas?
• RQ2 How well does the proposed method perform when only a small number of labeled inputs are available?
• RQ3 Can the framework generalize across diverse structured prediction tasks such as pose estimation, tracking, and time series prediction?
• RQ4 Does adversarial constraint learning outperform existing semi-supervised and weakly supervised baselines in low-data regimes?
• RQ5 Can the model leverage incomplete or partially labeled data by learning constraints from a simulator?

Key findings

• On the Multi-Person Multi-Object Tracking (MOT) benchmark, ACL with only 7 labeled groups achieved a PCK@0.1 score of 0.9469, outperforming supervised learning with all labels (0.9622) when combined with adversarial training.
• In pose estimation on the MPII Human Pose dataset, ACL with 7 labeled groups achieved a PCK@0.1 score of 0.8161, significantly outperforming supervised learning with the same labeled set.
• For time series prediction on the SML2010 dataset, ACL achieved a mean absolute error (MAE) of 1.933 for temperature and 3.042 for humidity, consistently outperforming supervised learning on the test set despite limited labeled data.
• The method effectively prevents overfitting to small labeled sets, as shown by lower test error compared to supervised baselines even when training error is higher.
• ACL enables learning from incomplete data—such as groups with only temperature data—by using a simulator to provide humidity samples, allowing constraint learning without full-label supervision.
• The framework achieves state-of-the-art performance across all evaluated tasks, especially in low-data regimes, demonstrating the effectiveness of implicit constraint learning.