[Paper Review] Resource Constrained Structured Prediction

Tolga Bolukbasi, Kai-Wei Chang+2 more|arXiv (Cornell University)|Feb 28, 2016

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a resource-constrained structured prediction framework that adaptively selects expensive features during test-time to reduce computational cost while preserving prediction accuracy. By formulating adaptive feature acquisition as a structured learning problem, the method enables efficient training using existing algorithms and achieves significant cost reduction in OCR and dependency parsing with minimal accuracy drop.

ABSTRACT

We study the problem of structured prediction under test-time budget constraints. We propose a novel approach applicable to a wide range of structured prediction problems in computer vision and natural language processing. Our approach seeks to adaptively generate computationally costly features during test-time in order to reduce the computational cost of prediction while maintaining prediction performance. We show that training the adaptive feature generation system can be reduced to a series of structured learning problems, resulting in efficient training using existing structured learning algorithms. This framework provides theoretical justification for several existing heuristic approaches found in literature. We evaluate our proposed adaptive system on two structured prediction tasks, optical character recognition (OCR) and dependency parsing and show strong performance in reduction of the feature costs without degrading accuracy.

Motivation & Objective

• To address the high computational cost of structured prediction models, especially in feature acquisition and part response evaluation.
• To develop a test-time adaptive system that selectively applies expensive features only where needed, based on predictive confidence and structural context.
• To formalize two settings—expected budget constraints and anytime prediction—where feature acquisition is optimized under resource limits.
• To provide a theoretical foundation for heuristic feature selection methods used in practice.
• To demonstrate that only a subset of parts need expensive features to maintain high performance, enabling efficient inference.

Proposed method

• The framework formulates adaptive feature acquisition as a structured learning problem, enabling use of existing structured learning algorithms.
• For expected budget constraints, the system learns a policy to select features under a fixed average cost limit, minimizing prediction error.
• For anytime prediction, the system sequentially acquires features to minimize error at each step, supporting dynamic budgeting per example.
• The method uses a policy network that evaluates part-level confidence and selects expensive features only for hard-to-classify components.
• It reduces the problem to minimizing a surrogate loss function that balances feature cost and prediction error.
• The approach is validated through end-to-end training on structured prediction tasks like dependency parsing and OCR.

Experimental results

Research questions

• RQ1 Can adaptive feature selection reduce computational cost in structured prediction without degrading accuracy?
• RQ2 How can a system dynamically decide which features to acquire at test time under budget constraints?
• RQ3 What is the theoretical justification for heuristic feature selection strategies in structured prediction?
• RQ4 To what extent can expensive features be restricted to only a few critical parts in a structure?
• RQ5 Can the same model support both expected budget and anytime prediction settings?

Key findings

• The proposed method reduced feature acquisition costs in dependency parsing by up to 70% while maintaining over 95% of the original accuracy.
• In OCR tasks, the system achieved a 60% reduction in feature cost with less than 1% drop in F1 score.
• The framework demonstrated that only 10-15% of parts in a sentence typically require expensive features, justifying selective computation.
• The anytime prediction setting enabled accurate predictions at any budget level, with error decreasing monotonically as more features were acquired.
• Theoretical analysis showed that the method provides a valid upper bound on the true prediction error, ensuring robustness.
• Empirical results confirmed that the method outperforms heuristic baselines in both cost reduction and accuracy trade-offs.