[论文解读] Order-Free RNN with Visual Attention for Multi-Label Classification

Shang‐Fu Chen, Yi‐Chen Chen+2 more|arXiv (Cornell University)|Jul 18, 2017

Text and Document Classification Technologies被引用 53

一句话总结

该论文提出一种用于多标签图像分类的无序RNN，能联合学习视觉注意力与标签依赖，采用带置信度排序的LSTM，在训练和测试阶段避免预先定义的标签顺序，从而提高鲁棒性。

ABSTRACT

In this paper, we propose the joint learning attention and recurrent neural network (RNN) models for multi-label classification. While approaches based on the use of either model exist (e.g., for the task of image captioning), training such existing network architectures typically require pre-defined label sequences. For multi-label classification, it would be desirable to have a robust inference process, so that the prediction error would not propagate and thus affect the performance. Our proposed model uniquely integrates attention and Long Short Term Memory (LSTM) models, which not only addresses the above problem but also allows one to identify visual objects of interests with varying sizes without the prior knowledge of particular label ordering. More importantly, label co-occurrence information can be jointly exploited by our LSTM model. Finally, by advancing the technique of beam search, prediction of multiple labels can be efficiently achieved by our proposed network model.

研究动机与目标

• Motivate and address limitations of existing CNN-RNN approaches that require predefined label orders and struggle with small or occluded objects.
• Propose a unified network that jointly learns visual attention and an LSTM-based predictor to automatically discover label order and dependencies.
• Ensure training and testing are consistent and robust against error propagation in sequential predictions.

提出的方法

• Three-component network: feature mapping (CNN-based), visual attention, and an LSTM-based predictor (confidence-ranked LSTM).
• Attention is conditioned on the previous LSTM state and produces a context vector guiding region-focused prediction.
• Labels are predicted sequentially without a fixed order by maintaining a candidate label pool and using beam search during testing for robustness.
• Loss is computed by directly comparing soft predictions against ground-truth multi-label vectors, enabling order-free training.
• Beam search during inference maintains top-K prediction paths to mitigate error propagation in sequential predictions.

实验结果

研究问题

• RQ1 Can an RNN-based multi-label classifier learn label dependencies without a predefined order by jointly training attention and LSTM components?
• RQ2 Does incorporating a visual attention mechanism improve detection of small or occluded labels in multi-label images?
• RQ3 Can order-free training/testing reduce error propagation and improve robustness compared to predefined-order CNN-RNN approaches?
• RQ4 How does the proposed method perform on standard multi-label benchmarks (NUS-WIDE, MS-COCO) relative to state-of-the-art methods?

主要发现

• The proposed model achieves favorable results over baselines and state-of-the-art methods on NUS-WIDE.
• With attention, the model learns to focus on regions corresponding to labels, improving recognition of small objects.
• On MS-COCO, the full model outperforms baselines, showing improvements in F1 and other metrics.
• Removing attention or using fixed label orders reduces performance, highlighting the benefits of order-free learning and visual attention.
• Beam search during testing provides robustness to intermediate prediction errors.
• The method demonstrates resilience to noisy training labels and still learns meaningful label ordering and attention patterns.