[Paper Review] Learning with Hierarchical Complement Objective

Haoyun Chen, Li-Huang Tsai+5 more|arXiv (Cornell University)|Nov 17, 2019

Domain Adaptation and Few-Shot Learning24 references4 citations

TL;DR

This paper proposes Hierarchical Complement Objective Training (HCOT), a novel training objective that leverages label hierarchy in vision tasks by jointly maximizing ground-truth class probability and neutralizing incorrect classes in a hierarchical manner. HCOT outperforms cross-entropy and COT on CIFAR-100, ImageNet-2012, and PASCAL-Context, achieving state-of-the-art performance in both image classification and semantic segmentation.

ABSTRACT

Label hierarchies widely exist in many vision-related problems, ranging from explicit label hierarchies existed in image classification to latent label hierarchies existed in semantic segmentation. Nevertheless, state-of-the-art methods often deploy cross-entropy loss that implicitly assumes class labels to be exclusive and thus independence from each other. Motivated by the fact that classes from the same parental category usually share certain similarity, we design a new training diagram called Hierarchical Complement Objective Training (HCOT) that leverages the information from label hierarchy. HCOT maximizes the probability of the ground truth class, and at the same time, neutralizes the probabilities of rest of the classes in a hierarchical fashion, making the model take advantage of the label hierarchy explicitly. The proposed HCOT is evaluated on both image classification and semantic segmentation tasks. Experimental results confirm that HCOT outperforms state-of-the-art models in CIFAR-100, ImageNet-2012, and PASCAL-Context. The study further demonstrates that HCOT can be applied on tasks with latent label hierarchies, which is a common characteristic in many machine learning tasks.

Motivation & Objective

• To address the limitation of cross-entropy loss in modeling hierarchical label structures, which assumes class independence.
• To improve model generalization by explicitly incorporating label hierarchy during training, especially in tasks with explicit or latent hierarchies.
• To design a training objective that penalizes incorrect classes based on their hierarchical distance from the ground-truth class.
• To demonstrate the effectiveness of HCOT across diverse vision tasks, including image classification and semantic segmentation.
• To show that HCOT can be applied to tasks with latent label hierarchies, such as semantic segmentation, without requiring explicit hierarchy annotations.

Proposed method

• HCOT introduces a new complement objective called Hierarchical Complement Entropy, which applies complement learning at both fine-level and coarse-level categories.
• The method maximizes the predicted probability of the ground-truth class while simultaneously neutralizing probabilities of incorrect classes in a hierarchical fashion.
• Classes sharing the same coarse-level category as the ground truth are penalized less than those on different branches in the label hierarchy.
• The training objective combines the primary cross-entropy loss with the hierarchical complement entropy loss, enabling joint optimization with minimal computational overhead.
• The approach is applied end-to-end in both classification and segmentation models, with no architectural changes required.
• HCOT is evaluated using standard metrics like top-1 accuracy, mIoU, and pixel accuracy on CIFAR-100, ImageNet-2012, and PASCAL-Context.

Experimental results

Research questions

• RQ1 Can a training objective that explicitly models label hierarchy improve performance in image classification with hierarchical labels?
• RQ2 How does HCOT compare to cross-entropy and COT in terms of accuracy and robustness on standard benchmarks?
• RQ3 Can HCOT effectively leverage latent label hierarchies in tasks like semantic segmentation where no explicit hierarchy is provided?
• RQ4 Does hierarchical penalization—differing by distance from the ground-truth class in the hierarchy—lead to better generalization and more structured predictions?
• RQ5 Is HCOT compatible with state-of-the-art models and training pipelines without requiring architectural modifications?

Key findings

• HCOT achieved a top-1 accuracy of 88.4% on CIFAR-100 using PreAct ResNet-18, outperforming cross-entropy (87.1%) and COT (87.8%).
• On ImageNet-2012, HCOT improved top-1 accuracy by 0.6% over cross-entropy and 0.4% over COT using ResNet-50.
• On PASCAL-Context, HCOT achieved a mIoU of 49.86% with EncNet and 51.35% with EncNet+JPU, surpassing cross-entropy and COT in both settings.
• Visualizations showed that HCOT-generated segments were less fragmented and contained fewer noise artifacts compared to cross-entropy and COT.
• The performance gain from HCOT was consistent across multiple backbone architectures and training strategies, indicating broad applicability.
• HCOT demonstrated that modeling hierarchical relationships in label space improves prediction confidence and alignment with true data distribution.