[Paper Review] Incorporating Multiple Cluster Centers for Multi-Label Learning

Senlin Shu, Fengmao Lv+4 more|arXiv (Cornell University)|Apr 17, 2020

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes a novel data augmentation method for multi-label learning by generating virtual examples from cluster centers of real data, which encode local label correlations and importances. A new regularization term enforces similarity between real and virtual examples, improving model smoothness and achieving state-of-the-art performance with average gains of 7.1% and 6.0% on standard metrics.

ABSTRACT

Multi-label learning deals with the problem that each instance is associated with multiple labels simultaneously. Most of the existing approaches aim to improve the performance of multi-label learning by exploiting label correlations. Although the data augmentation technique is widely used in many machine learning tasks, it is still unclear whether data augmentation is helpful to multi-label learning. In this article, we propose to leverage the data augmentation technique to improve the performance of multi-label learning. Specifically, we first propose a novel data augmentation approach that performs clustering on the real examples and treats the cluster centers as virtual examples, and these virtual examples naturally embody the local label correlations and label importances. Then, motivated by the cluster assumption that examples in the same cluster should have the same label, we propose a novel regularization term to bridge the gap between the real examples and virtual examples, which can promote the local smoothness of the learning function. Extensive experimental results on a number of real-world multi-label datasets clearly demonstrate that our proposed approach outperforms the state-of-the-art counterparts.

Motivation & Objective

• To investigate whether data augmentation can improve multi-label learning performance, despite its unclear utility in this context.
• To leverage clustering to generate virtual examples that capture local label correlations and label importances.
• To propose a regularization term based on the cluster assumption to bridge real and virtual examples, enhancing local smoothness of the learning function.
• To demonstrate the effectiveness of the proposed method through extensive experiments on real-world multi-label datasets.

Proposed method

• Cluster the real training examples to identify local data structures, using cluster centers as virtual examples.
• Treat each cluster center as a new training instance with a continuous label vector obtained by averaging the labels of all instances in the cluster.
• Propose a novel regularization term that encourages the model output of a real instance to be similar to that of its corresponding cluster center, promoting local smoothness.
• Integrate the regularization term into the learning objective, balancing between fitting real data and aligning predictions with virtual examples.
• Control the influence of virtual examples and regularization via hyperparameters α and γ, and balance model complexity using β.
• Use the number of clusters c as a tunable parameter to control the granularity of virtual example generation.

(a) One-error

Experimental results

Research questions

• RQ1 Can data augmentation techniques improve performance in multi-label learning, despite limited prior exploration in this domain?
• RQ2 How can cluster centers be used to generate meaningful virtual examples that reflect local label correlations and label importances?
• RQ3 Does enforcing similarity between real instances and their cluster centers improve the smoothness and generalization of the learning function?
• RQ4 What is the optimal balance between real data fitting, virtual example utilization, and model complexity in the proposed framework?

Key findings

• The proposed method achieves an average improvement of 7.1% over state-of-the-art methods on the scene dataset using the one-error metric.
• On the large-scale computer dataset, the method improves performance by 6.0% on average using the one-error metric.
• The model is relatively insensitive to the hyperparameter α, which controls the influence of virtual examples, indicating robustness.
• The regularization hyperparameter γ achieves optimal performance at intermediate values, confirming the effectiveness of bridging real and virtual examples.
• Model complexity control via β shows a U-shaped performance curve, with best results at intermediate values like β=1, indicating a need to balance overfitting and underfitting.
• The method is relatively insensitive to the number of clusters c, showing stable performance across a wide range of cluster counts, which simplifies hyperparameter tuning.

(b) Hamming loss