[Paper Review] Bayesian Network Based Label Correlation Analysis For Multi-label Classifier Chain

Ran Wang, Suhe Ye+2 more|arXiv (Cornell University)|Aug 6, 2019

Text and Document Classification Technologies45 references4 citations

TL;DR

This paper proposes BNCC, a Bayesian Network-based method to model label correlations in multi-label learning for improved Classifier Chain (CC) performance. By using conditional entropy to quantify label dependencies, constructing a Bayesian network with a novel scoring function, and deriving label order via topological sorting, BNCC achieves competitive performance with lower time complexity than ensemble methods like ECC.

ABSTRACT

Classifier chain (CC) is a multi-label learning approach that constructs a sequence of binary classifiers according to a label order. Each classifier in the sequence is responsible for predicting the relevance of one label. When training the classifier for a label, proceeding labels will be taken as extended features. If the extended features are highly correlated to the label, the performance will be improved, otherwise, the performance will not be influenced or even degraded. How to discover label correlation and determine the label order is critical for CC approach. This paper employs Bayesian network (BN) to model the label correlations and proposes a new BN-based CC method (BNCC). First, conditional entropy is used to describe the dependency relations among labels. Then, a BN is built up by taking nodes as labels and weights of edges as their dependency relations. A new scoring function is proposed to evaluate a BN structure, and a heuristic algorithm is introduced to optimize the BN. At last, by applying topological sorting on the nodes of the optimized BN, the label order for constructing CC model is derived. Experimental comparisons demonstrate the feasibility and effectiveness of the proposed method.

Motivation & Objective

• To address the critical challenge of determining optimal label order in Classifier Chain (CC) methods, which significantly impacts performance.
• To model complex label correlations—both positive and negative—using a probabilistic graphical model that captures conditional dependencies.
• To develop a scalable and efficient method that improves CC performance without increasing training or inference time complexity.
• To provide a data-driven, structure-optimized label ordering strategy based on label correlation analysis, replacing random or heuristic label sequences.
• To demonstrate that the proposed method outperforms existing approaches in both accuracy and efficiency on multi-label benchmark datasets.

Proposed method

• Uses conditional entropy to quantify the uncertainty of one label given the presence of others, serving as a measure of dependency between labels.
• Constructs a Bayesian network (BN) where nodes represent labels and edge weights represent the strength of dependency derived from conditional entropy.
• Proposes a novel scoring function to evaluate BN structures, favoring those that better capture label correlation patterns.
• Employs a heuristic algorithm to optimize the BN structure, balancing model complexity and correlation fidelity.
• Applies topological sorting on the optimized BN to derive a deterministic, dependency-aware label order for constructing the Classifier Chain.
• Integrates the resulting label order into the CC framework, where each classifier uses preceding labels as extended features to improve prediction accuracy.

Figure 1: Directed network composed of five labels.

Experimental results

Research questions

• RQ1 Can Bayesian networks effectively model complex, directed label correlations in multi-label learning problems?
• RQ2 Does a data-driven label ordering strategy based on BN-learned dependencies improve Classifier Chain performance compared to random or heuristic orders?
• RQ3 Can the proposed BNCC method achieve better performance than state-of-the-art multi-label learning methods while maintaining low time complexity?
• RQ4 How does the performance of BNCC compare to ensemble-based CC methods like ECC in terms of accuracy and computational efficiency?
• RQ5 Is the proposed scoring function and heuristic algorithm effective in learning a meaningful BN structure that reflects true label dependencies?

Key findings

• BNCC achieves statistically significant improvements in Hamming Loss, Fscore, and Micro-F1 compared to BR, CC, GCC, and CLR on multiple benchmark datasets.
• The method outperforms ECC in terms of both accuracy and efficiency, with a much lower time complexity in both training and testing phases.
• Paired Wilcoxon signed-rank tests show that BNCC is statistically superior to all compared methods in Fscore and Micro-F1, with p-values < 0.05.
• While BNCC is not significantly better than ECC in Macro-F1, it achieves comparable performance with substantially reduced computational cost.
• The proposed scoring function and heuristic algorithm successfully learn a BN structure that reflects real label dependencies, leading to effective label ordering.
• Empirical results confirm that incorporating label correlations via BN-based ordering enhances CC performance without increasing time complexity.

(a) Positive relationship between Rural - Village or Rural-Paddy