[Paper Review] Unsupervised Submodular Rank Aggregation on Score-based Permutations

Jun Qi, Liu, Xu+2 more|arXiv (Cornell University)|Jul 4, 2017

Rough Sets and Fuzzy Logic16 references3 citations

TL;DR

This paper proposes an unsupervised submodular rank aggregation framework using Lovász Bregman (LB) divergence for score-based permutations, leveraging convex and nested structured optimization to combine rankings from multiple sources. The method achieves state-of-the-art performance in automatic speech recognition, reducing phone error rate to 18.1%—a 1.2% improvement over the baseline Adaboost method—by effectively aggregating distributed neural network outputs without ground-truth labels.

ABSTRACT

Unsupervised rank aggregation on score-based permutations, which is widely used in many applications, has not been deeply explored yet. This work studies the use of submodular optimization for rank aggregation on score-based permutations in an unsupervised way. Specifically, we propose an unsupervised approach based on the Lovasz Bregman divergence for setting up linear structured convex and nested structured concave objective functions. In addition, stochastic optimization methods are applied in the training process and efficient algorithms for inference can be guaranteed. The experimental results from Information Retrieval, Combining Distributed Neural Networks, Influencers in Social Networks, and Distributed Automatic Speech Recognition tasks demonstrate the effectiveness of the proposed methods.

Motivation & Objective

• Address the lack of unsupervised rank aggregation methods tailored for score-based permutations, where rankings are represented by numeric scores rather than relative orders.
• Bridge the gap between submodular optimization and unsupervised rank aggregation by formulating the Lovász Bregman divergence as a convex objective function for score-based permutations.
• Develop efficient inference and stochastic optimization algorithms for large-scale applications such as distributed deep learning and information retrieval.
• Demonstrate the effectiveness of nested structured formulations in improving aggregation performance over linear and existing baselines.
• Enable scalable, label-free aggregation of outputs from multiple models, especially in settings where relevance scores are expensive or impossible to obtain.

Proposed method

• Formulate the Lovász Bregman (LB) divergence as a utility function to measure the divergence between a score-based permutation and an order-based consensus, using submodular functions and their Lovász extensions.
• Construct a linear structured convex optimization objective based on the LB divergence to minimize the divergence between aggregated scores and consensus rankings.
• Introduce a nested structured concave optimization framework to model hierarchical dependencies in the aggregation process, improving representational capacity.
• Apply stochastic optimization techniques to scale the training process, enabling efficient learning from large-scale, distributed model outputs.
• Use the Minorization-Maximization (MMAX) algorithm to iteratively approximate solutions to the non-convex nested formulation while maintaining convergence guarantees.
• Define the objective function using the inner product between the input score vector and the difference of submodular function gradients: $ d_{ ilde{f}}( extbf{x}||oldsymbol{ heta}) = extbf{x}^ op (h_{oldsymbol{ heta}}^f - h_{oldsymbol{ heta}}^f) $, where $ h $ represents the submodular function increment vector.

Experimental results

Research questions

• RQ1 Can the Lovász Bregman divergence be effectively adapted to unsupervised rank aggregation on score-based permutations, where inputs are numeric scores rather than relative orders?
• RQ2 How does the proposed linear-structured convex formulation compare to existing unsupervised aggregation methods like averaging or ULARA in terms of performance and scalability?
• RQ3 Does the nested structured concave formulation provide significant performance gains over linear and baseline methods in real-world applications such as automatic speech recognition?
• RQ4 To what extent can the proposed method scale to large-scale datasets with thousands of permutations and high-dimensional outputs?
• RQ5 What is the impact of model architecture depth on the performance of the nested structured LB divergence formulation, and can deeper structures be effectively optimized?

Key findings

• The nested structured LB divergence formulation achieved a phone error rate (PER) of 18.1% on the TIMIT dataset, outperforming the Adaboost baseline (18.3%) and all other methods tested.
• The method reduced PER by 1.2 percentage points compared to the Adaboost baseline, demonstrating significant gains in distributed automatic speech recognition.
• The linear-structured formulation achieved a PER of 18.7%, which was better than averaging (20.1%) and ULARA (19.5%), but worse than the nested formulation.
• The proposed method achieved state-of-the-art performance in combining outputs from 8 distributed DNNs in speech recognition, validating its effectiveness in real-world applications.
• The method demonstrated scalability due to low computational complexity and efficient stochastic optimization, enabling training on large-scale datasets with minimal supervision.
• The gains from the nested structure were attributed to better modeling of hierarchical dependencies in the aggregation process, though marginal improvements were observed due to forced-alignment bias in the data.