[Paper Review] Link Prediction Under Imperfect Detection: Collaborative Filtering for Ecological Networks

Xiao Fu, Eugene Seo+2 more|arXiv (Cornell University)|Oct 8, 2019

Complex Network Analysis Techniques33 references4 citations

TL;DR

This paper proposes a novel collaborative filtering framework for ecological bipartite networks that accounts for systematic imperfect detection in field observations. By integrating a Poisson N-mixture model with probabilistic nonnegative matrix factorization in latent space, the method models unobserved true interaction counts (N) while correcting for undercounting in observed data (Y), significantly outperforming existing link prediction methods on real pollination and host-parasite networks.

ABSTRACT

Matrix completion based collaborative filtering is considered scalable and effective for online service link prediction (e.g., movie recommendation) but does not meet the challenges of link prediction in ecological networks. A unique challenge of ecological networks is that the observed data are subject to systematic imperfect detection, due to the difficulty of accurate field sampling. In this work, we propose a new framework customized for ecological bipartite network link prediction. Our approach starts with incorporating the Poisson N-mixture model, a widely used framework in statistical ecology for modeling imperfect detection of a single species in field sampling. Despite its extensive use for single species analysis, this model has never been considered for link prediction between different species, perhaps because of the complex nature of both link prediction and N-mixture model inference. By judiciously combining the Poisson N-mixture model with a probabilistic nonnegative matrix factorization (NMF) model in latent space, we propose an intuitive statistical model for the problem of interest. We also offer a scalable and convergence-guaranteed optimization algorithm to handle the associated maximum likelihood identification problem. Experimental results on synthetic data and two real-world ecological networks data are employed to validate our proposed approach.

Motivation & Objective

• To address the challenge of systematic imperfect detection in ecological network data, where observed interaction counts are biased low due to field sampling limitations.
• To develop a statistically principled model that distinguishes between true zero interactions and missed detections in species interaction networks.
• To improve link prediction accuracy in ecological bipartite networks by modeling latent true interaction counts (N) rather than relying on observed, undercounted data (Y).
• To create a scalable and convergent optimization algorithm for maximum likelihood estimation under the proposed two-layer generative model.
• To validate the method on synthetic data and two real-world ecological networks, demonstrating superior performance over existing collaborative filtering and link prediction baselines.

Proposed method

• The method uses a two-layer generative model: first, a Poisson-distributed latent interaction matrix N represents true species interactions; second, observed counts Y are modeled as a Binomially distributed selection of N, accounting for imperfect detection.
• The model integrates probabilistic nonnegative matrix factorization (NMF) to capture low-rank structure in the latent interaction matrix N, enabling collaborative filtering for link prediction.
• A joint likelihood function is derived by marginalizing over the unobserved true counts N, allowing inference on observed data Y while accounting for detection uncertainty.
• An alternating optimization algorithm is proposed to maximize the likelihood, with convergence guarantees, enabling scalable inference on large ecological networks.
• The model incorporates species trait features to inform latent interaction patterns, improving predictive performance beyond standard NMF.
• The framework estimates not only predicted interactions but also latent true counts (N̂), offering ecologists insights into the true structure of ecological networks.

Experimental results

Research questions

• RQ1 Can a collaborative filtering approach be adapted to ecological networks where observed interaction counts are systematically undercounted due to imperfect detection?
• RQ2 How does incorporating a latent true interaction matrix (N) via a Poisson N-mixture model improve link prediction accuracy compared to standard matrix completion methods?
• RQ3 To what extent does the proposed model outperform existing collaborative filtering and link prediction baselines on real ecological networks with imperfect detection?
• RQ4 Can the estimated latent interaction counts (N̂) provide ecologically meaningful insights into the true structure of pollination and host-parasite networks?
• RQ5 How do trait-informed features influence the performance of the proposed model in predicting species interactions?

Key findings

• The proposed method outperformed all eight competing methods on all three metrics (rRMSE, AUROC, AUPRC) in predicting held-out interactions in the host-parasite network, with rRMSE of 3.538 and AUROC of 0.759.
• The model achieved the best performance on the pollination network, significantly outperforming standard matrix completion-based collaborative filtering (MC-CF) and neural collaborative filtering (NCF).
• The model estimated that 80.1% of possible links had a latent count greater than zero, suggesting that only 42% of actual interactions were observed—indicating severe undercounting in the data.
• Expert validation of top-10 predicted interactions (ranked by estimated latent count) showed strong agreement that these were more likely to interact than the bottom-10, supporting the model’s biological plausibility.
• The method outperformed even advanced models like NCF and BRT, demonstrating that modeling imperfect detection is critical for ecological link prediction.
• The model’s ability to estimate latent true counts (N̂) provides ecologists with a reference for designing future field studies, highlighting the extent of missing interactions.