[Paper Review] Quantitative evaluation of methods to analyze motion changes in single-particle experiments

Gorka Muñoz, Harshith Bachimanchi+34 more|arXiv (Cornell University)|Nov 29, 2023

Advanced Fluorescence Microscopy Techniques10 citations

TL;DR

This paper presents the two-track AnDi Challenge 2 to benchmark methods that detect and characterize changes in diffusion behavior in single-particle trajectories, using simulated FBM-based datasets and a detailed scoring framework.

ABSTRACT

The analysis of live-cell single-molecule imaging experiments can reveal valuable information about the heterogeneity of transport processes and interactions between cell components. These characteristics are seen as motion changes in the particle trajectories. Despite the existence of multiple approaches to carry out this type of analysis, no objective assessment of these methods has been performed so far. Here, we report the results of a competition to characterize and rank the performance of these methods when analyzing the dynamic behavior of single molecules. To run this competition, we implemented a software library that simulates realistic data corresponding to widespread diffusion and interaction models, both in the form of trajectories and videos obtained in typical experimental conditions. The competition constitutes the first assessment of these methods, providing insights into the current limitations of the field, fostering the development of new approaches, and guiding researchers to identify optimal tools for analyzing their experiments.

Motivation & Objective

• Motivate objective evaluation of methods that detect motion changes in single-particle trajectories in biological contexts.
• Provide ground-truth datasets with multiple diffusion models and interaction scenarios for method benchmarking.
• Foster cross-disciplinary development by organizing an open competition with clear scoring metrics.

Proposed method

• Develop an open software library (andi-datasets) to simulate realistic 2D diffusion with piecewise-constant parameters across several interaction models.
• Generate datasets based on fractional Brownian motion to produce SSM, MSM, DIM, TCM, and QTM motion scenarios.
• Organize the challenge into Track 1 (raw videos) and Track 2 (trajectories) with ensemble and single-trajectory prediction tasks.
• Define ground-truth ground models, state counts, and distributions of generalized diffusion coefficients K and anomalous exponents α for evaluation.
• Adopt objective metrics including Wasserstein distance for ensemble properties, Jaccard similarity and RMSE for CP detection, MSLE for K, MAE for α, and F1-score for diffusion-type classification.

Figure 1: Rationale for the challenge organization. a , The interactions of biomolecules in complex environments, such as the cell membrane, regulate physiological processes in living systems. These interactions produce changes in molecular motion that can be used as a proxy to measure interaction p

Experimental results

Research questions

• RQ1 Can current methods accurately identify the underlying diffusion model and number of states in ensemble data?
• RQ2 How well do methods estimate the distributions of diffusion parameters (K) and anomalous exponents (α) across states?
• RQ3 Can single-trajectory changepoint detection reliably segment trajectories and recover segment parameters (K, α) and diffusion type?
• RQ4 What is the comparative performance of analyzing raw videos versus extracted trajectories in detecting diffusion changes?

Key findings

• The competition design enables objective benchmarking of heterogeneous diffusion analysis methods.
• A dedicated Python package (andi-datasets) generates ground-truth datasets under several diffusion-interaction models for training and evaluation.
• Ground-truth ground truth is coupled with a comprehensive scoring framework using Wasserstein distance, Jaccard similarity, RMSE, MSLE, MAE, and F1-score across tasks.
• Datasets are organized into experiments and fields of view, with Track 1 using 200-frame videos and Track 2 using trajectory tables, both under fixed 128x128 pixel areas.
• Predictions can span ensemble-level parameters and single-trajectory CPs with flexibility for partial submissions, ranked via mean reciprocal rank.

Figure 2: Physical models of interaction and structure of the simulated datasets. a , Examples of 2-dimensional trajectories undergoing interactions inducing changes in their motion. From left to right: single-state model ( SSM ) without changes of diffusion; multi-state model ( MSM ) with time-depe