[Paper Review] MarmoNet: a pipeline for automated projection mapping of the common marmoset brain from whole-brain serial two-photon tomography

Henrik Skibbe, Akiya Watakabe+11 more|arXiv (Cornell University)|Aug 2, 2019

Cell Image Analysis Techniques12 references14 citations

TL;DR

MarmoNet is a deep learning pipeline using U-Net-based convolutional neural networks and image registration to automate the segmentation and projection mapping of axonal tracer signals in whole-brain serial two-photon tomography data from common marmosets. It enables high-accuracy, fully automated detection of sparse and dense axonal projections across complex, noisy brain tissue, significantly improving connectivity mapping efficiency and reproducibility in primate brain research.

ABSTRACT

Understanding the connectivity in the brain is an important prerequisite for understanding how the brain processes information. In the Brain/MINDS project, a connectivity study on marmoset brains uses two-photon microscopy fluorescence images of axonal projections to collect the neuron connectivity from defined brain regions at the mesoscopic scale. The processing of the images requires the detection and segmentation of the axonal tracer signal. The objective is to detect as much tracer signal as possible while not misclassifying other background structures as the signal. This can be challenging because of imaging noise, a cluttered image background, distortions or varying image contrast cause problems. We are developing MarmoNet, a pipeline that processes and analyzes tracer image data of the common marmoset brain. The pipeline incorporates state-of-the-art machine learning techniques based on artificial convolutional neural networks (CNN) and image registration techniques to extract and map all relevant information in a robust manner. The pipeline processes new images in a fully automated way. This report introduces the current state of the tracer signal analysis part of the pipeline.

Motivation & Objective

• To address the challenge of manual segmentation of axonal tracer signals in high-resolution, noisy two-photon microscopy images of marmoset brains.
• To develop a fully automated pipeline that robustly detects axonal projections across variable contrast, intensity, and structural complexity in brain tissue.
• To improve the scalability and reproducibility of large-scale neural connectivity mapping in the marmoset brain for the Brain/MINDS project.
• To reduce human error and time consumption in processing vast volumes of serial two-photon tomography data from anterograde tracer experiments.

Proposed method

• The pipeline uses a U-Net architecture with encoder-decoder structure for end-to-end segmentation of axonal tracer signals from two-photon fluorescence images.
• Input images are processed as 572×572×n pixel patches, with n=1 for cell body detection and n=2 for tracer segmentation (R and G channels).
• The network employs repeated 3×3 convolutions with ReLU activation and 2×2 max-pooling in the encoder path, followed by transposed convolutions (up-convolutions) and skip connections in the decoder path.
• Skip connections from encoder to decoder layers preserve spatial resolution and improve localization accuracy of segmented axonal structures.
• Data augmentation techniques, including rotation, gamma correction, non-linear spatial deformation, and global scaling, are applied to increase training set diversity and reduce overfitting.
• A weighted logistic regression loss function is used to optimize the network, with batch normalization and dropout layers to stabilize training and improve generalization.

Experimental results

Research questions

• RQ1 Can a deep learning-based pipeline achieve robust and automated segmentation of axonal tracer signals in complex, noisy marmoset brain tissue across diverse imaging conditions?
• RQ2 How does the integration of skip connections and multi-scale feature learning in a U-Net architecture improve the precision of axonal projection mapping compared to traditional heuristic methods?
• RQ3 To what extent does data augmentation enhance model generalization when training on limited biomedical image data with high variability in signal morphology?
• RQ4 Can the pipeline process large-scale whole-brain serial two-photon tomography images efficiently using a sliding window inference strategy?

Key findings

• The U-Net-based MarmoNet pipeline achieves high-accuracy segmentation of axonal tracer signals across diverse morphological patterns, including sparse, elongated axons and dense, blob-like bundles.
• The use of skip connections and multi-scale feature learning enables precise spatial localization of axonal projections, even in regions with low signal-to-noise ratio.
• Data augmentation significantly improved model generalization, reducing overfitting despite limited training data and high variability in image appearance.
• The pipeline enables fully automated processing of large whole-brain image stacks using a sliding window inference approach, making large-scale connectomics feasible.
• The method outperforms traditional heuristic approaches that rely on edge enhancement or morphological filtering, which struggle with variable contrast and complex background structures.