Log In Sign Up

Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy

by   Alvaro Gomariz, et al.

Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers, and therefore applicable to a very restricted number of experimental settings. We herein propose Marker Sampling and Excite, a neural network approach with a modality sampling strategy and a novel attention module that together enable (i) flexible training with heterogeneous datasets with combinations of markers and (ii) successful utility of learned models on arbitrary subsets of markers prospectively. We show that our single neural network solution performs comparably to an upper bound scenario where an ensemble of many networks is naïvely trained for each possible marker combination separately. In addition, we demonstrate the feasibility of our framework in high-throughput biological analysis by revising a recent quantitative characterization of bone marrow vasculature in 3D confocal microscopy datasets. Not only can our work substantially ameliorate the use of deep learning in fluorescence microscopy analysis, but it can also be utilized in other fields with incomplete data acquisitions and missing modalities.


page 4

page 5

page 7

page 9

page 18

page 19


Free annotated data for deep learning in microscopy? A hitchhiker's guide

In microscopy, the time burden and cost of acquiring and annotating larg...

DistNet: Deep Tracking by displacement regression: application to bacteria growing in the Mother Machine

The mother machine is a popular microfluidic device that allows long-ter...

Deep learning-based color holographic microscopy

We report a framework based on a generative adversarial network (GAN) th...

Methodologies for Successful Segmentation of HRTEM Images via Neural Network

High throughput analysis of samples has been a topic increasingly discus...

Comparison of semi-supervised learning methods for High Content Screening quality control

Progress in automated microscopy and quantitative image analysis has pro...

Towards a Digital Diatom: image processing and deep learning analysis of Bacillaria paradoxa dynamic morphology

Recent years have witnessed a convergence of data and methods that allow...