Log In Sign Up

Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis

by   Jianbo Jiao, et al.

Fetal brain magnetic resonance imaging (MRI) offers exquisite images of the developing brain but is not suitable for second-trimester anomaly screening, for which ultrasound (US) is employed. Although expert sonographers are adept at reading US images, MR images which closely resemble anatomical images are much easier for non-experts to interpret. Thus in this paper we propose to generate MR-like images directly from clinical US images. In medical image analysis such a capability is potentially useful as well, for instance for automatic US-MRI registration and fusion. The proposed model is end-to-end trainable and self-supervised without any external annotations. Specifically, based on an assumption that the US and MRI data share a similar anatomical latent space, we first utilise a network to extract the shared latent features, which are then used for MRI synthesis. Since paired data is unavailable for our study (and rare in practice), pixel-level constraints are infeasible to apply. We instead propose to enforce the distributions to be statistically indistinguishable, by adversarial learning in both the image domain and feature space. To regularise the anatomical structures between US and MRI during synthesis, we further propose an adversarial structural constraint. A new cross-modal attention technique is proposed to utilise non-local spatial information, by encouraging multi-modal knowledge fusion and propagation. We extend the approach to consider the case where 3D auxiliary information (e.g., 3D neighbours and a 3D location index) from volumetric data is also available, and show that this improves image synthesis. The proposed approach is evaluated quantitatively and qualitatively with comparison to real fetal MR images and other approaches to synthesis, demonstrating its feasibility of synthesising realistic MR images.


page 1

page 4

page 6

page 7

page 8

page 9

page 10


Anatomy-Aware Self-supervised Fetal MRI Synthesis from Unpaired Ultrasound Images

Fetal brain magnetic resonance imaging (MRI) offers exquisite images of ...

MRI to FDG-PET: Cross-Modal Synthesis Using 3D U-Net For Multi-Modal Alzheimer's Classification

Recent studies suggest that combined analysis of Magnetic resonance imag...

Dual-cycle Constrained Bijective VAE-GAN For Tagged-to-Cine Magnetic Resonance Image Synthesis

Tagged magnetic resonance imaging (MRI) is a widely used imaging techniq...

Unsupervised Brain Abnormality Detection Using High Fidelity Image Reconstruction Networks

Recent advances in deep learning have facilitated near-expert medical im...

Semi-supervised mp-MRI Data Synthesis with StitchLayer and Auxiliary Distance Maximization

In this paper, we address the problem of synthesizing multi-parameter ma...

Disentangling A Single MR Modality

Disentangling anatomical and contrast information from medical images ha...

Factorisation-based Image Labelling

Segmentation of brain magnetic resonance images (MRI) into anatomical re...