Self-driving Multimodal Studies at User Facilities

01/22/2023

∙

Multimodal characterization is commonly required for understanding materials. User facilities possess the infrastructure to perform these measurements, albeit in serial over days to months. In this paper, we describe a unified multimodal measurement of a single sample library at distant instruments, driven by a concert of distributed agents that use analysis from each modality to inform the direction of the other in real time. Powered by the Bluesky project at the National Synchrotron Light Source II, this experiment is a world's first for beamline science, and provides a blueprint for future approaches to multimodal and multifidelity experiments at user facilities.

READ FULL TEXT

Self-driving Multimodal Studies at User Facilities

Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis

MAEA: Multimodal Attribution for Embodied AI

How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents

Machine learning enabling high-throughput and remote operations at large-scale user facilities

Multimodal Neuroimaging Data Integration and Pathway Analysis

Ducho: A Unified Framework for the Extraction of Multimodal Features in Recommendation

Multimodal Skip-gram Using Convolutional Pseudowords

Self-driving Multimodal Studies at User Facilities

Related Research

Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis

MAEA: Multimodal Attribution for Embodied AI

How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents

Machine learning enabling high-throughput and remote operations at large-scale user facilities

Multimodal Neuroimaging Data Integration and Pathway Analysis

Ducho: A Unified Framework for the Extraction of Multimodal Features in Recommendation

Multimodal Skip-gram Using Convolutional Pseudowords