Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

06/16/2020
by   Levi McClenny, et al.
0

In many real-world applications of deep learning, estimation of a target may rely on various types of input data modes, such as audio-video, image-text, etc. This task can be further complicated by a lack of sufficient data. Here we propose a Deep Multimodal Transfer-Learned Regressor (DMTL-R) for multimodal learning of image and feature data in a deep regression architecture effective at predicting target parameters in data-poor domains. Our model is capable of fine-tuning a given set of pre-trained CNN weights on a small amount of training image data, while simultaneously conditioning on feature information from a complimentary data mode during network training, yielding more accurate single-target or multi-target regression than can be achieved using the images or the features alone. We present results using phase-field simulation microstructure images with an accompanying set of physical features, using pre-trained weights from various well-known CNN architectures, which demonstrate the efficacy of the proposed multimodal approach.

READ FULL TEXT
research
12/31/2019

Side-Tuning: Network Adaptation via Additive Side Networks

When training a neural network for a desired task, one may prefer to ada...
research
10/11/2022

Transfer Learning with Joint Fine-Tuning for Multimodal Sentiment Analysis

Most existing methods focus on sentiment analysis of textual data. Howev...
research
10/18/2017

Photo-Guided Exploration of Volume Data Features

In this work, we pose the question of whether, by considering qualitativ...
research
05/25/2023

Representation Transfer Learning via Multiple Pre-trained models for Linear Regression

In this paper, we consider the problem of learning a linear regression m...
research
12/09/2014

Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition

We propose a transfer deep learning (TDL) framework that can transfer th...
research
09/20/2023

Kosmos-2.5: A Multimodal Literate Model

We present Kosmos-2.5, a multimodal literate model for machine reading o...
research
02/24/2019

Medical Multimodal Classifiers Under Scarce Data Condition

Data is one of the essential ingredients to power deep learning research...

Please sign up or login with your details

Forgot password? Click here to reset