Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task

11/09/2022
by   Jannik Kossen, et al.
0

We introduce a challenging decision-making task that we call active acquisition for multimodal temporal data (A2MT). In many real-world scenarios, input features are not readily available at test time and must instead be acquired at significant cost. With A2MT, we aim to learn agents that actively select which modalities of an input to acquire, trading off acquisition cost and predictive performance. A2MT extends a previous task called active feature acquisition to temporal decision making about high-dimensional inputs. Further, we propose a method based on the Perceiver IO architecture to address A2MT in practice. Our agents are able to solve a novel synthetic scenario requiring practically relevant cross-modal reasoning skills. On two large-scale, real-world datasets, Kinetics-700 and AudioSet, our agents successfully learn cost-reactive acquisition behavior. However, an ablation reveals they are unable to learn to learn adaptive acquisition strategies, emphasizing the difficulty of the task even for state-of-the-art models. Applications of A2MT may be impactful in domains like medicine, robotics, or finance, where modalities differ in acquisition cost and informativeness.

READ FULL TEXT
research
05/07/2019

Autonomous Open-Ended Learning of Interdependent Tasks

Autonomy is fundamental for artificial agents acting in complex real-wor...
research
06/14/2022

Temporal Multimodal Multivariate Learning

We introduce temporal multimodal multivariate learning, a new family of ...
research
03/31/2020

Peri-Diagnostic Decision Support Through Cost-Efficient Feature Acquisition at Test-Time

Computer-aided diagnosis (CADx) algorithms in medicine provide patient-s...
research
05/03/2023

Efficient Online Decision Tree Learning with Active Feature Acquisition

Constructing decision trees online is a classical machine learning probl...
research
11/03/2018

Dynamic Feature Acquisition Using Denoising Autoencoders

In real-world scenarios, different features have different acquisition c...
research
02/27/2023

Acquisition Conditioned Oracle for Nongreedy Active Feature Acquisition

We develop novel methodology for active feature acquisition (AFA), the s...
research
05/16/2022

Autonomous Open-Ended Learning of Tasks with Non-Stationary Interdependencies

Autonomous open-ended learning is a relevant approach in machine learnin...

Please sign up or login with your details

Forgot password? Click here to reset