Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video

03/13/2022
by   Bin Li, et al.
0

The temporal answering grounding in the video (TAGV) is a new task naturally deriving from temporal sentence grounding in the video (TSGV). Given an untrimmed video and a text question, this task aims at locating the matching span from the video that can semantically answer the question. Existing methods tend to formulate the TAGV task with a visual span-based question answering (QA) approach by matching the visual frame span queried by the text question. However, due to the weak correlations and huge gaps in semantics in features between the textual question and visual answer, existing methods adopting visual span predictor fail to perform well in the TAGV task. In this work, we propose a visual-prompt text span localizing (VPTSL) method, which enhances the text span localization in the pre-trained language model (PLM) with the visual highlight features. Specifically, the context query attention is utilized to perform cross-modal modeling between the textual and visual features. Then, the highlight features are obtained through the highlight module with a linear layer to provide the visual prompt. To alleviate the differences in semantics and correlations between textual and visual features, we design the text span predictor by encoding the question, the subtitles, and the visual prompt in the PLM. As a result, the TAGV task is formulated to predict the span of subtitles matching the answering frame timeline. Extensive experiments on the medical instructional dataset, namely MedVidQA, show that the proposed VPTSL outperforms other state-of-the-art (SOTA) methods by 28.36 in mIOU score with a large margin, which demonstrates the effectiveness of visual prompt and the text span predictor.

READ FULL TEXT

page 1

page 7

research
10/26/2022

Visual Answer Localization with Cross-modal Mutual Knowledge Transfer

The goal of visual answering localization (VAL) in the video is to obtai...
research
10/11/2022

Learning to Locate Visual Answer in Video Corpus Using Question

We introduce a new task, named video corpus visual answer localization (...
research
10/12/2021

Relation-aware Video Reading Comprehension for Temporal Language Grounding

Temporal language grounding in videos aims to localize the temporal span...
research
09/08/2022

Frame-Subtitle Self-Supervision for Multi-Modal Video Question Answering

Multi-modal video question answering aims to predict correct answer and ...
research
09/04/2023

Can I Trust Your Answer? Visually Grounded Video Question Answering

We study visually grounded VideoQA in response to the emerging trends of...
research
07/11/2019

Two-stream Spatiotemporal Feature for Video QA Task

Understanding the content of videos is one of the core techniques for de...
research
07/25/2018

Linear Span Network for Object Skeleton Detection

Robust object skeleton detection requires to explore rich representative...

Please sign up or login with your details

Forgot password? Click here to reset