Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video
share
The temporal answering grounding in the video (TAGV) is a new task naturally deriving from temporal sentence grounding in the video (TSGV). Given an untrimmed video and a text question, this task aims at locating the matching span from the video that can semantically answer the question. Existing methods tend to formulate the TAGV task with a visual span-based question answering (QA) approach by matching the visual frame span queried by the text question. However, due to the weak correlations and huge gaps in semantics in features between the textual question and visual answer, existing methods adopting visual span predictor fail to perform well in the TAGV task. In this work, we propose a visual-prompt text span localizing (VPTSL) method, which enhances the text span localization in the pre-trained language model (PLM) with the visual highlight features. Specifically, the context query attention is utilized to perform cross-modal modeling between the textual and visual features. Then, the highlight features are obtained through the highlight module with a linear layer to provide the visual prompt. To alleviate the differences in semantics and correlations between textual and visual features, we design the text span predictor by encoding the question, the subtitles, and the visual prompt in the PLM. As a result, the TAGV task is formulated to predict the span of subtitles matching the answering frame timeline. Extensive experiments on the medical instructional dataset, namely MedVidQA, show that the proposed VPTSL outperforms other state-of-the-art (SOTA) methods by 28.36 in mIOU score with a large margin, which demonstrates the effectiveness of visual prompt and the text span predictor.
READ FULL TEXT