Knowledge Graph Extraction from Videos

07/20/2020
by   Louis Mahon, et al.
0

Nearly all existing techniques for automated video annotation (or captioning) describe videos using natural language sentences. However, this has several shortcomings: (i) it is very hard to then further use the generated natural language annotations in automated data processing, (ii) generating natural language annotations requires to solve the hard subtask of generating semantically precise and syntactically correct natural language sentences, which is actually unrelated to the task of video annotation, (iii) it is difficult to quantitatively measure performance, as standard metrics (e.g., accuracy and F1-score) are inapplicable, and (iv) annotations are language-specific. In this paper, we propose the new task of knowledge graph extraction from videos, i.e., producing a description in the form of a knowledge graph of the contents of a given video. Since no datasets exist for this task, we also include a method to automatically generate them, starting from datasets where videos are annotated with natural language. We then describe an initial deep-learning model for knowledge graph extraction from videos, and report results on MSVD* and MSR-VTT*, two datasets obtained from MSVD and MSR-VTT using our method.

READ FULL TEXT

page 2

page 5

page 7

research
02/16/2018

Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing

Machine Learning has been the quintessential solution for many AI proble...
research
11/20/2021

Textbook to triples: Creating knowledge graph in the form of triples from AI TextBook

A knowledge graph is an essential and trending technology with great app...
research
01/28/2020

Bringing Stories Alive: Generating Interactive Fiction Worlds

World building forms the foundation of any task that requires narrative ...
research
04/04/2021

KnowGraph@IITK at SemEval-2021 Task 11: Building KnowledgeGraph for NLP Research

Research in Natural Language Processing is making rapid advances, result...
research
02/20/2021

NUBOT: Embedded Knowledge Graph With RASA Framework for Generating Semantic Intents Responses in Roman Urdu

The understanding of the human language is quantified by identifying int...
research
06/10/2021

SemEval-2021 Task 11: NLPContributionGraph – Structuring Scholarly NLP Contributions for a Research Knowledge Graph

There is currently a gap between the natural language expression of scho...
research
10/09/2020

Sentence, Phrase, and Triple Annotations to Build a Knowledge Graph of Natural Language Processing Contributions – A Trial Dataset

Purpose: The aim of this work is to normalize the NLPCONTRIBUTIONS schem...

Please sign up or login with your details

Forgot password? Click here to reset