V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation

03/23/2019
by   Anh Nguyen, et al.
12

We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential information of the command, while effectively encodes the fine-grained actions. The experimental results on our new large-scale dataset show that V2CNet outperforms recent state-of-the-art methods by a substantial margin, while its output can be applied in real robotic applications. The source code and trained models will be made available.

READ FULL TEXT

page 2

page 5

page 7

page 11

page 12

research
10/01/2017

Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks

We present a new method to translate videos to commands for robotic mani...
research
04/19/2021

Temporal Query Networks for Fine-grained Video Understanding

Our objective in this work is fine-grained classification of actions in ...
research
09/21/2017

AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection

We propose AffordanceNet, a new deep learning approach to simultaneously...
research
09/10/2019

Learning Actions from Human Demonstration Video for Robotic Manipulation

Learning actions from human demonstration is an emerging trend for desig...
research
03/05/2022

MetaFormer: A Unified Meta Framework for Fine-Grained Recognition

Fine-Grained Visual Classification(FGVC) is the task that requires recog...
research
12/22/2017

SFCN-OPI: Detection and Fine-grained Classification of Nuclei Using Sibling FCN with Objectness Prior Interaction

Cell nuclei detection and fine-grained classification have been fundamen...
research
10/20/2019

Processing Large Datasets of Fined Grained Source Code Changes

In the era of Big Code, when researchers seek to study an increasingly l...

Please sign up or login with your details

Forgot password? Click here to reset