Action Recognition and State Change Prediction in a Recipe Understanding Task Using a Lightweight Neural Network Model

01/23/2020
by   Qing Wan, et al.
0

Consider a natural language sentence describing a specific step in a food recipe. In such instructions, recognizing actions (such as press, bake, etc.) and the resulting changes in the state of the ingredients (shape molded, custard cooked, temperature hot, etc.) is a challenging task. One way to cope with this challenge is to explicitly model a simulator module that applies actions to entities and predicts the resulting outcome (Bosselut et al. 2018). However, such a model can be unnecessarily complex. In this paper, we propose a simplified neural network model that separates action recognition and state change prediction, while coupling the two through a novel loss function. This allows learning to indirectly influence each other. Our model, although simpler, achieves higher state change prediction performance (67 accuracy for ours vs. 55 train (10K ours vs. 65K+ by (Bosselut et al. 2018)).

READ FULL TEXT

page 1

page 2

research
02/04/2018

Human Action Adverb Recognition: ADHA Dataset and A Three-Stream Hybrid Model

We introduce the first benchmark for a new problem --- recognizing human...
research
06/17/2019

A Temporal Sequence Learning for Action Recognition and Prediction

In this work[This work was supported in part by the National Science Fou...
research
11/20/2017

Action Recognition with Coarse-to-Fine Deep Feature Integration and Asynchronous Fusion

Action recognition is an important yet challenging task in computer visi...
research
08/25/2020

Spatiotemporal Action Recognition in Restaurant Videos

Spatiotemporal action recognition is the task of locating and classifyin...
research
03/21/2018

T-RECS: Training for Rate-Invariant Embeddings by Controlling Speed for Action Recognition

An action should remain identifiable when modifying its speed: consider ...
research
04/16/2018

An information-theoretic on-line update principle for perception-action coupling

Inspired by findings of sensorimotor coupling in humans and animals, the...
research
05/24/2019

Human vs. Muppet: A Conservative Estimate of HumanPerformance on the GLUE Benchmark

The GLUE benchmark (Wang et al., 2019b) is a suite of language understan...

Please sign up or login with your details

Forgot password? Click here to reset