OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking

by   Xueqing Wu, et al.

Open-vocabulary state tracking is a more practical version of state tracking that aims to track state changes of entities throughout a process without restricting the state space and entity space. OpenPI is to date the only dataset annotated for open-vocabulary state tracking. However, we identify issues with the dataset quality and evaluation metric. For the dataset, we categorize 3 types of problems on the procedure level, step level and state change level respectively, and build a clean dataset OpenPI-C using multiple rounds of human judgment. For the evaluation metric, we propose a cluster-based metric to fix the original metric's preference for repetition. Model-wise, we enhance the seq2seq generation baseline by reinstating two key properties for state tracking: temporal dependency and entity awareness. The state of the world after an action is inherently dependent on the previous state. We model this dependency through a dynamic memory bank and allow the model to attend to the memory slots during decoding. On the other hand, the state of the world is naturally a union of the states of involved entities. Since the entities are unknown in the open-vocabulary setting, we propose a two-stage model that refines the state change prediction conditioned on entities predicted from the first stage. Empirical results show the effectiveness of our proposed model especially on the cluster-based metric. The code and data are released at


page 1

page 2

page 3

page 4


Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking

Open domain entity state tracking aims to predict reasonable state chang...

Tracking entities in technical procedures – a new dataset and baselines

We introduce TechTrack, a new dataset for tracking entities in technical...

Entity Tracking in Language Models

Keeping track of how states and relations of entities change as a text o...

Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric

Combining the Color and Event cameras (also called Dynamic Vision Sensor...

OpenPI2.0: An Improved Dataset for Entity Tracking in Texts

Representing texts as information about entities has long been deemed ef...

PeTra: A Sparsely Supervised Memory Model for People Tracking

We propose PeTra, a memory-augmented neural network designed to track en...

Semi-Automated Computer Vision based Tracking of Multiple Industrial Entities – A Framework and Dataset Creation Approach

This contribution presents the TOMIE framework (Tracking Of Multiple Ind...

Please sign up or login with your details

Forgot password? Click here to reset