Playing for Data: Ground Truth from Computer Games

08/07/2016
by   Stephan R. Richter, et al.
0

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.

READ FULL TEXT

page 2

page 8

page 9

page 11

page 12

research
05/12/2019

Style transfer based data augmentation in material microscopic image processing

Recently progress in material microscopic image semantic segmentation ha...
research
06/11/2020

Transferring and Regularizing Prediction for Semantic Segmentation

Semantic segmentation often requires a large set of images with pixel-le...
research
07/16/2018

Unlimited Road-scene Synthetic Annotation (URSA) Dataset

In training deep neural networks for semantic segmentation, the main lim...
research
12/09/2016

Automatic Model Based Dataset Generation for Fast and Accurate Crop and Weeds Detection

Selective weeding is one of the key challenges in the field of agricultu...
research
08/05/2016

Play and Learn: Using Video Games to Train Computer Vision Models

Video games are a compelling source of annotated data as they can readil...
research
09/05/2016

UnrealCV: Connecting Computer Vision to Unreal Engine

Computer graphics can not only generate synthetic images and ground trut...
research
08/05/2020

Interprocess Communication in FreeBSD 11: Performance Analysis

Interprocess communication, IPC, is one of the most fundamental function...

Please sign up or login with your details

Forgot password? Click here to reset