Learning to Simulate Dynamic Environments with GameGAN

by   Seung Wook Kim, et al.

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.



page 1

page 3

page 4

page 6

page 7

page 8

page 9

page 16


An Approach to Partial Observability in Games: Learning to Both Act and Observe

Reinforcement learning (RL) is successful at learning to play games wher...

A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals

Two key challenges within Reinforcement Learning involve improving (a) a...

DriveGAN: Towards a Controllable High-Quality Neural Simulation

Realistic simulators are critical for training and verifying robotics sy...

Shaping Belief States with Generative Environment Models for RL

When agents interact with a complex environment, they must form and main...

Text2Action: Generative Adversarial Synthesis from Language to Action

In this paper, we propose a generative model which learns the relationsh...

Act to Reason: A Dynamic Game Theoretical Model of Driving

The focus of this paper is to propose a driver model that incorporates h...

Generalization of Agent Behavior through Explicit Representation of Context

In order to deploy autonomous agents in digital interactive environments...

Code Repositories


Learning to Simulate Dynamic Environments with GameGAN (CVPR 2020)

view repo


playing with game gan

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.