Learning to Simulate Dynamic Environments with GameGAN

by   Seung Wook Kim, et al.

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.


page 1

page 3

page 4

page 6

page 7

page 8

page 9

page 16


An Approach to Partial Observability in Games: Learning to Both Act and Observe

Reinforcement learning (RL) is successful at learning to play games wher...

A Coupling Approach to Analyzing Games with Dynamic Environments

The theory of learning in games has extensively studied situations where...

A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals

Two key challenges within Reinforcement Learning involve improving (a) a...

Shaping Belief States with Generative Environment Models for RL

When agents interact with a complex environment, they must form and main...

DriveGAN: Towards a Controllable High-Quality Neural Simulation

Realistic simulators are critical for training and verifying robotics sy...

DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation

VLN-CE is a recently released embodied task, where AI agents need to nav...

Generalization of Agent Behavior through Explicit Representation of Context

In order to deploy autonomous agents in digital interactive environments...

Code Repositories


Learning to Simulate Dynamic Environments with GameGAN (CVPR 2020)

view repo


playing with game gan

view repo

Please sign up or login with your details

Forgot password? Click here to reset