Procedural Content Generation via Machine Learning (PCGML)

02/02/2017 ∙ by Adam Summerville, et al. ∙ AppAbove Drexel University University of California Santa Cruz NYU college Georgia Institute of Technology 0

This survey explores Procedural Content Generation via Machine Learning (PCGML), defined as the generation of game content using machine learning models trained on existing content. As the importance of PCG for game development increases, researchers explore new avenues for generating high-quality content with or without human involvement; this paper addresses the relatively new paradigm of using machine learning (in contrast with search-based, solver-based, and constructive methods). We focus on what is most often considered functional game content such as platformer levels, game maps, interactive fiction stories, and cards in collectible card games, as opposed to cosmetic content such as sprites and sound effects. In addition to using PCG for autonomous generation, co-creativity, mixed-initiative design, and compression, PCGML is suited for repair, critique, and content analysis because of its focus on modeling existing content. We discuss various data sources and representations that affect the resulting generated content. Multiple PCGML methods are covered, including neural networks, long short-term memory (LSTM) networks, autoencoders, and deep convolutional networks; Markov models, n-grams, and multi-dimensional Markov chains; clustering; and matrix factorization. Finally, we discuss open problems in the application of PCGML, including learning from small datasets, lack of training data, multi-layered learning, style-transfer, parameter tuning, and PCG as a game mechanic.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 6

page 7

page 8

page 9

page 11

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Procedural content generation (PCG), the creation of game content through algorithmic means, has become increasingly prominent within both game development and technical games research. It is employed to increase replay value, reduce production cost and effort, to save storage space, or simply as an aesthetic in itself. Academic PCG research addresses these challenges, but also explores how PCG can enable new types of game experiences, including games that can adapt to the player. Researchers also addresses challenges in computational creativity and ways of increasing our understanding of game design through building formal models [1].

In the games industry, many applications of PCG are what could be called “constructive” methods, using grammars or noise-based algorithms to create content in a pipeline without evaluation. Many techniques use either search-based methods [2]

(for example using evolutionary algorithms) or solver-based methods 

[3] to generate content in settings that maximize objectives and/or preserve constraints. What these methods have in common is that the algorithms, parameters, constraints, and objectives that create the content are in general hand-crafted by designers or researchers. While it is common to examine existing game content for inspiration, machine learning methods have far less commonly been used to extract data from existing game content in order to create more content.

Concurrently, there has been an explosion in the use of machine learning to train models based on datasets [4]. In particular, the resurgence of neural networks under the name deep learning has precipitated a massive increase in the capabilities and application of methods for learning models from big data [5, 6]

. Deep learning has been used for a variety of tasks in machine learning, including the generation of content. For example,

generative adversarial networks have been applied to generating artifacts such as images, music, and speech [7]. But many other machine learning methods can also be utilized in a generative role, including -grams, Markov models, autoencoders, and others [8, 9, 10]. The basic idea is to train a model on instances sampled from some distribution, and then use this model to produce new samples.

This paper is about the nascent idea and practice of generating game content from machine-learned models. We define Procedural Content Generation via Machine Learning (abbreviated PCGML) as the generation of game content by models that have been trained on existing game content. The difference to search-based [2] and solver-based [3, 11] PCG is that while the latter approaches might use machine-learned models for evaluation, the content generation happens through search in content space; in PCGML, the content is generated directly from the model. The difference to experience-driven PCG [12] is that the learned models are models of game content, not models of player experience, behavior or preference. Other approaches, such as learning-based PCG [13], are more specific methodologies with restricted applicability.

The content models could be of many different kinds and trained using very different training algorithms, including neural networks, probabilistic models, decision trees, and others. The generation could be partial or complete, autonomous, interactive, or guided. The content could be almost anything in a game, such as levels, maps, items, weapons, quests, characters, rules, etc.

This paper focuses on game content that is directly related to game mechanics. In other words, we focus on functional rather than cosmetic game content. We define functional content as artifacts that, if they were changed, could alter the in-game effects of a sequence of player actions. The main types of cosmetic game content that we exclude are textures and sound, as those do not directly impact the effects of in-game actions the way levels or rules do in most games, and there is already much research on the generation of such content outside of games [14, 15]. This is not a value judgment, and cosmetic content is extremely important in games; however, it is not the focus of this paper. Togelius et al. [16] previously defined a related categorization with the terms necessary and optional. We note that while there exists some overlap between necessary and functional, it is possible to have optional functional content (e.g., optional levels) and necessary cosmetic content (e.g., the images and sound effects of a player character).

It is important to note a key difference between game content generation and procedural generation in many other domains: most game content has strict structural constraints to ensure playability. These constraints differ from the structural constraints of text or music because of the need to play games in order to experience them. Where images, sounds, and in many ways also text can be consumed statically, games are dynamic and must be evaluated through interaction that requires non-trivial effort—in Aarseth’s terminology, games are ergodic media [17]. A level that structurally prevents players from finishing it is not a good level, even if it’s visually attractive; a strategy game map with a strategy-breaking shortcut will not be played even if it has interesting features; a game-breaking card in a collectible card game is merely a curiosity; and so on. Thus, the domain of game content generation poses different challenges from that of other generative domains. Of course, there are many other types of content in other domains which pose different, and in some sense more difficult challenges, such as lifelike and beautiful images or evocative musical pieces; however, in this paper we focus on the challenges posed by game content by virtue of its necessity for interaction.

The remainder of this paper is structured as follows. Section II describes the various use cases for PCGML, including various types of generation and uses of the learned models for purposes that are not strictly generative. Section IV-B discusses the key problem of data acquisition and the recurring problem of small datasets. Section III includes a large number of examples of PCGML approaches. As we will see, there is already a large diversity of methodological approaches, but only a limited number of domains have been attempted. In Section IV, we outline a number of important open problems in the research and application of PCGML.

Ii Use Cases for PCGML

Procedural Content Generation via Machine Learning shares many uses with other forms of PCG: in particular, autonomous generation, co-creation/mixed initiative design, and data compression. However, because it has been trained on existing content, it can extend into new use areas, such as repair and critique/analysis of new content.

Ii-a Autonomous Generation

The most straightforward application of PCGML is autonomous PCG: the generation of complete game artifacts without human input at the time of generation. Autonomous generation is particularly useful when online content generation is needed, such as in rogue-like games.

PCGML is well-suited for autonomous generation because the input to the system can be examples of representative content specified in the content domain. With search-based PCG using a generate-and-test framework, a programmer must specify an algorithm for generating the content and an evaluation function that can validate the fitness of the new artifact [16]. However, these are rarely coded in the same space as the output artifact, requiring the designer to use a different domain to express their ideas. With PCGML, a designer can create a set of representative artifacts in the target domain as a model for the generator, and then the algorithm can generate new content in this style. PCGML avoids the complicated step of experts having to codify their design knowledge and intentions.

Ii-B Co-creative and Mixed-initiative Design

A more compelling use case for PCGML is AI-assisted design, where a human designer and an algorithm work together to create content. This approach has previously been explored with other methods such as constraint satisfaction algorithms and evolutionary algorithms [18, 19, 11].

Again, because the designer can train the machine-learning algorithm by providing examples in the target domain, the designer is “speaking the same language” the algorithm requires for input and output. This has the potential to reduce frustration, user error, user training time, and lower the barrier to entry because a programming language is not required to specify generation or acceptance criteria.

PCGML algorithms are provided with example data, and thus are suited to auto-complete game content that is partially specified by the designer. Within the image domain, we have seen work on image inpainting, where a neural network is trained to complete images where parts are missing [20]. Similarly, machine learning methods could be trained to complete partial game content.

Ii-C Repair

With a library of existing representative content, PCGML algorithms can identify areas that are not playable (e.g., if an unplayable level or impossible rule set has been specified) and offer suggestions for how to fix them. Summerville and Mateas [21] use a special tile that represents where an AI would choose to move the player in their training set, so that the algorithm only generates playable content; the system inherently has learned the difference between passable and impassable terrain. Jain et. al. [22] used a sliding window and an autoencoder to repair illegal level segments – because they did not appear in the training set, the autoencoder replaced them with a nearby window seen during training.

Ii-D Recognition, Critique, and Analysis

A use case for PCGML that sets it apart from other PCG approaches is its capacity for recognition, analysis, and critique of game content. Given the basic idea of PCGML is to train some kind of model on sets of existing game content, these models could be applied to analyzing other game content, whether created by an algorithm, players, or designers.

For example, by supervised training on sets of game content artifacts and player response data, machine learning models can be trained to classify game content into various types, such as broken or whole, underworld or overworld, etc. They can be trained to predict quantities such as difficulty, completion time, or emotional tenor

[23, 24]. They also have the potential to identify uniqueness, for example by noting how frequently a particular pattern appears in the training set, or judging how typical a complete content artifact is with regard to an existing set. Being able to identify which user-created levels are truly novel, and then enhance them further so as to fit with the aesthetics of the game and repair broken parts, could be very useful. Such models could be used to automatically evaluate game content, as is already done within many applications of search-based PCG, and potentially be used with other generative methods.

Ii-E Data Compression

One of the original motivations for PCG, particularly in early games such as Elite [25], was data compression. There was not enough space on disk for the game universe. The same is true for some of today’s games such as No Man’s Sky [26]

. The compression of game data into fewer dimensions through machine learning could allow more efficient game content storage. By exploiting the regularities of a large number of content instances, we can store the distinctive features of each more cheaply. Unsupervised learning methods such as autoencoders might be particularly well suited to this.

Figure 1: Our taxonomization of PCGML techniques. We have two categorizations (1) the underlying data structure (graph, grid, or sequence) and (2)

the training method (matrix factorization, expectation maximization, frequency counting, evolution, and back propagation). Marks are colored for the specific type of content that was generated: red circles are platformer levels, orange squares are “dungeons”, the dark blue x is real time strategy levels, light blue triangles are collectible game cards, and the purple star is interactive fiction. Citations for each are listed.

Iii Methods of PCGML

We organize PCGML techniques using the following two dimensions:

  • [leftmargin=*]

  • Data Representation The underlying representation of the data used for training and generation. We consider three representations: Sequences, Grids, and Graphs. We note that it is possible for the same type of content to be represented in many different formats (e.g., platformer levels have been represented as all three representations), and that wildly different content can be represented in the same format (e.g., levels and Magic cards as sequences) .

  • Training Method The machine learning technique utilized for training the model. We consider five broad categories of training algorithms: Back Propagation, Evolution, Frequency Counting, Expectation Maximization, and Matrix Factorization. We note that it is both possible for the same underlying machine learned representation to be trained via different techniques and for two different techniques to utilize the same underlying class of training method (e.g., neural networks can be trained via back propagation [27] [21] [28] or evolution [29] and Expectation Maximization

    can be used to train a Bayesian Network

    [30]

    or K-Means centroids

    [31]).

This organization has the benefits of highlighting commonalities across different techniques and game content. Figure 1 shows a graphical representation of the different approaches that have been attempted. In the following sections we highlight different PCGML methodologies according to this taxonomy and discuss potential future work to address gaps in the coverage of the prior approaches.

Iii-a Sequences

Sequences represent a natural format for many different types of content, both textual (Magic cards) and game levels. We note that the only game levels that has been handled as a sequence have come from the early Super Mario Bros. games where the player can only traverse from left-to-right, meaning that there is a natural ordering of the two-dimensional space into a one-dimensional sequence.

Iii-A1 Frequency Counting

Frequency counting refers to methods wherein the data is split and the frequencies of each type of atomic generative piece (e.g., tile for a tilemap based game) are found, determining the probabilities of generation. These need not simply be the raw frequencies, but are more likely the conditional probability of a piece given some state. Markov chains are a class of techniques that learn conditional probabilities of the next state in a sequence based on the current state. This state can incorporate multiple previous states via the construction of

-grams. An -gram model (or -gram for short) is simply an -dimensional array, where the probability of each state is determined by the states that precede it.

Dahlskog et al. trained -gram models on the levels of the original Super Mario Bros. game, and used these models to generate new levels [32]. As -gram models are fundamentally one-dimensional, these levels needed to be converted to strings in order for -grams to be applicable. This was done through dividing the levels into vertical “slices,” where most slices recur many times throughout the level [33]. This representational trick is dependent on there being a large amount of redundancy in the level design, something that is true in many games. Models were trained using various levels of , and it was observed that while creates essentially random structures and creates barely playable levels, and create rather well-shaped levels. See Figure 2 for examples of this.

(a)
(b)
(c)
Figure 2: Mario levels reconstructed by -grams with set to 1, 2, and 3 respectively. Figures reproduced with permission from  [32].

Summerville et al.  [34] extended these models with the use of Monte Carlo Tree Search (MCTS) to guide generation. Instead of solely relying on the learned conditional probabilities, they used the learned probabilities during roll-outs (generation of whole levels) that were then scored based on an objective function specified by a designer (e.g., allowing them to bias the generation towards more or less difficult levels). The generated levels could still only come from observed configurations, but the utilization of MCTS meant that playability guarantees could be made and allowed for more designer control than just editing of the input corpus.

Figure 3: The ANN Representation. At each column in a given level, an ANN inputs visual assets (i.e. musical voices) from at least one tile type of Super Mario Bros. and outputs a different type of visual asset. This example figure shows ground tiles being input to the ANN while the brick tile placements are output from predictions made through NeuroEvolution of Augmenting Topologies [35]. To best capture the regularities in a level, each column is also provided information about the most recently encountered inputs (i.e. input values for the three previous columns). The inputs and outputs are then fed back into the ANN at each subsequent tick. Once trained, ANNs can potentially suggest reasonable placements for new human composed in-game tiles. Figures reproduced with permission from [29].

Iii-A2 Evolution

Evolutionary approaches rely on a genotype (typically represented as an array of numbers) that represent a parameterization of the type of instance that is being optimized. While evolutionary approaches have been used to generate content, they are not under the purview of what we are considering here. Instead, we are looking at evolutionary approaches that are used to generate generators where the objective function is derived from how well the generator is able to match the dataset.

Hoover et al. [29] generate levels for Super Mario Bros. by extending a representation called functional scaffolding for musical composition (FSMC) that was originally developed to compose music. The original FSMC representation posits 1) music can be represented as a function of time and 2) musical voices in a given piece are functionally related [36]. Through a method for evolving Artificial Neural Networks (ANNs) called NeuroEvolution of Augmenting Topologies (NEAT) [35], additional musical voices are evolved to be played simultaneously with an original human-composed voice.

To extend this musical metaphor and represent Super Mario Bros. levels as functions of time, each level is broken down into a sequence of tile-width columns. The height of each column extends the height of the screen. While FSMC represents a unit of time by the length of an eighth-note, a unit of time in this approach is width of each column.

At each unit of time, the system queries the ANN to decide a height to place a tile. FSMC inputs a musical note’s pitch and duration to the ANNs, this approach translates pitch to the height at which a tile is placed and duration to the number of times a tile-type is repeated at that height. For a given tile-type or musical voice, this information is then input to an ANN that is trained on two-thirds of the existing human-authored levels to predict the value of a tile-type at each column (as shown in Figure 3). The idea is that the ANN will learn hidden relationships between the tile-types in the human-authored levels that can then help humans construct entire levels from as little starting information as the layout of a single tile-type.

Like in FSMC, this approach combines a minimal amount of human-authored content with the output from previous iterations. The output of the network is added to the newly created level and fed back as input into the generation of new tile layouts. By acknowledging and iterating on the relationships between design pieces inherent in a human-produced level, this method can generate maps that both adhere to some important aspects of level design while deviating from others in novel ways.

It should be noted that there are many uses of evolution and evolution-like algorithms for PCG that rely heavily on machine learning for the fitness functions, but because generation happens through a global search process they are not PCGML. Another interesting combination of search and machine learning for PCG is the DeLeNoX algorithm, which uses unsupervised learning to continuously reshape the search space and novelty search to search for content to explore the search space [37].

Iii-A3 Back Propagation

Artificial Neural Networks are universal function approximators and have seen use in the field of PCGML as an approximator for a designer. ANNs can be trained via evolutionary techniques as discussed in the previous section, but they are commonly trained via back propagation. Back propagation refers to the propagation of errors through the ANN, with each weight in the ANN being changed proportional to the amount of error for which it is responsible. Where the evolutionary approaches are only able to score the entire network, back propagation tunes parts of the network responsible for errors; however, this comes at the cost that all functions in the ANN must be differentiable (which evolutionary approaches do not require).

Summerville and Mateas [21]

used Long Short-Term Memory Recurrent Neural Networks (LSTM RNNs)

[38] to generate levels learned from a tile representation of Super Mario Bros. and Super Mario Bros. 2 (JP) [39] levels. LSTMs are a variant of RNNs that represent the current state-of-the-art for sequence based tasks, able to hold information for 100’s and 1000’s of time steps, unlike the 5-6 of standard of RNNs. Summerville and Matteas used a tile representation representing levels as a linear string with 3 different representations used for experimentation. They also included simulated player path information in the input data, forcing the generator to generate exemplar paths in addition to the level geometry. Finally, they included information about how deep into the string the level geometry was, causing the generator to learn both level progression and when a level should end.

Summerville et al. [40] extended this work to incorporate actual player paths extracted from 4 different YouTube playthroughs. The incorporation of a specific player’s paths biased the generators to generate content suited to that player (e.g., the player that went out of their way to collect every coin and question-mark block had more coins and question-marks in their generated levels).

Figure 4: Example output of the LSTM approach, including generated exemplar player path. Figure reproduced with permission from [21].

A rare example of game content generation that is not a level generator is Magic: The Gathering [41] card generation. The first to use a machine learning approach for this is Morgan Milewicz with @RoboRosewater [42] a twitter bot that uses LSTMs to generate cards. Trained on the entirety of the corpus of Magic cards it generates cards, represented as a sequence of text fields (e.g., Cost, Name, Type, etc.). A limitation of this representation and technique is that by treating cards as a sequence there is no way to condition generation of cards on fields that occur later in the sequence (e.g., Cost occurs near the end of the sequence and as such can not be used to condition the generation unless all previous fields are specified).

Building on the work of @RoboRosewater is Mystical Tutor by Summerville and Mateas [28]. Using sequence-to-sequence learning [43]

, wherein an encoding LSTM encodes an input sequence into a fixed length vector which is then decoded by a decoder LSTM, they trained on the corpus of

Magic: The Gathering. They corrupted the input sequences by replacing lines of text with a MISSING token and then tried to reproduce the original cards as the output sequence. The corruption of the input along with the sequence-to-sequence architecture allows the generator to be conditioned on any piece of the card, addressing one of the limitations of @RoboRosewater. This work shows multiple different paradigms for PCGML, showing how it can be used on different game content and as a co-creative design assistant, as in Hoover et al.’s [29] previously mentioned work.

Figure 5: Partial card specification and the output. Figure reproduced with permission from [28].

Iii-B Grids

Most game levels (particularly of the pre-3D era) can be thought of as two-dimensional grids. Sometimes these representations are lossy (e.g. a non-tile entity is forcibly aligned to the grid, even if it could actually be at a non-tile position), but generally represent a natural representation for many different kinds of levels (e.g. platformers, dungeons, real time strategy maps, etc.).

Iii-B1 Frequency Counting

An extension to the previously discussed one dimensional Markov chains are Multi-dimensional Markov Chains (MdMCs) [44], wherein the state represents a surrounding neighborhood and not just a single linear dimension. Snodgrass and Ontañón [45] present an approach to level generation using MdMCs. An MdMC differs from a standard Markov chain in that it allows for dependencies in multiple directions and from multiple states, whereas a standard Markov chain only allows for dependence on the previous state alone. In their work, Snodgrass and Ontañón represent video game levels as 2-D arrays of tiles representing features in the levels. For example, in Super Mario Bros. they use tiles representing the ground, enemies, and ?-blocks, etc. These tile types are used as the states in the MdMC. That is, the type of the next tile is dependent upon the types of surrounding tiles, given the network structure of the MdMC (i.e. the states that the current state’s value depends on).

They train an MdMC by building a probability table according to the frequency of the tiles in training data, given the network structure of the MdMC, the set of training levels, and the set of tile types. A new level is then sampled one tile at a time by probabilistically choosing the next tile based upon the types of the previous tiles and the learned probability table.

Figure 6: Sections from a Super Mario Bros. level (top) and Kid Icarus level sections generated using the constrained MdMC approach (bottom-left), and using an MRF approach (bottom right). Figures reproduced with permission from [46, 47].

In addition to their standard MdMC approach, Snodgrass and Ontañón have explored hierarchical [48] and constrained [46] extensions to MdMCs in order to capture higher level structures and ensure usability of the sampled levels, respectively. They have also developed a Markov random field approach (MRF) [47] that performed better than the standard MdMC model in Kid Icarus, a domain where platform placement is pivotal to playability. Figure 6 shows a section of a Super Mario Bros. level (top) and a section of a Kid Icarus level (bottom-left) sampled using a constrained MdMC approach, and a section of Kid Icarus level (bottom-right) sampled using the MRF approach.

A recent approach by Gumin [49] is loosely inspired by quantum mechanics and uses a “superposition” of tiles to generate images and levels from a representative example tile set. This approach is a variant of MRF, except instead of solely sampling, samples are chosen via “collapsing of the wave function” (i.e. probabilistically choosing a tile and propagating constraints that that choice enforces). This in turn can propagate other changes and either deterministically chooses tiles that no longer have any other possible choices or reduces the possible set of other tiles. The probabilities and configurations are determined by finding each window in the input, and the number of times that window occurs. This approach was initially explored for bitmap generation, but has since been expanded for use with 3-D tile sets as well as for level generation [50, 51]. The source code and examples of the bitmap project can be found online [49].

Figure 7: Level section from the clustering approach. Figure reproduced with permission from [31].

Iii-B2 Back Propagation

In [22] Jain et al. show how autoencoders [52] may be trained to reproduce levels from the original Super Mario Bros. game. The autoencoders are trained on series of vertical level windows and compress the typical features of Mario levels into a more general representation. They experimented with the width of the level windows and found that four tiles seems to work best for Mario levels. They proceeded to use these networks to discriminate generated levels from original levels, and to generate new levels as transformation from noise. They also demonstrated how a trained autoencoder may be used to repair unplayable levels, changing tile-level features to make levels playable, by inputting a broken/unplayable level to the autoencoder and receiving a repaired one as output (see Figure 8).

(a) Original

(b) Unplayable

(c) Repaired
Figure 8: The original window is overwritten with a wall making the game unplayable. The autoencoder repairs the window to make it playable, although it chooses a different solution to the problem. Figures reproduced with permission from [22].
Figure 9: Varying resource amounts when generating resource locations for StarCraft II. Figures reproduced with permission from [53].

Lee et al. [53]

use convolutional neural networks to predict resource locations in maps for

StarCraft II [54]. Resource locations are sparsely represented in StarCraft II maps, but are decisive in shaping gameplay. They are usually placed to match the topology of maps. Exploiting this fact, they transform StarCraft II maps into heightmaps, downsample them, and train deep neural networks to predict resource locations. The neural networks are shown to perform well in some cases, but struggled in other cases, most likely due to overfitting the small training dataset. By adding a number of postprocessing steps to the network, the authors created a tool that allows map designers to vary the amount of resource locations in an automatically decorated map, varying the frequency of mineral placements, shown in Figure 9.

Iii-B3 Matrix Factorization

Matrix factorization is a way of extracting latent features from high-dimensional data. These techniques have been explored in the context of learning from level data and from other level generators.

Many generators (both machine and human) create in a particular identifiable style with a limited expressive range [55]. Shaker and Abou-Zleikha [56] use Non-Negative Matrix Factorization (NNMF) to learn patterns from 5 non-ML-based generators of Super Mario Bros. levels (Notch, Parameterized, Grammatical Evolution, Launchpad, and Hopper), to create a wider variety of content than any of the single methods. To their knowledge, this is the first time that NNMF has been used to generate content. NNMF requires only non-negative values in its matrix decomposition, thus the basis representations are localized and purely additive, and therefore more intuitive to analyze than PCA.

Their system begins by generating levels of length with each of the generators. These levels are represented as vectors recording the locations or amount of content types at each level column: Platforms, Hills, Gaps, Items, and Enemies. These vectors are combined into matrices, one for each level content type. The algorithm then uses a multiplicative update NNMF algorithm [57] to factor the data matrices into approximate “part matrices” which represent level patterns and coefficient matrices corresponding to weights for each pattern. The part matrices can be examined to see what types of patterns appear globally and uniquely in different generators, and multiplying these part matrices by novel coefficient vectors can be used to generate new levels. This allows the NNMF method to explore far outside the expressive range of the original generators.

Figure 10:

Example of interpolation between two Zelda rooms (the leftmost and rightmost rooms). Figure reproduced with permission from

[30].

While platformers, specifically Super Mario Bros., have dominated the field, Summerville et al. [30] generated levels learned from The Legend of Zelda [58] series. Their approach relies on hierarchical segmentation of the data, first generating the high level topological structure of a dungeon (discussed in Section III-C

) and then generating rooms as a grid of tiles via Principal Component Analysis (PCA). PCA finds a compressed representation of the original 2-D room arrays by taking the eigenvectors of the original high dimensional space and only keeping the most informative, in this work only keeping the top 20 most informative accounting for 95% of all variation in an approach inspired by EigenFaces

[59]. These compressed representations are represented as weight vectors that can be interpolated between to generate new room content as seen in Figure 10.

Iii-C Graphs

Graphs are the most general data representation considered, with the previous data representations being easily representable as graphs. However, this generality comes at a cost, which is the lack of well-defined structural properties (e.g. in a grid, above and below are implicitly encoded in the structure).

Iii-C1 Expectation Maximization

Expectation Maximization (EM) is an iterative algorithm that seeks to find the Maximum Likelihood Estimate (MLE) or Maximum A Posteriori (MAP) estimate for the parameters of a model. This is done via a two step process, first an Expectation (E) step wherein the current model’s likelihood is computed given the training data and a Maximization (M) step where the parameters are changed to maximize the previously calculated likelihood (which are used in the next E step). EM is a general training method for any model that is able to provide a likelihood of the model given training data.

K-means is a clustering method where an a priori defined number of clusters are computed based on the means of the data. Each E step determines which data points belong to which cluster, resulting in new means being calculated for each cluster in the M step. Guzdial and Riedl [60]

used a hierarchical clustering approach using K-means clustering with automatic K estimation to train their model. They utilized gameplay video of individuals playing through

Super Mario Bros. to generate new levels. They accomplished this by parsing each Super Mario Bros. gameplay video frame-by-frame with OpenCV [61] and a fan-authored spritesheet, a collection of each image that appears in the game. Individual parsed frames could then combine to form chunks of level geometry, which served as the input to the model construction process. In total Guzdial and Riedl made use of nine gameplay videos for their work with Super Mario Bros., roughly 4 hours of gameplay in total.

Guzdial and Riedl’s model structure was adapted from [62], a graph structure meant to encode styles of shapes and their probabilistic relationships. The shapes in this case refer to collections of identical sprites tiled over space in different configurations. For further details please see [60], but it can be understood as a learned shape grammar, identifying individual shapes and probabilistic rules on how to combine them. First the chunks of level geometry were clustered to derive styles of level chunks, then the shapes within that chunk were clustered again to derive styles of shapes, and lastly the styles of shapes were clustered to determine how they could be combined to form novel chunks of level. After this point generating a new level requires generating novel level chunks in sequences derived from the gameplay videos. An example screen of generated content using this approach is shown in Figure 7.

Figure 11: Example of Scheherazade-IF gameplay. Figure reproduced with permission from [63].

As previously discussed, Summerville et al. [64] generated dungeons using two different data representations, with the high level topological structure defined as a graph with rooms as nodes and linkages (e.g., doors, bombable walls, teleportation portals) as edges. The room-to-room structure of a dungeon is learned along with high level parameters such as dungeon size and length of the optimal player path using a Bayes Net [65]

, a graphical structure of the joint probability distribution. Bayes Nets are trainable via a number of different techniques (e.g., Gibbs Sampling, Variational Message Passing), but all are based broadly around maximizing the likelihood of a model given the data. After training the Bayes Net, a designer can “observe” specific parameters of the model such as how many rooms it should have, how many rooms the player would need to traverse to complete the level, etc. and then sample the rest room by room.

Iii-C2 Frequency Counting

PCGML has focused on graphical games, and particularly the levels of graphical games. However, there exists some work in the field of generating interactive fiction, text-based games like choose your own adventure stories. Guzdial et al. adapted Scheherazade [66], a system for automatically learning to generate stories, into Scheherazade-IF [63], which can derive entire interactive fiction games from a dataset of stories.

Both Scheherazades rely on exemplar stories crowdsourced from Amazon Mechanical Turk (AMT). The reliance on crowdsourcing rather than finding stories “in the wild” allows Scheherazade to collect linguistically simple stories and stories related to a particular genre or setting (e.g., a bank robbery, a movie date, etc).

Scheherazade-IF’s structures its model as plot graphs [67], directed graphs with events as vertices and sequentiality information encoded as edges. The system derives events from the individual sentences of the story, learning what events can occur via an adaption of the OPTICS clustering algorithm [68]. The clusters found by OPTICS are found by only allowing points in a cluster if it keeps the density of the cluster the same. Unlike K-means, OPTICS is able to find clusters of arbitrary shape, assuming the shapes are of consistent density. The density is determined by looking in a region around each point and counting the number of other points. The ordering of these primitive events can then be derived from the exemplar stories, which enables the construction of a plot graph for each type of story (e.g., bank robbery). Scheherazade-IF uses these learned plot graphs as the basis for an interactive narrative experience, allowing players to take the role of any of the characters in the plot graph (robber or bank teller in bank robbery) and make choices based on the possible sequences of learned events.

Iii-D Unexplored Approaches

From Figure 1

we see that only slightly over half of the possible combinations have been touched on. Notably, evolutionary approaches have only been used to train ANNs for sequences, but given the generality of evolutionary approaches it seems possible for generators for both graph and grid approaches (given an appropriate objective function). EM approaches are commonly used for training Hidden Markov Models (HMMs)

[69] where the states are hidden from observation and only the output of a hidden state is observed. HMMs seem like a logical extension to the existing Markov chain work, as they allow for higher order structures to be learned instead of just column-to-column transitions. Extending Markov Random Fields similarly, Conditional Random Fields (CRF) [70] act as if the connected grid is based on an unobserved, latent state and have been used in image generation [71].

While it may seem that matrix factorization techniques would be ill-suited to sequences and graphs, those are actually fruitful areas of research. Global Vectors (GloVe) for word representation [72] utilize matrix factorization to embed categorical vocabularies into lower-dimensional real-valued vectors. These vectors are then used for analogical reasoning [73] or for generation [74], but have yet to see use in the generation of game content. Graphs can be represented as adjacency matrices, where the rows and columns are the nodes and each entry in the matrix represents whether nodes are connected (1) or not (0). This enables the field of spectral graph theory, and it seems possible that matrix factorization techniques could learn important properties of graphs represented this way.

Finally, until recently graphs have been an unexplored data structure for back propagation based neural networks. Recent work from Kipf and Welling  [75] on graph convolutional networks have only been used for classification purposes, but given the lag between pixel convolutional networks being used for classification (1998) [76] and for generation (2015) [77] this seems more a matter of time than a matter of possibility.

Iv Open Problems and Outlook

In this section, we describe some of the major issues facing PCGML and open areas for active research. Because games as a ML research domain are considerably different from other ML domains, such as vision and audio learning, the problems that arise are commensurately different. In particular, game datasets are typically much smaller than other domains’ datasets, they are dynamic systems that rely on interaction, and the availability of high-quality clean data is limited. In the following sections we discuss the issue of limited data as well as underexplored applications of PCGML for games including style transfer, parameter tuning, and potential PCGML game mechanics.

Iv-a Ensuring Solvability and Playability

A key aspect of procedural generation for games is the generation of playable artifacts. So far, most of this work has relied on designer specified constraints on the generated works, either via constraints after generation [46] or applying constraints at each step of sampling [34]. While there have been approaches that utilize player paths (either synthetic [21] or actual [40]), these do not ensure playability but instead bias the generators to create playable levels. Future work is required to train a generator to only generate playable levels, but we envision this could be achieved via penalizing the generator at training time, as in generative adversarial networks.

Iv-B Data Sources and Representations

In this section we investigate how training data, varying in source, quality, and representation, can impact PCGML work. A commonly held view in the machine learning community is that more data is better. Halevy, et al. [78] make the case that having access to more data, not better algorithms or better data, is the key difference between why some problems are able to be solved in an effective manner with machine learning while others are not (e.g., language translation is a much harder problem than document classification, but more data exists for the translation task). For some types of machine learning a surfeit of data exists enabling certain techniques, such as images for generative adversarial networks [79] or text for Long Short-Term Memory recurrent neural networks (LSTMs) [80]; however, video games do not have this luxury. While many games exist, they typically share no common data structures. Additionally, even if they do they typically do not share semantics (e.g., sprites in the NES share similarity in how they are displayed but sprite 0x01 is unlikely to have similar semantics between Super Mario Bros. [81] and Metroid [82], despite the fact that both are platformers). As such, data sources are often limited to within a game series [30, 21] and more commonly within individual games [32, 48, 60]. This in turn means that data scarcity is likely to plague PCGML for the foreseeable future, as there is a small, finite amount of official artifacts (e.g., maps) for most games. However, for many games there are vibrant fan communities producing hundreds, thousands, and even millions of artifacts, such as maps [83], which could conceivably provide much larger data sources.

Recently, Summerville et al. created the video game level corpus (VGLC) [84], a collection of video game levels represented in several easily parseable formats. The corpus contains 428 levels from 12 games in 3 different file formats. This marks an initial step in creating a shared data source that can be used for commonality across PCGML research. That said, the repository is limited in scale (12 games and 428 levels is small for standard machine learning) and the representations are lossy (e.g., in Mario, levels only contain a single Enemy type mapping for all Goombas, Koopa Troopas, etc.), so additional effort is needed to increase the scope and fidelity of the representations.

Assuming bountiful data existed for video games, bigger questions abound: 1) How should the data be represented? 2) What is a training instance? For certain classes of PCGML, such as image generation, obvious data structures exist (e.g., RGB matrices), but there are no common structures for games. Games are a combination of some amount of content (levels, images, 3-D models, text, boards, etc.) and the rules that govern them (What happens when input  is held down in state ? What happens if is clicked on? Can player  move piece ? etc.). Furthermore, the content is often dynamic during playtime, so even if the content were to capture with perfect fidelity (e.g., a Goomba starts at tile ) how it will affect gameplay requires play (e.g., by the time the player can see the Goomba it has moved to tile ). Thus, issues of representation serve as a second major challenge, after access to sufficient quantities of quality data.

Iv-C Learning from Small Datasets

As previously mentioned, it is a commonly held tenet that more data is better; however, games are likely to always be data-constrained. The English Gigaword corpus [85] which has been used in over 450 papers at the time of writing is approximately 27 GB. The entire library of games for the Nintendo Entertainment System is approximately 237 MB, over 2 orders of magnitude smaller. The most common genre, platformers, makes up approximately of the NES library (and is the most common genre for PCG research) which is roughly another order of magnitude smaller. For specific series it is even worse, with the largest series (Mega Man [86]) making up

of the NES library, for 4 orders of magnitude smaller than a standard language corpus. Standard image corpora are even larger than the Gigaword corpus, such as the ImageNet corpus

[87] at 155GB. All told, work in this area is always going to be constrained by the amount of data, even if more work is put into creating larger, better corpora.

While most machine learning is focused on using large corpora, there is work focused on One-Shot Learning/Generalization [88]. Humans have the ability to generalize very quickly to other objects of the same type after being shown a single object (or very small set, e.g., ) and One-Shot Learning is similarly concerned with learning from a very small dataset. Most of the work has been focused on image classification, but games are a natural testbed for such techniques due to the paucity of data.

Iv-D Learning on Different Levels of Abstraction

As described above, one key challenge that sets PCGML for games apart from PCGML for domains such as images or sound is the fact that games are multi-modal dynamic systems. This means that content generated will have interactions: generated rules operate on generated levels which in turn change the consequences of those rules and so on. A useful high-level formal model for understanding games as dynamic systems that create experiences is the Mechanics, Dynamics, Aesthetics (MDA) framework by Hunicke et al. [89]. “Mechanics describes the particular components of the game, at the level of data representation and algorithms. Dynamics describes the run-time behavior of the mechanics acting on player inputs and each other’s outputs over time. Aesthetics describes the desirable emotional responses evoked in the player, when she interacts with the game system.” If the end goal for PCGML for games is to generate content or even complete games with good results at the aesthetic level, the problem is inherently a hierarchical one. It is possible that learning from data will need to happen at all three levels simultaneously in order to successfully generate content for games.

Iv-E Datasets and Benchmarks

Procedural content generation, and specifically PCGML is a developing field of research. As such, there are not many publicly available datasets and no widely used standardized evaluation benchmarks.

Publicly available, large datasets are important, as they will reduce the barrier for entry into PCGML, which will allow the community to grow more quickly. As previously mentioned, the VGLC [84] is an important step in creating widely available datasets for PCGML. However, it currently only provides level data. Other data sets exist for object models111opengameart.org and gameplay mechanics222http://www.squidi.net/three/index.php, but they have not been utilized by the PCGML community. By bringing these data sets to the attention of researchers, we hope to promote the exploration and development of approaches for generating those types of content. Additionally, creating, improving, and growing both new and existing data sets is necessary for refining and expanding the field.

Widespread benchmarks allow for the comparison of various techniques in a standardized way. Previously, competitions have been used to compare various level generation approaches [90, 91]. However, many of the competitors in the Super Mario Bros. competition included hard coded rules, and the GVG-AI competition provides descriptions of the games, but not the training data necessary for machine learning approaches.

In addition to competitions, there has been some work in developing metric-based benchmarks for comparing techniques. Horn et al. [55] proposed a benchmark for Super Mario Bros.

level generators in a large comparative study of multiple generators, which provided several evaluation metrics that are commonly used now, but only for level generators for platforming games. In recent years, Canossa and Smith

[92] presented several general evaluation metrics for procedurally generated levels, which could allow for cross-technique comparisons. Again, however, these only apply to level generation and not other generatable content. Other machine learning domains, such as image processing, have benchmarks in place [76] for testing new techniques and comparing various techniques against each other. For the field of PCGML to grow, we need to be able to compare approaches in a meaningful way.

Iv-F Style Transfer

Style and concept transfer is the idea that information learned from one domain can enhance or supplement knowledge in another domain. Style transfer has most notably been explored for image modeling and generation [93, 94]. For example, in recent work Gatys et al. [95]

used neural networks to transfer the style of an artist onto different images. Additionally, Deep Dream

[96] trains neural networks on images, and then generates new images that excite particular layers or nodes of the network. This approach could be adapted to learn from game content (e.g., levels, play data, etc.) and generate content that excites layers associated with different elements from the training data. More traditional forms of blending domain knowledge have been applied sparingly to games, such as game creature blending [97] and interactive fiction blending [98]. However, until very recently no one has applied this machine learning-oriented style transfer to procedural content generation.

Recently Snodgrass and Ontañón [99] explored domain transfer in level generation across platforming games, and Guzdial et al. [31] used concept blending to meld different level designs from Super Mario Bros. together (e.g., castle, underwater, and overworld levels). These approaches transfer and blend level styles, but do not attempt to address the game mechanics explicitly; both approaches ensure playable levels, but do not attempt transfer or blending between different mechanics. Gow and Corneli [100] proposed a framework for blending two games together into a new game, including mechanics, but no implementation yet exists.

The above approaches are important first steps, but style transfer needs to be explored more fully in PCG. Specifically, approaches to transferring more than just aesthetics are needed, including game mechanic transfer. The above approaches only apply transfer to game levels, but there are many other areas where style transfer can be applied (e.g., character models, stories, and quests).

Iv-G Exposing and Exploring the Generative Space

One avenue that has yet to be deeply explored in PCGML is opening the systems up to allow for designer input. Most systems are trained on a source of data and the editorial control over what data is fed in is the main interaction for a designer, but it is easy to imagine that a designer would want to control things such as difficulty, complexity, or theming (e.g., the difference between an above ground level and below ground level in Super Mario Bros.). The generative approach of Summerville et al. [30] allows a designer to set some high-level design parameters (number of rooms in a dungeon, length of optimal player path), but this approach only works if all design factors are known at training time and a Bayes Net is a suitable technique. Similarly, Snodgrass and Ontañón [46] allow the user to define constraints (e.g., number of enemies, distance of the longest gap, etc.), but require hand crafted constraint checkers for each constraint used.

The nature of Generative Adversarial Networks has allowed for the discovery of latent factors that hold specific semantics (e.g., subtracting the latent space image of blank-faced person from a smiling one is the smile vector) [101]. Interpolating and extrapolating along these latent dimensions allows a user to generate content while freely tuning the parameters they see fit. We envision that this type of approach could lead to similar results in PCG (e.g., subtract Mario 1-1 from Mario 1-2 for the underground dimension, subtract Mario 1-1 from Mario 8-3 for the difficulty dimension). Furthermore, if multiple games are used as training input, it is theoretically possible that one could interpolate between two games (e.g., find the half-way point between Mario and Sonic) or find other more esoteric combinations ().

Iv-H Using PCGML as a Game Mechanic

Most current work in PCGML focuses on replicating designed content to provide the player infinite or novel variations on gameplay, informed by past examples. Another possibility is to use PCGML as the main mechanic of a game, presenting the PCGML system as an adversary or toy for the player to engage with. Designs could include enticing the player to, for example, generate content that is significantly similar to or different from the corpus the system was trained on, identify content examples that are outliers or typical examples of the system, or train PCGML systems to generate examples that possess certain qualities or fulfill certain objective functions, teaching the player to operate a model by feeding it examples that shape its output in one direction or the other. This would allow a PCGML system to take on several design pattern roles, including AI as Role-Model, Trainee, Editable, Guided, Co-Creator, Adversary, and Spectacle 

[102].
Role-model: a PCGML system replicates content generated by players of various levels of skill or generates content suitable for players of certain skill levels. New players are trained by having them replicate the content or by playing the generated content in a form of generative tutorial.
Trainee: the player trains a PCGML system to generate a piece of necessary content (e.g., part of a puzzle or level geometry).
Editable: rather than training the AI to generate the missing puzzle piece via examples, the player changes the internal model’s values until acceptable content is generated.
Guided: the player corrects PCG system output in order to fulfill increasingly difficult requirements. The AI, in turn, learns from the player’s corrections, following the player’s guidance.
Co-creator: the player and a PCGML system take turns in creating content, moving toward some external requirement. The PCGML system learns from the player’s examples.
Adversary: the player produces content that the PCGML system must replicate by generation to survive or vice versa in a “call and response” battle.
Spectacle: the PCGML system is trained to replicate patterns that are sensorically impressive or cognitively interesting.

V Conclusion

In this survey paper, we give an overview of an emerging machine learning approach to Procedural Content Generation, including describing and contrasting the existing examples of work taking this approach and outlining a number of challenges and opportunities for future research. We intend the paper to play a similar role as the Search-Based Procedural Content Generation paper [2], which pointed out existing research as well as work that was yet to be done. Much research that was proposed in that paper was subsequently carried out by various authors. In Table LABEL:tab:summary, we present a summary the Procedural Content Generation via Machine Learning papers discussed throughout this paper. As can be seen from this table, and indeed from the rest of the paper, there is much work left to do. The vast majority of work has so far concerned two-dimensional levels, in particular Super Mario Bros.

levels. Plenty of work remains in applying these methods to other domains, including rulesets, items, characters, and 3-D levels. There is also very rapid progress within machine learning in general in the moment, and in particular within the deep learning field and in methods with generative capabilities such as Generative Adversarial Networks. There is plenty of interesting and rewarding work to do in exploring how these new capabilities can be adapted to function with the particular constraints and affordances of game content.

References

  • [1] N. Shaker, J. Togelius, and M. J. Nelson, Procedural Content Generation in Games: A Textbook and an Overview of Current Research.   Springer, 2016.
  • [2] J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search-based procedural content generation: A taxonomy and survey,” Computational Intelligence and AI in Games, IEEE Transactions on, vol. 3, no. 3, pp. 172–186, 2011.
  • [3] A. M. Smith and M. Mateas, “Answer set programming for procedural content generation: A design space approach,” Computational Intelligence and AI in Games, IEEE Transactions on, vol. 3, no. 3, pp. 187–200, 2011.
  • [4] J. Montgomery, “Machine learning in 2016: what does nips2016 tell us about the field today?” 2016, https://blogs.royalsociety.org/in-verba/2016/12/22/machine-learning-in-2016-what-does-nips2016-tell-us-about-the-field-today/.
  • [5] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61, pp. 85–117, 2015.
  • [6] I. Goodfellow, Y. Bengio, and A. Courville, “Deep learning,” 2016, book in preparation for MIT Press. [Online]. Available: http://www.deeplearningbook.org
  • [7] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.
  • [8] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription,” preprint arXiv:1206.6392, 2012.
  • [9] S. Fine, Y. Singer, and N. Tishby, “The hierarchical hidden markov model: Analysis and applications,” Machine learning, vol. 32, no. 1, pp. 41–62, 1998.
  • [10] K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra, “Draw: A recurrent neural network for image generation,” preprint arXiv:1502.04623, 2015.
  • [11] G. Smith, J. Whitehead, and M. Mateas, “Tanagra: Reactive planning and constraint solving for mixed-initiative level design,” IEEE Transactions on Computational Intelligence and AI in Games, no. 99, 2011.
  • [12] G. N. Yannakakis and J. Togelius, “Experience-Driven Procedural Content Generation,” IEEE Transactions on Affective Computing, vol. 2, no. 3, pp. 147–161, 2011.
  • [13] J. Roberts and K. Chen, “Learning-based procedural content generation,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 7, no. 1, pp. 88–101, 2015.
  • [14] L.-Y. Wei, S. Lefebvre, V. Kwatra, and G. Turk, “State of the art in example-based texture synthesis,” in Eurographics 2009, State of the Art Report, EG-STAR.   Eurographics Association, 2009, pp. 93–117.
  • [15] Z. Ren, H. Yeh, and M. C. Lin, “Example-guided physically based modal sound synthesis,” ACM Transactions on Graphics (TOG), vol. 32, no. 1, p. 1, 2013.
  • [16] J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search-Based Procedural Content Generation,” in

    Applications of Evolutionary Computation

    .   Springer, 2010, pp. 141–150.
  • [17] E. J. Aarseth, Cybertext: Perspectives on ergodic literature.   JHU Press, 1997.
  • [18] N. Shaker, M. Shaker, and J. Togelius, “Ropossum: An authoring tool for designing, optimizing and solving cut the rope levels.” 2013.
  • [19] A. Liapis, G. N. Yannakakis, and J. Togelius, “Sentient sketchbook: Computer-aided game level authoring.” in FDG, 2013, pp. 213–220.
  • [20]

    C. Guillemot and O. Le Meur, “Image inpainting: Overview and recent advances,”

    IEEE signal processing magazine, vol. 31, no. 1, pp. 127–144, 2014.
  • [21] A. Summerville and M. Mateas, “Super Mario as a string: Platformer level generation via LSTMs,” DiGRA/FDG, 2016.
  • [22] R. Jain, A. Isaksen, C. Holmgård, and J. Togelius, “Autoencoders for Level Generation, Repair, and Recognition,” in ICCC, 2016.
  • [23] G. N. Yannakakis, P. Spronck, D. Loiacono, and E. André, “Player modeling,” in Dagstuhl Follow-Ups, vol. 6.   Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2013.
  • [24] M. Guzdial, N. Sturtevant, and B. Li, “Deep static and dynamic level analysis: A study on infinite mario,” in Third Experimental AI in Games Workshop, vol. 3, 2016.
  • [25] D. Braben and I. Bell, “Elite,” 1984.
  • [26] Hello Games, “No Man’s Sky,” 2016.
  • [27] A. Isaksen, D. Gopstein, J. Togelius, and A. Nealen, “Discovering Unique Game Variants,” in Computational Creativity and Games Workshop at ICCC, 2015.
  • [28] A. Summerville and M. Mateas, “Mystical tutor: A magic: The gathering design assistant via denoising sequence-to-sequence learning,” AIIDE, 2016.
  • [29] A. K. Hoover, J. Togelius, and G. N. Yannakis, “Composing video game levels with music metaphors through functional scaffolding,” Comp. Creativity & Games at ICCC, 2015.
  • [30] A. Summerville and M. Mateas, “Sampling hyrule: Sampling probabilistic machine learning for level generation,” AIIDE, 2015.
  • [31] M. Guzdial and M. Riedl, “Learning to blend computer game levels,” preprint arXiv:1603.02738, 2016.
  • [32] S. Dahlskog, J. Togelius, and M. J. Nelson, “Linear levels through n-grams,” in Intl.Academic MindTrek Conf., 2014.
  • [33] S. Dahlskog and J. Togelius, “Patterns as objectives for level generation,” in EvoApplications, 2013.
  • [34] A. J. Summerville, S. Philip, and M. Mateas, “MCMCTS PCG 4 SMB: Monte carlo tree search to guide platformer level generation,” AIIDE, 2015.
  • [35] K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” Evolutionary Computation, vol. 10, pp. 99–127, 2002.
  • [36] A. K. Hoover, P. A. Szerlip, and K. O. Stanley, “Functional scaffolding for composing additional musical voices,” Computer Music Journal, vol. 38, no. 4, pp. 80–99, 2014.
  • [37] A. Liapis, H. P. Martínez, J. Togelius, and G. N. Yannakakis, “Transforming exploratory creativity with delenox,.” in ICCC, 2013, pp. 56–63.
  • [38] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computing, 1997.
  • [39] S. Miyamoto and T. Tezuka, “Super mario bros. 2,” 1986.
  • [40] A. Summerville, M. Guzdial, M. Mateas, and M. Riedl, “Learning player tailored content from observation: Platformer level generation from video traces using LSTMs,” in AIIDE, 2016.
  • [41] W. of the Coast, “Magic: The Gathering,” 1993.
  • [42] M. Milewicz, RoboRosewater.   https://twitter.com/roborosewater?lang=en, 2016.
  • [43] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in CoRR, 2014.
  • [44] W. Ching, S. Zhang, and M. Ng, “On multi-dimensional markov chain models,” Pacific Journal of Optimization, vol. 3, no. 2, 2007.
  • [45] S. Snodgrass and S. Ontañón, “Experiments in map generation using Markov chains,” in FDG, vol. 14, 2014.
  • [46] S. Snodgrass and S. Ontañón, “Controllable procedural content generation via constrained multi-dimensional Markov chain sampling,” in IJCAI, 2016, pp. 780–786.
  • [47] S. Snodgrass and S. Ontanon, “Learning to generate video game maps using Markov models,” TCIAIG, 2016.
  • [48] S. Snodgrass and S. Ontañón, “A hierarchical MdMC approach to 2D video game map generation,” AIIDE, 2015.
  • [49] M. Gumin, “WaveFunctionCollapse,” GitHub repository, 2016, https://github.com/mxgmn/WaveFunctionCollapse.
  • [50] Freehold Games, “The Caves of Qud,” 2016.
  • [51] Arcadia-Clojure, “Proc Skater 2016,” 2016.
  • [52]

    P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in

    Intl. Conf. on Machine Learning.   ACM, 2008, pp. 1096–1103.
  • [53] S. Lee, A. Isaksen, C. Holmgård, and J. Togelius, “Predicting Resource Locations in Game Maps Using Deep Convolutional Neural Networks,” in AIIDE.   AAAI, 2016.
  • [54] B. Entertainment, “StarCraft II,” 2010.
  • [55] B. Horn, S. Dahlskog, N. Shaker, G. Smith, and J. Togelius, “A comparative evaluation of procedural level generators in the Mario AI framework,” FDG, 2014.
  • [56] N. Shaker and M. Abou-Zleikha, “Alone we can do so little, together we can do so much: A combinatorial approach for generating game content.” AIIDE, vol. 14, 2014.
  • [57] D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791, 1999.
  • [58] Nintendo, “The Legend of Zelda,” 1986.
  • [59]

    M. A. Turk and A. P. Pentland, “Face recognition using eigenfaces,” in

    CVPR.   IEEE, 1991, pp. 586–591.
  • [60] M. Guzdial and M. Riedl, “Game level generation from gameplay videos,” in AIIDE, 2016.
  • [61]

    K. Pulli, A. Baksheev, K. Kornyakov, and V. Eruhimov, “Real-time computer vision with opencv,”

    Commun. ACM, vol. 55, no. 6, pp. 61–69, Jun. 2012.
  • [62] E. Kalogerakis, S. Chaudhuri, D. Koller, and V. Koltun, “A probabilistic model for component-based shape synthesis,” ACM Transactions on Graphics, vol. 31, no. 4, 2014.
  • [63] M. Guzdial, B. Harrison, B. Li, and M. O. Riedl, “Crowdsourcing open interactive narrative,” in FDG, 2015.
  • [64] A. J. Summerville, M. Behrooz, M. Mateas, and A. Jhala, “The learning of Zelda: Data-driven learning of level topology,” PCG Workshop at FDG, 2015.
  • [65] S. Wright, “Correlation and causation,” 1921.
  • [66] B. Li, “Learning knowledge to support domain-independent narrative intelligence,” Ph.D. dissertation, Georgia Institute of Technology, 2015.
  • [67] P. Weyhrauch, “Guiding interactive fiction,” Ph.D. dissertation, Carnegie Mellon University, 1997.
  • [68] M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander, “Optics: ordering points to identify the clustering structure,” in SIGMOD, vol. 28, no. 2.   ACM, 1999, pp. 49–60.
  • [69] L. E. Baum and T. Petrie, “Statistical inference for probabilistic functions of finite state markov chains,” Math. Statistics, vol. 37, no. 6, pp. 1554–1563, 1966.
  • [70] J. Lafferty, A. McCallum, F. Pereira et al., “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in ICML, vol. 1, pp. 282–289.
  • [71] X. Liang, B. Zhuo, P. Li, and L. He, “CNN based texture synthesize with semantic segment,” preprint arXiv:1605.04731, 2016.
  • [72] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” 2014.
  • [73] B. Eisner, T. Rocktäschel, I. Augenstein, M. Bošnjak, and S. Riedel, “emoji2vec: Learning emoji representations from their description,” preprint arXiv:1609.08359, 2016.
  • [74] E. Abraham and A. Parrish, “Semantic arithmetic,” 2017.
  • [75] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” 5th International Conference on Learning Representations, 2017.
  • [76]

    Y. LeCun, C. Cortes, and C. J. Burges, “The MNIST database of handwritten digits,” 1998.

  • [77] A. Dosovitskiy, J. Tobias Springenberg, and T. Brox, “Learning to generate chairs with convolutional neural networks,” in IEEE CVPR, 2015, pp. 1538–1546.
  • [78] F. Pereira, P. Norvig, and A. Halevy, “The unreasonable effectiveness of data,” IEEE Intelligent Systems, vol. 24, no. undefined, pp. 8–12, 2009.
  • [79] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” arXiv:1611.07004, 2016.
  • [80] A. Karpathy, “The unreasonable effectiveness of recurrent neural networks,” 2015. [Online]. Available: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
  • [81] S. Miyamoto and T. Tezuka, “Super Mario Bros.” 1985.
  • [82] G. Yokoi and Y. Sakamoto, “Metroid,” 1986.
  • [83] D. McFerran, “More than a million super mario maker levels have been uploaded in a week,” 2015. [Online]. Available: http://www.nintendolife.com/news/2015/09/more_than_a_million_super_mario_maker_levels_have_been_uploaded_in_a_week
  • [84] A. J. Summerville, S. Snodgrass, M. Mateas, and Ontañón, “The VGLC: The video game level corpus.”
  • [85] D. Graff, J. Kong, K. Chen, and K. Maeda, “English gigaword,” Linguistic Data Consortium, 2003.
  • [86] Capcom, “Mega Man,” 1987.
  • [87] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR.   IEEE, 2009, pp. 248–255.
  • [88] F.-F. Li, R. Fergus, and P. Perona, “One-shot learning of object categories,” Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 594–611, 2006.
  • [89] R. Hunicke, M. LeBlanc, and R. Zubek, “Mda: A formal approach to game design and game research,” in AAAI Workshop on Challenges in Game AI, vol. 4, 2004, p. 1.
  • [90] J. Togelius, N. Shaker, S. Karakovskiy, and G. N. Yannakakis, “The Mario AI Championship 2009-2012,” AI Magazine, vol. 34, no. 3, pp. 89–92, 2013.
  • [91] S. L. Ahmed Khalifa, Diego Perez-Liebana and J. Togelius, “General video game level generation,” in Genetic and Evolutionary Computation, 2016.
  • [92] A. Canossa and G. Smith, “Towards a procedural evaluation technique: Metrics for level design,” FDG, 2015.
  • [93] S. Bruckner and M. E. Gröller, “Style transfer functions for illustrative volume rendering,” in Computer Graphics Forum, vol. 26, no. 3, 2007, pp. 715–724.
  • [94] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, “Image analogies,” in SIGGRAPH.   ACM, 2001, pp. 327–340.
  • [95] L. A. Gatys, A. S. Ecker, and M. Bethge, “A neural algorithm of artistic style,” arXiv:1508.06576, 2015.
  • [96] A. Mordvintsev, C. Olah, and M. Tyka, “Inceptionism: Going deeper into neural networks,” Google Research Blog. Retrieved June, vol. 20, 2015.
  • [97] P. Ribeiro, F. C. Pereira, B. Marques, B. Leitao, and A. Cardoso, “A model for creativity in creature generation.” in GAME-ON, 2003, p. 175.
  • [98] J. Permar and B. Magerko, “A conceptual blending approach to the generation of cognitive scripts for interactive narrative,” in AIIDE, 2013.
  • [99] S. Snodgrass and S. Ontañón, “An approach to domain transfer in procedural content generation of two-dimensional videogame levels,” in AIIDE, 2016.
  • [100] J. Gow and J. Corneli, “Towards generating novel games using conceptual blending,” in AIIDE, 2015.
  • [101] T. White, “Sampling generative networks: Notes on a few effective techniques,” preprint arXiv:1609.04468, 2016.
  • [102] M. Treanor, A. Zook, M. P. Eladhari, J. Togelius, G. Smith, M. Cook, T. Thompson, B. Magerko, J. Levine, and A. Smith, “AI-based game design patterns,” in FDG, 2015.