How to best design game content together with content generation algorithms is a hard and important question. A number of prototype systems for mixed-initiative design have been created to showcase ways in which humans and algorithms can design game content together [31, 22, 28, 6]. Many different modes of interaction have been devised, including those where the computer program provides suggestions to the human designer, evaluates their output, tests for playability, et cetera. However, all of these systems for AI-assisted game content generation are geared towards a single user.
In this paper, we address the challenge of AI-assisted collaborative
game content creation, that is, where multiple users interact with a procedural content generation system to create game content. Similar to an open source system like Wikipedia, there would be a central content repository, where anyone could make a contribution to the content. But additionally, the system should help users create content through various AI functionalities, such as providing suggestions, testing, and feedback. Most importantly, the initiative for design should be mixed. For example, the system might ask users to design specific types of content that it thinks should be designed, or to test or evaluate artifacts that others might have designed.
In this paper, we describe a prototype system for collaborative mixed-initiative level design, where users design levels for the puzzle game Baba is You. The system includes features for editing levels, automatically playtesting levels, helping design levels through an evolutionary algorithm, rating levels, and suggesting novel levels to design. All features are built around a central level archive which is structured like the map of elites from the MAP-Elites algorithm. We also report preliminary results from an informal user study, shining some light on how the system can be used.
Ii-a Procedural Content Generation
Procedural Content Generation is the process of using a computer program to create a content . These techniques have been used since the early days of computer games and still remain a popular technique today. PCG is typically divided based on the technology behind the generation process. Based on that, there are three main categories: Constructive techniques , Search-Based techniques 
, and Machine Learning techniques. Search based approaches are very common in academia for their generality and ease of use. These techniques use a search/optimization algorithm to search the space of potential content for a good artifacts. A fitness function is defined in order to guide the search. Previously, researchers have used it to generate bullet patterns , levels [21, 22], flowers , etc.
With the advancement in quality-diversity search based methods , game researchers have started to focus on using it in more projects . Quality-diversity techniques are search based techniques that try to generate a set of diverse solutions with a high quality. A well-known example is Map-Elites , an evolutionary algorithm that uses a multi-dimensional map instead of a population to maintain its results. This map is constructed by dividing the solution space into a group of cells based on defined behavior characterstics. Any new solution found will have to evaluated for its location in the map and placed in the correct cell. In most situations, solutions in the same cell compete within the population and only the fittest individual survives. Because of the map maintaince and the cell competition, Map-Elites can guarantee a map of diverse and high quality solutions, after a finite number of iterations. These properties of Map-Elites have been used in other research topics throughout academia [1, 2, 21, 20, 11, 10, 7].
Ii-B AI Mixed Initiative in Games
AI mixed initiative process is human and AI collaboration together on a creative task. This task can include painting [17, 8, 19], designing shoes , composing music , etc. In this paper, we are focusing only on collaboration in the context of level design.
Researchers have been experimenting with different approaches to find the best way of collaboration between human and machines. One of the earliest approaches is to use search based evolutionary algorithms to suggest content edits for the user based on a specific fitness function [31, 30, 28]
. A drawback of using evolutionary algorithms that it doesn’t provide a lot of diversity between the suggestions which become very limiting during collaboration if the suggestion are not the best. Similarly, machine learning methods suffer from a similar problem, as the trained model usually suggests one solution. The advantage of using machine learning techniques is the ability to modify models using active learning to cater more to the user’s personality and style[16, 15].
Liapis et al. has tried to solve this problem by introducing several different suggestions that improve different fitness functions during the design of 2D strategy maps . Unfortunately, the system was evaluating multiple different functions making it slow and hard to scale. With the rise of the quality diversity techniques , Alvarez et al. started using constrained Map-Elites  towards designing a collaborative tool to designing top down dungeons [1, 2].
Ii-C Open source projects
Open source collaborative projects are projects that allow users to contribute specific content to create the improvement of a system. The loose structure of open source projects may sometimes not help users engage with it. As such, they only join when necessary and have time. Brito et al. suggests gamifing the process in order to have more engagement with the sytem and provided a conceptual framework on how to do so in crowd sourcing projects . Several research projects gamified the process in order to guarantee user participation and collect more data [13, 9, 12].
In games, this is not the case; most games are closed source. Developers spend a huge amount of money to develop a game over the course of multiple years without sharing the data. Very few commercial developers allow users to build their own content and share ideas through their system such as Super Mario Maker (Nintendo, 2015), Skyrim (Bethesda, 2011), Time Fcuk (Edmund McMillen, 2009), etc. Most of these games cannot be accessed without purchase and the content is not used outside of engagement. From a research perspective, very few projects use open sourcing to generate more game content. Barros et al.’s work  is known for using crowd-sourced data for game development.
Ii-D Baba is You
Baba is You (Arvi “Hempuli” Teikari, 2019) is a puzzle game where players can manipulate the rules of a level and properties of the game objects through Sokoban-like movements of pushing word blocks around the map. Since the mechanics act as constraints and solutions for the player, the game provides an interesting study for procedural level generation where game levels and game rules are intertwined.
Every ‘Baba is You’ level must have a win condition and a controllable character that can reach the defined win condition in order to successfully solve the level. These two rules are defined by ‘X-IS-WIN’ and ‘X-IS-YOU’ respectively - where X is any type of game object that can exist in the game level. Figure 1 shows two simple levels from ‘Baba is You’. Figure 0(a) shows a simple level where the player controls the ‘Baba’ object (with the ‘Baba-IS-YOU’ rule is currently active) and tries to reach the ‘Flag’ object to win the level (with the ‘Flag-IS-WIN’ rule is active at the start of the level). The rules do not have to be initially made at the start of the game and can be manipulated at any point while the player is attempting to solve the level. Figure 0(b) shows the same level as figure 0(a) but the rule ‘Flag-IS-WIN’ is not satisfied at the beginning. The player will need to push the word ‘Flag’ to complete the sentence ’Flag-IS-WIN’ - thereby activating that rule so they can win the level.
Rules for the level are defined by statements readable as English text and read from up-down and left-right. So ’X-IS-YOU’ would be interpreted as a valid rule, but not ’YOU-IS-X.’ As such, all rules must take one of these 3 formats:
X-IS-(KEYWORD) where KEYWORD belongs to a word in the keyword word class that manipulates the property of the game object class X (i.e. ’WIN’, ’YOU’, ’MOVE’, etc.)
X-IS-X a reflexive rule stating that the game object class ’X’ cannot be changed to another game object class.
X-IS-Y a transformative rule changing all game objects of class X into game objects of class Y.
The objects themselves ultimately do not matter in the game. For example, a player can control Baba, Keke, the walls, or any other object or multiple objects at once in the game so long as it can be represented by a physical object as a word and the rule ’X-IS-YOU’ is connected where X represents the object that the player is currently controlling. This applies to all rules created initially before the player starts to solve the level and any rules that are created or removed while the player is solving the level. Game object classes that are affected by any rule have their properties updated accordingly. For example, creating the rule ’KEKE-IS-MOVE’ within the game would make all KEKE objects start autonomously moving within the game after each action is taken thereafter. If the rule is removed after a future action is taken, all KEKE objects would stop moving subsequently.
Iii Baba is Y’all
Baba is Y’all 111http://game.engineering.nyu.edu/babaisyall/ is an Open Source Mixed initiative tool that allows users to build their own Baba is You levels and save them online. The base functionality of editing, saving and playing others’ levels is similar to Super Mario Maker (Nintendo, 2015), though of course not as advanced. The system implements the jam version 222https://hempuli.itch.io/baba-is-you of Baba is You and not the more full-fledged commercial version as the jam version has enough variety in game mechanics that allows the users to build different levels. However, Baba is Y’all is not just an ordinary level editor that saves the levels online. The level archive is structured as a quality-diversity evolutionary algorithm , specifically MAP-Elites . The overall process of users editing levels and building on each others’ level can be seen as an instantiation of this algorithm, which seeks to improve the overall quality of the saved levels while maintaining the diversity of the levels. Every time a user submits a new level, the system saves it with mechanically similar levels where they will be rated with respect to each other. Later, when a user is trying to play a new level that is different from what they have previously play, the system can provide a different level with sufficient quality. The system also pushes the users towards designing levels that lack a specific mechanic so players can always find something new and interesting to play. Lastly, the system provides a simple evolutionary algorithm as part of the creation process to help inspire designers.
To make sure the system is simple and easy to access by everyone, we decided to design it as a web application where anyone with a broswer can access it. We modularized the system into four main modules, to make it easy to maintain and update at any time.
Game Module: the core game framework where the user can test game levels by solving it themselves or by using an AI solver.
Editor Module: the level editor where the user can design Baba is You levels and submit it to our map module.
Generator Module: provides the user with access to an evolutionary algorithm that can help them generate or modify levels.
Map Module: responsible for maintaining the different game levels and suggests new levels to play or make.
These modules communicate with each other based on the sequence of actions the user makes while navigating the site.
Iii-a Game Module
The game module is responsible for simulating a Baba is You level. It also allows users to test the playability of levels either by directly playing through the level themselves or by allowing a solver agent to attempt to solve it. This component is used in the Editor module, the Generator module, and the rating page. After a level has been saved to the database, users can attempt to solve the level in fewer moves than what was originally solved by the creator.
The module receives an input level in the form of a 2D array of characters where each character represents a different game sprite. Figure 2 shows an example input ascii level and its corresponding game level with different game sprites. Game sprites are divided into two main different classes: object class and keyword class. Object class are the actual game object that game rules manupilate and they are the sprites that can modify the game state, while keyword class defines the actual rules of the level. For example, figure 2 shows five different keyword class sprites (BABA, IS, YOU, FLAG, and WIN). These sprites are arranged in two rules: ‘BABA IS YOU’ allowing the player to control all the Baba objects and ‘FLAG IS WIN’ indicating that reaching any flag object will make the player win the level. The system has a total of 32 different sprites: 11 object classes and 21 keyword classes.
Because the game allows rule manipulation, object classes mean little in the game except for providing variety of objects for rules to affect and/or aesthetic pleasure. Because the game rules are always changing, the system keeps track at every state of all the active rules. Once the win condition has been met, the game module records the current solution, the active rules at the start of the level, and the active rules when the solution has been reached. These properties are saved to be used and interpreted by the Map module. The rules activated are used as the level’s characteristic feature representation and saved as a chromosome to the MAP-Elites matrix.
The game module provides an AI solver called ’KEKE’ (based on one of the characters traditionally used as an autonomous ’NPC’ in the game). KEKE uses a best-first search tree search algorithm that tries to solve the input level. The branching space is based on the five possible inputs a player can do within the game: move left, move right, move up, move down, and do nothing. The algorithm uses a heuristic function based on a weighted average of the Manhattan distance to the centroids for 3 different groups: keyword objects, objects associated with the ’WIN’ rule, and objects associated with the ’PUSH’ rule. These were chosen based on their critical importance for the user solving the level - as winning objects are required to complete the level, keyword objects allow for manipulation of active rules, and pushable objects can directly and indirectly affect the layout of a level map and therefore the accessability of player objects to reach winning objects. The heuristic function is represented by the following equation:
where is the final heuristic value for placement in the priority queue, is the minimum Manhatttan distance from any player object to the nearest winnable object, is the minimum Manhatttan distance from any player object to the nearest word sprite, and is the minimum Manhatttan distance from any player object to the nearest pushable object.
Since the system runs the solver on the user-end of the site, the solver agent has a limit of 10000 search steps to avoid computational strain to the user’s machine. As such, the agent is unable to solve levels with long or complex solutions due to its limiting search space. The average limit for solution steps for KEKE has not been tested yet since the focus of this project was to allow mixed-initiative collaboration with level design.
Iii-B Editor Module
The editor module of the system allows human users to create their own Baba is You levels in the same vain of Super Mario Maker (Nintendo, 2015). Figure 3 shows the editor window that is available for the user. The user can place and erase any game sprite or keyword at any location on the map using the provided tools. As a basis, the user can start modifying either a blank map, a basic map (a map with X-IS-YOU and Y-IS-WIN rules already placed with X and Y objects), or an elite level provided by the Map Module. Similar to Mario Maker, the created levels can only be submitted after they are tested by the human player or the AI agent to check for solvability. For testing the level, the editor module sends the level information to the game modulto allow the user to test it.
Our editor also nudges the player towards creating levels with a certain mechanical goals (as seen in figure 3). The mechanical goals are defined based on the presence of certain active rules in the game either at the start state or at the winning state of the current level. The mechanical goals are written in form X-IS-KEYWORD such as X-IS-MOVE or X-IS-STOP. The X is never specified since any object in Baba is You can be used for anything while the KEYWORD is pre-defined. A list of all the different goals will be discussed later in section III-D. Any combination of goals can be represented in the level - and unmade combinations will be presented to the user in order of simiplicity. Through this process, the user and the system help each other to develop a large and diverse level pool. The user provides the system with new, novel levels, while the system guides the user by telling them what levels are not existing in the current pool to encourage originality and diversity. At any point, if the designer struggles with a concept while designing a level, they can transfer their current level to the Generator module to automatically modify the level for them. After mutating and evolving their level through the Generator, they can transfer their level back to the Editor module to do the final edits manually before submitting it to the Map module.
Iii-C Generator Module
The Generator module is a procedural content level generator. In particularly, it is an evolutionary level generator using a fitness function based on a version of tile-pattern Kullback-Liebler Divergence (ETPKLDiv) algorithm . Figure 4 shows the current interface used by the evolver. As mentioned before, the Evolver module can be accessed from the Editor module at any time and can export its output back to the Editor module. With this transfer process, the generation process loses its pure procedurally generated quality. However, this process encourages mixed-initiative interaction between the algorithm and the user. This also creates an ”algorithm-first” approach to creating levels, where the levels are generated towards a fitness value and then edited by the user later. The evolver interface provides the user with multiple customizations such as the initialization method, stopping criteria, evolution pausing, and an application of a mutation function allowing manual user control. With these features, the user is not directly changing the evolution process itself, but instead guiding and limiting the algorithm towards generating the level they want.
The original ETPKLDiv algorithm uses a 1+1 evolution strategy, a.k.a. a hillclimber, to improve the similarity between the current evolved chromosomes and a reference level. The algorithm uses a sliding window of a fixed size to calculate the probability of each tile configuration (called tile patterns) in both the reference level and the evolved level and tries to minimize the Kullback-Liebler Divergence between them. For this project, we used a window size of 3x3. This was to maximize the probability of generating initial rules for a level, since rules in Baba is You are made up of 3 tiles.
In our project, we used 2+2 evolution strategy instead of 1+1 used by Lucas and Volz to allow slightly more diversity in the population. We also modified the fitness function to allow it to compare with more than one level. The fitness value also includes the playability of the level (), the ratio of empty tiles to total level tiles (), and the ratio of unnecessary objects (). The final fitness equation for a level is as follows:
The value is the percentage of unnecessary objects in the map. The value can be calculated as follow:
where is the number of objects sprites initialized in the level without a related object-word sprite and is the total number of object sprites initialized in the level. This value is implemented in order to prevent noise within the level due to having object tiles that cannot be manipulated in any way or have relevancy to the level. A human-made level may include these ”useless” tiles for aesthetic purposes or to give the level a theme - similar to the original ’Baba is You’ levels. However, the PCG algorithm optimizes towards efficiency and minimalist levels, therefore ignoring the subjective aspect of a level’s quality. This factor can be explored further in future work.
The value is a binary constraint value that determines whether a level is potentially winnable or not. The value can be calculated as follows:
This is to ensure any levels that are absolutely impossible to play or win are penalized in the population and less likely to be mutated and evolved from in future generations. We used a simple playability constraint check instead of checking for playability using the solver because the solver take time to check for playability. Also all playable levels by the solver usually end up being easy levels due to the limited search space we are given for the best first algorithm.
The value is the ratio of empty space tiles to all of the tiles in the level. The equation can be calculated as follows:
where is the number of empty spaces in the level and is the total number of tiles found in the level. The value is multiplied with a value of 0.1 to avoid heavy penalization for having any empty spaces in a level and to prevent encouragement for levels to mutate towards populating the level with an overabundance of similar tiles in order to eliminate any empty space. This instead causes the algorithm to favor smaller and more compact levels over levels with a large area.
The Generator module is not run as a back-end process to find more levels, instead it has to be done manually by the user. This is done due to the fact that some generated levels cannot be solved without human input. One might wonder why not generate a huge corpus of levels and ask the users later to test them for the system. This could result in the system generating a multitude of levels that are either impossible to solve or are solvable but not subjectively ”good” levels - levels the user would not find pleasing or enjoyable. This overabundance of ”garbage” levels could lead to a waste of memory and a waste human resources. By allowing the user direct control over which levels are submitted from the generation algorithm, it still guarantees that the levels are solable and with sufficient quality and promote using the tool in a mixed-initiative approach.
Iii-D Map module
The Map module is the core module of the system. It provides three main functionalities: level archiving, suggesting levels to create, and suggesting levels to be played and rated. In order to achieve all of these functionalities, we need an optimal way of representing the relationship between the created levels. A way to measure similarity between levels so we can find the best levels between them and easily discover new levels every time. We ended up using the same structure as the Map-Elites algorithm  in order to guarantee quality and diversity between the levels.
|X-IS-X||objects of class X cannot be changed to another class|
|X-IS-Y||objects of class X will transform to class Y|
|X-IS-PUSH||X can be pushed|
|X-IS-MOVE||X will autonomously move|
|X-IS-STOP||X will prevent the player from passing through it|
|X-IS-KILL||X will kill the player on contact|
|X-IS-SINK||X will destroy any object on contact|
|X-IS-[PAIR]||both rules ’X-IS-HOT’ and ’X-IS-MELT’ are present|
|X,Y-IS-YOU||two distinct objects classes are controlled by the player|
When a level is submitted to be archived, the system uses the the list of active rules at the start and the end of the level as behavior characteristic for the input level to determine its location in the map. There are 9 different rules checked for in each level - based on the possible rule mechanics that can be made in the Game module system. Table I shows the full list of possible rules. Since these rules can be active at the beginning or at the end, it makes the number of behavior characteristics equal to 18 instead of 9 which proivde us with a map of cells.
In this project we are using a multi population per each cell of the Map-Elites similar to the constrained Map-Elites . In our project, we are using two tables/populations per cell named: rated table and elite table. The ’Rated’ table contains all the levels that are created either by humans, the system, or both. On the other hand, the ’Elite’ table only contains one level per cell. This level must have an average rating score of at least 3.5 out of 5 based on 5 different ratings. The elite can be replaced if a new level appears with higher average rating than the previous one.
A rating for a single level is determined by comparison to another level within the entire ’Rated’ table population. The user must determine the better level based on two qualities: hardness and design. A level that is considered ’harder’ represents that the solution search space for the level takes longer to arrive at - and therefore is more challenging to the user. A level that is considered to have ’better design’ represents that the level is more visually pleasing and elegant with its map representation - a quality that is hard to generate automatically with AI. The minimum score rating for a level is 1 and adds 2 for each comparison category it wins - resulting in a maximum score of 5.
The user at anytime can check the state of the MAP-Elites population by visiting the level matrix page. The level matrix shows 50 randomly chosen cells that can be filtered and sorted according to a certain behavior characteristic. Figure 5 shows the level map interface provided to the user. The retrived levels are sorted from simplest to hardest and the set can be reshuffled to retrieve new levels at anytime. From here, the user can choose between playing a level that has already been made to find a faster solution, editing a submitted level to either create a better level for a MAP-Elites cell population or contribute to a different cell’s population by altering the rules, or selecting an unpopulated cell to make a new novel level representation. Our main intention is to encourage users to create new levels and contribute to filling the cells of the MAP-Elites matrix.
The study results were collected 32 hours after the announcement release of the system. Out of the 262,144 cells in the MAP-Elites matrix, 38 had at least one level. 58 levels were made in total - 12 were user-only created levels, 11 were evolver-only created levels, and 35 were evolver-user collaboration levels. 18 users had accounts registered on the site. The average number of solution steps for the levels was 27.93 steps.
An average of 2.924 rules were used in the level generation. A full distribution of the rules activated in each level can be shown in Figure 6. By looking at the figure, we can see even with the small amount of participants we had, the system was able to cover all the different mechanics that the game can offer. We can’t guarantee that this high coverage was due to the system provided goals during the level creation, but it is a good indication that the system is working and helping the users to explore new level ideas. Figure 7 shows some examples of the different levels submitted to the sytem.
A survey was also made available to participants through the site. We recieved two responses. One participant used the evolve tool, one participant made an account on the site, and no participants rated levels. One participant mentioned as an additional comment that they were unsure how to save a level they made.
In the time span of data collection for this paper, the MAP-Elites matrix was 0.01% filled. The lack of user response was partly due to the short time window given, which in turn was due to the deadline for this conference. However, there is also a lack of a strong incentive for creating the levels, and the the learning curve for interacting with the system is steeper than it should be. The amount of unmade levels and prompts for the user to participate in could also be overwhelming. Regardless, the levels generated were fairly distributed with the rules - giving cause to believe that the level prompts given by the system were at least attempted by the userbase.
Typically with crowd-sourced tasks, users participating are given an incentive to complete the task. Whether it be small monetary incentives, social recognition within the community, or personal benefit from contributing to a bigger cause, users need some reason to participate. We hypothesized that users would participate in this study based on the reason of contribution to a bigger cause and contributing to a large pool of levels along with other users and evolutionary AI. However, the user may have been overwhelmed with how many tasks were needed to be completed. It might therefore make sense to add elements of gamification to the system to motivate users.
Baba is Y’all’s target audience was aimed towards users who are natural problem solvers, creative, and interested in puzzle games and content generation. This group of people would be the most motivated to contribute to task-oriented generation of levels - and thus be able to populate the cells of the MAP-Elites matrix. Other users who find themselves outside of this niche can still contribute to this generating process by evaluating levels and providing feedback to the creators and the system itself - improving the quality of the level creation overall. The end result is a full map of levels of ”good” quality - made by both the creators and the evaluators.
Baba is Y’all is an ongoing level creation and generation prototype system for creating Baba is You levels based on rule representations using collaborative mixed-initiative techniques between human users and evolutionary algorithms. The system prompts the user to create levels based on what it is missing in its current level set so as to create a exhaustive library of levels with specific mechanic behaviors. While there are still many more levels to be made in the creation space, the application serves as a proof of concept for task-oriented level design aided, guided, and organized by an AI system.
The project is the beginning of many more explorations with this system and collaborative mixed-initiative level design as a concept. Ideally, the individual modules of the Baba is Y’all system can be swapped out to incorporate other variations of generation algorithms, level editors, and/or mapping organization algorithms. We would like to expand this system to use other popular games for collaborative level design. Some examples include designing Zelda-like dungeons needing specific mechanics or tool combinations in order to be completed or designing platforming levels that require specific combinations of actions to reach the goal. We would also like to experiment with altering level space organization, level editing user interface, and different procedural content generation and evolutionary algorithms. For example, it would be useful if the designer could change the fitness function, with a suggested fitness function being to satisfy the mechanics suggested by the system.
To speed up the level generation process, we would like to run a back-end system parallel to the web-based system that can quickly generate, solve, and upload new novel levels to fill in the MAP-Elites cells. In this project, a level was only submitted if and only if the level was provided with a solution. For a user-excluded generator, only a solver agent could be used. The levels would also need to include a diversity metric and a dynamically updating input source from the user created levels in order to maintain visually appealing but solvable levels that satisfy filling the population of cells in the MAP-Elites matrix.
For user-end features we would like to examine collaborations between multiple users and multiple types of evolutionary algorithms all at once to create levels. This would broaden the scope and possibilities of level design and development even further to allow more creativity and evolutionary progress within the system. We would also allow the option for users to personally save their levels to their account and search for other user-specific levels. This will hopefully make the site more engaging for the user - with less prioritization on population of cells as was originally designed.
Ahmed Khalifa acknowledges the financial support from the NSF Award number 1717324 - “RI: Small: General Intelligence through Algorithm Invention and Selection.”.
-  (2019) Empowering quality diversity in dungeon design with interactive constrained map-elites. In 2019 IEEE Conference on Games (CoG), pp. 1–8. Cited by: §II-A, §II-B.
-  (2020) Interactive constrained map-elites analysis and evaluation of the expressiveness of the feature dimensions. arXiv preprint arXiv:2003.03377. Cited by: §II-A, §II-B.
-  (2018) Who killed albert einstein? from open data to murder mystery games. IEEE Transactions on Games 11 (1), pp. 79–89. Cited by: §II-C.
-  (2018) Deep interactive evolution. In International Conference on Computational Intelligence in Music, Sound, Art and Design, pp. 267–282. Cited by: §II-B.
-  (2015) Towards a framework for gamification design on crowdsourcing systems: the game approach. In 2015 12th International Conference on Information Technology-New Generations, pp. 445–450. Cited by: §II-C.
-  (2013) A mixed-initiative tool for designing level progressions in games. In Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 377–386. Cited by: §I.
-  (2020) Mech-elites: illuminating the mechanic space of gvgai. arXiv preprint arXiv:2002.04733. Cited by: §II-A.
-  (2015) Drawing apprentice: an enactive co-creative agent for artistic collaboration. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition, pp. 185–186. Cited by: §II-B.
-  (2012) Increased diels-alderase activity through backbone remodeling guided by foldit players. Nature biotechnology 30 (2), pp. 190. Cited by: §II-C.
-  (2019) Mapping hearthstone deck spaces through map-elites with sliding boundaries. In Proceedings of The Genetic and Evolutionary Computation Conference, pp. 161–169. Cited by: §II-A.
-  (2019) Covariance matrix adaptation for the rapid illumination of behavior space. arXiv preprint arXiv:1912.02400. Cited by: §II-A.
-  (2014) Game of words: tagging places through crowdsourcing on public displays. In Proceedings of the 2014 conference on Designing interactive systems, pp. 705–714. Cited by: §II-C.
-  (2014) The cure: design and evaluation of a crowdsourcing game for gene selection for breast cancer survival prediction. JMIR Serious Games 2 (2), pp. e7. Cited by: §II-C.
-  (2019) Procedural content generation through quality diversity. In 2019 IEEE Conference on Games (CoG), pp. 1–8. Cited by: §II-A.
-  (2019) Friend, collaborator, student, manager: how design of an ai-driven game level editor affects creators. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13. Cited by: §II-B.
-  (2018) Co-creative level design via machine learning. arXiv preprint arXiv:1809.09420. Cited by: §II-B.
-  (2017) A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477. Cited by: §II-B.
-  (2009) Automatic content generation in the galactic arms race video game. IEEE Transactions on Computational Intelligence and AI in Games 1 (4), pp. 245–263. Cited by: §II-A.
-  (2018) Creative arcs in improvised human-computer embodied performances. In Proceedings of the 13th International Conference on the Foundations of Digital Games, pp. 1–6. Cited by: §II-B.
-  (2019) Intentional computational level design. In Proceedings of The Genetic and Evolutionary Computation Conference, pp. 796–803. Cited by: §II-A.
-  (2018) Talakat: bullet hell generation through constrained map-elites. In Proceedings of The Genetic and Evolutionary Computation Conference, pp. 1047–1054. Cited by: §II-A, §II-A, §II-B, §III-D.
-  (2013) Sentient sketchbook: computer-assisted game level authoring. Cited by: §I, §II-A, §II-B.
-  (2019) Tile pattern kl-divergence for analysing and evolving game levels. Proceedings of the Genetic and Evolutionary Computation Conference. External Links: Cited by: §III-C, §III-C.
-  (2016) Ai duet. Experiments with Google. See, https://experiments. withgoogle. com/ai/ai-due t. Cited by: §II-B.
-  (2015) Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909. Cited by: §II-A, §III-D, §III.
-  (2016) Quality diversity: a new frontier for evolutionary computation. Frontiers in Robotics and AI 3, pp. 40. Cited by: §II-A, §II-B, §III.
-  (2015) Petalz: search-based procedural content generation for the casual gamer. IEEE Transactions on Computational Intelligence and AI in Games 8 (3), pp. 244–255. Cited by: §II-A.
Ropossum: an authoring tool for designing, optimizing and solving cut the rope levels.
Ninth Artificial Intelligence and Interactive Digital Entertainment Conference, Cited by: §I, §II-B.
-  (2016) Procedural content generation in games. Springer. Cited by: §II-A.
-  (2010) Launchpad: a rhythm-based level generator for 2-d platformers. IEEE Transactions on computational intelligence and AI in games 3 (1), pp. 1–16. Cited by: §II-B.
-  (2010) Tanagra: a mixed-initiative level design tool. In Proceedings of the Fifth International Conference on the Foundations of Digital Games, pp. 209–216. Cited by: §I, §II-B.
-  (2018) Procedural content generation via machine learning (pcgml). IEEE Transactions on Games 10 (3), pp. 257–270. Cited by: §II-A.
-  (2011) Search-based procedural content generation: a taxonomy and survey. IEEE Transactions on Computational Intelligence and AI in Games 3 (3), pp. 172–186. Cited by: §II-A.