Your Gameplay Says it All: Modelling Motivation in Tom Clancy's The Division

by   David Melhart, et al.
University of Malta
Ubisoft Massive

Is it possible to predict the motivation of players just by observing their gameplay data? Even if so, how should we measure motivation in the first place? To address the above questions, on the one end, we collect a large dataset of gameplay data from players of the popular game Tom Clancy's The Division (Ubisoft, 2016). On the other end we ask them to report their levels of competence, autonomy, relatedness and presence using the in-house designed Ubisoft Perceived Experience Questionnaire. After processing the survey responses in an ordinal fashion we employ preference learning methods, based on support vector machines, to infer the mapping between gameplay and the four motivation factors. Our key findings suggest that gameplay features are strong predictors of player motivation as the obtained models reach accuracies of near certainty, in particular, from 93


page 4

page 7

page 9


Survey about cyberattack protection motivation in higher education: Academics at Slovenian universities, 2017

This paper reports on a study aiming to explore factors associated with ...

I Feel I Feel You: A Theory of Mind Experiment in Games

In this study into the player's emotional theory of mind of gameplaying ...

Towards General Models of Player Experience: A Study Within Genres

To which degree can abstract gameplay metrics capture the player experie...

Exploring Apprenticeship Learning for Player Modelling in Interactive Narratives

In this paper we present an early Apprenticeship Learning approach to mi...

From Pixels to Affect: A Study on Games and Player Experience

Is it possible to predict the affect of a user just by observing her beh...

Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch

Is it possible to predict moment-to-moment gameplay engagement based sol...

Influence des mécanismes dissociés de ludifications sur l'apprentissage en support numérique de la lecture en classe primaire

The introduction of serious games as pedagogical supports in the field o...

I Introduction

The holy grail for game design and development is to create engaging experiences for the players. Players who are self-motivated to return to the game and keep playing are critical for a game’s success [1, 2, 3]. The central role of motivation for the design of games, and the experiences they elicit, has been highlighted by a growing number of studies which adopt psychological theories of motivation within games [4, 5, 6, 7, 8]. Such studies, however, follow a top-down integration of phenomenological models of motivation, which aim to identify and explain stereotypical player behaviour. Over the last decade, games user research and industry-based game testing has shifted its focus towards quantitative approaches based on player analytics [9, 10] with the aim to shed more light onto the understanding of player behaviour and experience. These approaches focus mainly on either clustering players based on their behavioural patterns or predicting objectively-defined aspects of their gameplay behaviour for monetization purposes (e.g. churn prediction) [3]. In that regard, the majority of approaches that aim to capture aspects of player experience (such as engagement or motivation) based on player analytics remain qualitative, given the complexity of measuring subjective notions of user experience in games [11].

Motivated by the lack of quantitative studies on the relationship between motivation and play, in this paper we introduce a data-driven player modelling approach [12] by assuming there is an unknown underlying function between what a player does in the game behaviourally—as manifested through her gameplay data—and her motivation. In particular, we assume that solely behavioural data from a player’s gameplay would yield accurate predictors of motivation in games. To define motivation we rely theoretically on Self Determination Theory [13]—a well-established positive psychological framework for motivation—and examine four core factors: competence, autonomy, relatedness and presence, which is often associated with the theory in the domain of videogames [1].

To study motivation quantitatively we are grounded on recent developments in motivation measurement tools, namely the Ubisoft Perceived Experience Questionnaire (UPEQ) [14], which was developed as a game-specific tool observing player motivation. To infer the relationship between player motivation and gameplay we collect data from more than 400 players of Tom Clancy’s The Division (Ubisoft, 2016). We process and aggregate this data and collect surveys on the players’ motivation in relation to the game independently. We use the UPEQ questionnaire to measure players’ general levels of competence, autonomy, relatedness and presence in the game. Given the subjective nature of the reported notions we adopt the second-order data processing approach [15] and we process the reported UPEQ Likert-scale values of the players as ordinal data, and not as scores. We then apply simple statistical rank-based models and preference learning [16]

methods based on support vector machines to infer the function between gameplay and reported factors of motivation. Our results suggest that factors of reported motivation can be predicted with high accuracy just relying on a few high-level gameplay features. In particular, the nonlinear machine learned models manage to predict the four motivation factors of unseen players with at least

accuracy; the best model reaches an accuracy of for presence. The obtained results add to the existing evidence for the benefits of ordinal data processing on subjectively-defined notions [15] and they also validate that motivation can be captured qualitatively with supreme accuracy in the examined game only based on behavioural high-level data of playing.

This paper is novel in a number of ways. First, this is the first time player motivation is modelled computationally only through gameplay data in games. Second, we introduce a second-order [15] methodology for treating Likert-scale scores which are used frequently in game testing and games user research at large. The ordinal approach we adopt compares the subjective scores of all players with each other and hence generates combinatorially very large datasets based only on small sets of participants. The approach is also effective in eliminating reporting biases of respondents, thereby better approximating the ground truth of reported motivation. Third, we model aspects of player motivation using preference learning based solely on a small number of key gameplay features. Finally, for the first time we evaluate the method in Tom Clancy’s The Division (Ubisoft, 2016) on over 400 players and the predictive capacity of the motivation models for this game reach near certainty (i.e., over of accuracy).

Ii Background: Measuring and Modelling Motivation

This section gives an overview of related research on player modelling and motivation studies in games (Section II-A) and then it introduces the fundamental principles of Self Determination Theory (SDT) and the notion of presence within the framework of UPEQ (Section II-B). The section ends with a discussion on the strengths of preference learning for modelling subjectively-defined psychological constructs such as motivation (Section II-C).

Ii-a Player Modelling and Player Motivation

Player modelling is one of the primary foci of AI research in the field of videogames [3] and generally concerned with the prediction of player behaviour or other cognitive and affective processes. As games offer a complex yet constrained environment for modelling dynamic interactions, user modelling is also gaining more and more traction among industry practitioners, who use analytics-based player modelling methods to inform game design, development, and stakeholder decisions [11]. Player models, among many uses, can inform and shape the monetization strategy of a game, they can be directly applied as drivers of personalised content generation [17, 18], and they can equip agents for believable user testing [19].

Among the many aims of player modelling within game development, one can distinguish the two core tasks of clustering and prediction [3]

. The former is based on unsupervised learning methods with the aim to cluster players within groups of common behavioural pattern; approaches include k-means, self-organising maps

[9], matrix factorisation and archetypal analysis [20, 21], and sequence mining [22, 23, 10]

. The latter uses supervised learning approaches to predict patterns of playing such as completion time

[24] and churn [25, 26, 27]. When predicting something about the player a distinction can be made between the prediction of the player’s behaviour (i.e., what would a player do?) [28] and the prediction of aspects of the game experience (i.e., what would a player feel?) [3], often using behavioural data but also other modalities of player input such as physiological data [29, 30], independently or in a multimodal fashion [31, 32].

The literature involving theories of motivation as applied to games concerns primarily theoretical models that inform the design of experimental protocols instead of defining target outputs for predictive modelling. Indicatively, Borbora et al.

[33], use Yee’s [34] player motivation typology to enhance churn prediction, while Shim et al. [35] use motivational survey data to model player enjoyment. Birk et al. [36], also incorporated motivational survey data along with top-down personality and player profiles to model enjoyment and effort. Although Birk et al. include high-level telemetry of progression (game stage) as a control variable in their experiments, they rely mainly on survey data. In contrast to the above studies—which apply motivational survey data as top-down domain knowledge that forms the input of enjoyment models—we instead focus on predicting motivation based on game metrics alone. Similarly to this study, Canossa et al. [5] applied regression models of telemetry to predict psychological survey data; however, that study was not aimed at player motivation modelling but instead used the Reiss Motivational Profile [37] as a tool for personality profiling based on personal motivational drives. In this paper we focus on the structure of motivation by considering four of its core manifestations but our models do not rely on personality profiles of the players. Instead, we only focus on and try to infer the unknown mapping between gameplay features and motivation.

Ii-B From Theory to Measures of Motivation

The self-determination theory (SDT) is a well-established positive psychology theory of the facilitation of motivation based on the work of Deci and Ryan [38], which has been adopted to a wide variety of domains, including education [39], job satisfaction [40], parenting [41], health and exercise [42], and videogames [43, 44, 1, 8]. The core theory was developed to contrast earlier frameworks of motivation as a unitary concept [45, 46], by focusing on the dichotomy of the intrinsic and extrinsic locus of causality behind motivation [47]. The latter is facilitated by external or internal rewards, pressures, and expectations, while the former is based on the intrinsic properties of the activity itself, namely how well it can support the three basic psychological needs of competence, autonomy, and relatedness. Videogames include a fair amount of pressures and rewards which can promote extrinsic motivation[48], and yet they are generally regarded as good facilitators of intrinsic motivation [44]. Even when short-term shifts in motivation are observed during gameplay, games support the necessary psychological needs for the facilitation of intrinsic motivation on a higher level [8]. In the context of videogames, Ryan et al. [43] describe the basic psychological needs underlying intrinsic motivation as:

  1. Competence or a sense of accomplishment and a desire for the mastery of an action, which manifests through the proximal and distal goals of the players. This need is generally tied to self-efficacy and a sense of meaningful progression. It is supported through the interactions the players have to master in order to complete the game but not completion in itself.

  2. Autonomy

    or a sense of control and a desire for self-determined action, which manifests through meaningful choices, tactics, and strategic decisions the players can take. It is supported through rule systems and different game mechanics that both structure the play experience but allow for a high degree of freedom and meaningfully different outcomes.

  3. Relatedness or a sense of belonging and a desire to connect and interact with others, which manifests through interactions with other players and believable computer agents. It is supported by multilayer interactions, believable and rich non-player characters, narrative design, and even interactions with other players outside the games as well.

  4. Presence or the feeling of a mediated experience is a main facilitator of both competence and autonomy, and can be viewed as having physical, emotional, and narrative components [1, 43]. Indeed, the feeling of presence or the pursue of immersion can be a driving force behind the motivation of gameplay [49, 50, 8]. Based on the strong relationship between STD and presence, both the Player Experience of Need Satisfaction Questionnaire [43] and UPEQ [14] use it to measure a level of involvement with the game which can facilitate other positive psychological needs.

It is important to note that the above factors are not contributing equally to the formulation of intrinsic motivation; while competence or relatedness are regarded as the core catalysts, autonomy generally plays a supporting role in the facilitation of motivation. Nevertheless, in absence of autonomy, motivation can only be considered introjected or compulsive [51]. Within games the main drive of intrinsic motivation is generally competence because of how the activity is structured, while relatedness contributes to enhancing the experience [1].

In this paper we rely on SDT and recent advances on measurement tools to quantify the four above-mentioned aspects of motivation. For that purpose we use UPEQ, a game-tailored questionnaire designed to measure the factors of SDT as affected by the gameplay experience. UPEQ was developed by researchers at Massive Entertainment [14] specifically to predict gameplay outcomes relevant for industry designers and stakeholders. Earlier work [14] has demonstrated that UPEQ is able to predict playtime, money spent on the game, and group playtime based on measured factors of SDT. Beyond its utility, UPEQ also addresses the limitations of prior domain-specific SDT questionnaires, such as the Game Engagement Questionnaire [52], BrainHex [53], and the Player Experience of Need Satisfaction [43], while focusing on the adaptation of the Basic Need Satisfaction Scale(s) [54] into a survey specific to videogame play. The result is a reliable and consistent assessment tool with a strong theoretical foundation in SDT.

Ii-C The Ordinal Nature of Motivation

In the proposed methodology for this study we opt for preference learning (PL) methods because of the strong connection between this ordinal machine learning paradigm and how player experience operates in games. In essence, PL models certain psychological processes by focusing on the differences between occurrences instead of their absolute values [55, 15]. This approach falls much closer to the players’ cognitive processes—e.g. anchoring-bias [56, 57], adaptation [58], habituation [59], and other recency-effects [60]—that help them evaluate their own experience internally.

During recent years there is growing evidence supporting the strength of preference learning for modelling emotions and user experiences both on a conceptual [55, 15] and a technical basis [61, 62, 63, 64]

. Conceptually, treating subjective-defined ground truth data as ordinal variables brings the representation of data closer to the players’ underlying true attitudes

[55]. This is based on the observation that one’s affective state is always relative to a certain adaptation level [15]; i.e., emotions have a shifting baseline based on previous experiences. Even though traditionally psychological data is treated as nominal categories [65], by now there is ample evidence suggesting higher validity and reliability when labels of emotion or experience are treated as ordinal [66, 61, 67]. Indicatively, Martinez et al. [61] compared classification and PL across a number of modelling tasks found that ranking yields more robust models than classification. A number of other studies compared the processing of affective annotations as both ratings (e.g., Likert items) and rankings and found that 1) first-order data processing (i.e. ranks) yields higher reliability and inter-rater agreement and 2) second-order processing of the absolute rating values was also beneficial with regards to both reliability and validity [62, 63, 15]. Recently Camilleri et al. [32] applied PL for obtaining general models of players across games and Melhart et al. [64]

showed that ranking Support Vector Machines (SVMs) are more robust than support vector classifiers in predicting arousal across dissimilar affective corpora.

Given the above theoretical framework on the ordinal nature of experience and the large body of recent empirical evidence on the benefits of the ordinal modelling approach [15], in this paper we view player motivation as an emotional construct [1] with ordinal properties. As a result we compare player feedback on relative grounds and use PL to model the ranking between the levels of reported motivation in players as measured by the factors of UPEQ. In that regard, we do not focus on the internal validity of the survey data—as this has been the focus or earlier work [14]—and we instead consider the UPEQ scores as the underlying ground truth we need to approximate. After acquiring a general score for all the measured factors for each participant, we return to analysing and modelling the data as ordinal values, thereby following a second-order modelling approach [15].

Iii Preference Learning for Modelling Motivation

Preference Learning (PL) is a supervised machine learning technique [16], in which an algorithm learns to infer the preference relation between two variables. PL is a robust method, which relies on relative associations instead of absolute values or class boundaries and is instead based on the pairwise transformation of the original dataset into a representation of the differences between feature vectors in the query [68]. This transformation of the dataset reformulates the original problem in a way that a binary classifier can solve it. In our new dataset, we associate the direction of the preference relation with one of two classes. As an example, we observe the preference relation: ( is preferred over ) based on their associated output: . Through the pairwise transformation two new features are created: , associated with and , associated with . This comparison between each pair of feature vectors provides new datapoints. is a subset of all possible unique combinations because a clear preference relation is not always inferable.

This study uses ranking Support Vector Machines (SVM) [69] as they are implemented in the Preference Learning Toolbox111[70], which is based on the LIBSVM library [71]

. We choose SVMs in this initial study as they have proven their ability to yield robust models even with a limited amount of data and input features. SVMs were originally employed to solve classification tasks by maximizing the margins of a hyperplane separating the datapoints projected into a higher dimensional feature space

[72] but were later adopted to solve PL tasks as well [69]. We use both linear and non-linar SVMs with radial basis function (RBF) kernels. Unlike linear SVMs, which aim for a linear separation between datapoints, RBF SVMs emphasize the local proximity of datapoints, fitting the maximum-margin hyperplane in a transformed feature space [72]. For tuning our algorithms, we rely on the regularization term which controls the trade-off between maximizing the margin and minimizing the classification error of the training set, and—in case of RBF kernels—the hyperparameter, which controls how each comparison between datapoints is weighted in the non-linear topology by limiting the variance of the similarity measure between points.

Iv The Game and the Data

In this section we first present the examined game in detail and we then discuss the nature of the collected data and the ways we pre-processed it for modelling motivation.

Fig. 1: An example of the gameplay of Tom Clancy’s The Division (Ubisoft, 2016). Image taken from: No copyright infringement intended.

Iv-a Tom Clancy’s The Division

The data we analyse in this study is in-game behavioural data (player metrics) and survey questionnaire responses from players of Tom Clancy’s The Division (Ubisoft, 2016), hereafter The Division, collected between 2016 and 2018. The Division is an online multiplayer action-role playing game (Fig. 1), that combines a character progression system with third-person, cover-based, tactical shooting combat mechanics. The game is set in a post-apocalyptic New York which is hit by a smallpox epidemic. Players, as government agents, have to work together (and against each other) to scavenge and investigate the city, which fell into chaos in the aftermath of the pandemic and the rise of organised crime activity.

The core of the game is a progression system, in which players gain new levels by participating in different in-game activities including story-focused and optional missions to unlock new abilities, and gain new equipment including weapons and armour. The strength of a player can be measured by their level (up to 30) and the quality of their equipment is expressed in Gear Score points. In the player versus environment (PvE) sections of the game, players can group up and complete missions together. The game also features a competitive player versus player (PvP) area—called the Dark Zone—which has its own progression system. In this special area players can still group up to complete missions for better equipment; however, they can also turn on each other and become Rogue by killing other players and taking their rewards for themselves. After reaching the maximum level, players can participate in Incursions, which are particularly difficult missions for groups. Ubisoft also released a number of expansions for the game in the form of downloadable content (DLC), which added new areas, equipment, and both PvE and PvP content to the game.

The game was not only well received (80/100 Metacritic score on consoles222 but was also the best selling game of Ubisoft at the time of its release333 As the game integrates different systems from massively multiplayer online role playing games and multiplayer shooters and supports different play styles and interaction modes (i.e. player-environment and player-player), it provides a rich and complex game test-bed for research on motivation. As the popularity of online multiplayer games is on the rise and the second instalment of The Division set to release on March 15, 2019444, a study on how the gameplay of The Division shapes the motivational factors of its players is both timely and relevant.

Iv-B Dataset and Preprocessing

The collected data consists of aggregated information on the in-game activity of players over a long period of time and their corresponding UPEQ survey scores. These two types of data were collected independently, with the gameplay features recorded between 2016 and 2018 and the survey data collected through a web interface separately in 2018. As such, the survey data measures a general disposition of the players.

The dataset consists of one datapoint per player and, in total, players participated in the above-mentioned data collection process. Approximately of the subjects were young adults, between 18 and 34 years old, while were underage (15-17), between 35 and 54, and above 55. Country-wise, of the respondents were from the United Kingdom, from Australia, from Sweden, from Denmark, from Finland, from Norway, from New Zealand, with the remaining not providing an answer.

The dataset is cleaned of datapoints with missing values, corrupted entries, and outliers to prevent skewing any statistical analysis process. An extensive pruning was necessary due to outliers distorting the distribution of general game metrics (see Section

IV-C) and due to noise generated by the data logging service which inflated playtime. After the cleaning process the dataset contains players.

Iv-C Extracted Features

To represent computationally the player’s behaviour within the game, we ad-hoc design high-level gameplay features. While most of these are simple aggregated game metrics describing the time allocation and progression of the player, of them are exclusive categories of distinct play styles based on sequence-based profiling of the player’s in-game activities [10]. Additionally, the dataset contains Likert scores that represent the four motivation factors of each player as measured by the UPEQ survey. The three types of data considered in this study as detailed as follows:

  1. Game Metrics: These features can be categorised as relating to general playtime (Days Played, Days in Groups, Days in the Dark Zone, Sessions, Playtime, Group Playtime, Dark Zone Playtime, Playtime as Rogue); completion (Non-Daily Missions, Daily Missions, Side Missions, Days with Incursions, Incursions); progression (Gear-Score, Dark Zone Rank, Level, Early Level 30, Reached Level 30); early gameplay (Level , Early Playtime, Early Group Playtime, Early Dark Zone Playtime, Early Playtime as Rogue); and DLC gameplay (Underground Playtime, Survival Playtime, Season-Pass).

  2. Player Types: The different player types are named Adventurer, Elite, PvE All-Rounder, and Social Dark Zone Player. These types have been derived through a traditional k-means clustering of highly aggregated data and have been discontinued.

  3. Motivation Factors: UPEQ scores the four factors of motivation in the form of averaged Likert-scale values. While computing the mean of ordinal data can be problematic conceptually [63, 15], average survey scores are a wide-spread method of using Likert-like data as they can still show certain tendencies within the scores (e.g., a higher score is assumed to correspond to an overall more positive response). As mentioned in Section II-C, we adopt a second-order modelling approach [15] and treat these scores as ordinal data through pairwise comparisons across all players.

Iv-D Ordinal Dataset Processing

As the original dataset contains one datapoint per player, individual feature vectors used for the preference learning task are independent.This means that during the preparation of the PL experiments, each datapoint is compared to every other point during the pairwise transformation of the dataset. This transformation applies a preference threshold () parameter, which controls the margin of significance under which two datapoints are considered equal. The purpose of a threshold is to counter the noise in the ground truth data which can skew modelling results. Additionally to translating the relationship of datapoints into preference relations, this step also creates new datapoints for the machine learning task. The size of the dataset is nearing a quadratic proportion to the original dataset, with training and testing points on average depending on the ground truth and the optimal parameter (see Section VI-A). Furthermore, because each pairwise comparison creates two new datapoints—describing the preference relation in both directions—the transformation balances the baseline of the classification task to accuracy.

V Descriptive Statistical Analysis

Fig. 2: Rank correlation matrix of the game metrics, play style classes (motivation model input), and motivation factors (motivation model output).

In this section we present a descriptive statistical analysis which offers a first overview of the properties of the dataset. In particular, we use Kendall’s

to explore connections between individual input features (game metrics and play styles) and the four corresponding target outputs (motivation factors measured by UPEQ). We choose over Spearman’s because of its lower gross error-sensitivity and asymptotic variance [73]. In the analysis of this Section we use two statistical significance thresholds () of and .

As evidenced by Fig. 2, the data yields significant correlations between and within game metrics and play styles, however, there is little to no linear relationship between these features and motivation factors. The analysis finds that between () and () of the observed correlations are significant within the game metrics, which is not surprising given that most of the data is focused on the time players spend on different activities. As such, many of the more general features (e.g., Days Played) should correlate with the frequency of specific activities (e.g., Completed Daily Missions) and vice versa. Looking at the correlations between game metrics and play styles, we observe fewer significant connections, between () and (). The Adventurer and the PvE All-Rounder are the play style categories which have the most significant correlations with the game metrics. This result makes intuitive sense as the Adventurer class denotes novice players with generally lower amounts of playtime and PvE All-rounder represents players with high completion rates, group playtime, and participation in endgame content.

High correlations between game metrics and motivation factors are not as frequent: only between () and () of the observed correlations are significant. The motivation factor correlated with the most game metrics is relatedness. As evidenced by Fig. 2, relatedness—a motivation factor related to connection and interaction—reveals higher correlations with features denoting both group play (e.g., Group Playtime) and PvP actions (e.g., Playtime as Rogue). Autonomy is also correlated with a number of gameplay features, but mostly with completion (e.g., Daily Missions) and early play (e.g., Playtime as Rogue) aspects of gameplay.

Given that play styles are conceived as high level distinct clusters of behavioural patterns, it is not surprising that the number of correlations within the group are low ( () and ()) and only separate Adventurer—the novice—from the other classes. Correlations between play styles and motivation factors are between ( () and (). The Adventurer and the PvE All-Rounder are the classes that correlate with competence and relatedness, suggesting that these factors are easier to predict linearly via play styles.

Finally, there is a high number of correlations between the motivation factors ( for both values). The only correlations which are not significant are between relatedness and autonomy and between relatedness and presence. This result supports quantitatively the theoretical assumption—see Section II-B—that the catalyst of intrinsic motivation in games is the competence of a player; both player autonomy and relatedness appear to have supporting roles as factors of motivation. As the UPEQ questionnaire measures mainly spatial presence [14], it makes intuitive sense that there are is no effect between presence and relatedness because while the former describes a form absorption by the game world, the latter often requires communication outside of the game.

While the above results showcase the descriptive capacity of UPEQ to a good degree, the limited number of significant effects obtained for the four reported factors of motivation highlight that the mapping between gameplay and reported motivation is complex and far from linear. As more sophisticated methods are needed to capture that mapping, in the remainder of this paper we present experiments using machine learning for deriving both linear and non-linear relationships between gameplay features and motivation factors.

Vi Models of Player Motivation

This section presents the results of the machine learned models of player motivation based on different feature sets. As the extracted play styles are more complex descriptors of the player behaviour than the aggregated game metrics, we examine their capacity of predicting motivation independently (see Section VI-C and VI-B) but also in fusion (Section VI-D). Before delving into the details of the obtained results we outline the validation and parameter tuning process we followed for our PL models.

Vi-a Validation and Parameter Tuning

All PL models are validated with 10-fold cross validation. To prevent data leakage, the training and test folds are separated before the normalisation and pairwise transformation of the data. We apply z-normalisation to both the training and the test set before the transformation. To preserve the independence of the test set, we assume that it is drawn from the same distribution as the training set and apply the same transformation to the corresponding test set as well.

The optimal parameters of the RankSVMs are found through exhaustive search within value bounds. In particular we search exhaustively the triplets of , and values that yield the highest 10-fold cross-validation accuracies. The regularisation term is searched within , the RBF parameter in , and the optimal preference threshold in . While the best parameter was found to be over all experiments, and were more sensitive to the topology of the data; see Fig. 3. The and parameters selected were , for competence; , for autonomy; , for relatedness; , for presence for linear SVMs and , for competence; , for autonomy; , for relatedness; , for presence for RBF SVMs.

Fig. 3: Test accuracy sensitivity of models based on all available features to two core RankSVM parameters: and . When RBF SVMs are used, is .

Vi-B Models Based on Play Styles

In the first round of experiments we only use the four play styles as input of the SVM model to test their predictive capacity of motivation; Figure (a)a shows the final results obtained. Despite the low dimensionality of the input feature set, both linear and non-linear models are able to surpass the baseline respectively with and accuracy on average across all models. Although the best models surpass the threshold, the obtained performance on average is not impressive. We can safely conclude that the four play styles on their own are baseline-level predictors of player motivation and they do not suffice for building accurate models of motivation.

(a) Play styles
(b) Game metrics
(c) All features
Fig. 7: Average accuracy of the best linear and RBF SVMs. The dotted line shows the baseline. The error bars represent confidence intervals.

Vi-C Models Based on Game Metrics

In the second round of PL experiments we train the models only considering the game metrics as model inputs—as discussed in Section IV-C. As evidenced by the bar plots of Figure (b)b, game metrics are fairly successful in predicting the reported motivation factors. In particular, linear SVM models are successful with an average accuracy of across all models while the best models for individual factors are performing at almost accuracy on certain folds: , , , and , respectively, for competence, autonomy, relatedness, and presence. Relatedness appears to be the easiest factor to predict for the linear models, which is not surprising given that relatedness correlated with the most individual game metrics during the statistical analysis. On the other hand, linear SVMs struggle with autonomy, which can be explained by the low amount of correlations between autonomy and the other features found during the descriptive statistical analysis of the data.

Non-linear kernels further improve the model’s performance to a accuracy on average across all models. The best individual models vastly outperform the corresponding linear models reaching almost accuracy (competence: ; autonomy: ; relatedness: ; and presence: ). Compared to linear models, RBF SVMs appear to be more robust across any motivation factor as they manage to improve greatly even the poor-performing linear models (i.e., autonomy. Unlike the poor performances obtained with the models based solely on play styles, models based on game metrics are very accurate and robust across all four factors.

Vi-D Models Based on All Features

We assume that high level play style profiles can enhance the predictive capacity of game metrics by adding domain-specific information. As it is evidenced by Figure (c)c, including all available features for the PL task, improves the accuracy of the non-linear models beyond the capabilities of models based on game metrics alone. On the one hand, the linear models are only reaching on average across all tests (, , , and , respectively, for competence, autonomy, relatedness and presence) which is comparable to the performance obtained by the models based on game metrics. On the other hand, models using a non-linear RBF kernel reach acurracies of on average and achieve accuracy values above in their highest performing folds: , , and , respectively, for competence, autonomy, relatedness and presence.

Interestingly, despite relatedness being one of the easier factors to predict for linear SVMs models, the RBF kernel SVMs yield relatively lower accuracies on average. Nevertheless, similarly to non-linear models based on game metrics, RBF SVMs trained on all available features show a comparable predictive capacity across the four different motivation factors. Among the four, autonomy seems to be substantially more difficult to predict when using linear models.

Vi-E Visualising the Motivation Models

This section demonstrates the applicability of the models for game design, by re-examining the features describing players, ordered by the predictions of each model. To create a rank order between players, the models are refit to the whole dataset with the best parameters. We then create the global order across players’ motivation by comparing the output of the best obtained SVM models through a round robin tournament among all players in the dataset. We run the tournament for each of the four motivation factors separately. Finally we order the players based on the outcome of these tournaments and—for the sake of readability—we visualise the feature sets for the top 10 and the bottom 10 players in the order for each of the four motivation factors (Fig. 12).

As evidenced by Figure (a)a, the global order of predicted competence suggests that most of the top players ordered by this factor are playing more in general, team up early (and generally spend more time in groups), and have higher completion and progression rates compared to the players that rank at lowest competence. This is particularly evident in PvP, end-game PvE, and daily missions. It is not surprising that the players topping the competence rank are mostly associated with the PvE All-Rounder play style, whereas the players at the bottom tier have been classified mostly as novices (i.e., Adventurer). An interesting observation is that players rushing through the content (Early Level 30 coupled with lower general playtime) report lower competence than players building their characters more slowly. The connection between features and predicted ranking, however, is not always observable in a linear fashion. As players rate their experience without an external frame of reference, higher values in certain features do not necessarily mean higher ranking as well. As an example, the model selects to place three Adventurer players at the top of the competence rank. While these users have no end-game experience (no completed Incursions), they spent more time in groups and playing in the Dark Zone (i.e., the game’s PvP environment), than novice players in the bottom rank.

The players ranked by predicted autonomy are illustrated in Figure (b)b. While playtime in general does not have such an effect on the order of autonomy—as it has on the order of competence—there are observable patterns within the distribution of playtime between high and low tier players. While low tier players are spending their time in groups and focusing on PvE content (i.e., spending little time in the Dark Zone; not playing as a rogue), the high tier players are focusing more on PvP action even during the early game. These players are also spending their time more diversely on average, experiencing more options that the game has to offer, and thus having better completion rates. Daily Missions, which are recurring but optional activities for extra in-game equipment, also appear to be much more prominent in the top autonomy players.

The top and bottom ranked players based on relatedness is shown on Figure (c)c. The order based on relatedness is mainly affected by player-player interaction, which manifests as either group play (e.g. Group Playtime, Days in Groups, Incursion Days, etc.) or PvP action (Rogue Playtime), while other features remain comparable between the top and bottom tiers. Interestingly, highly ranked players are not possessing a season pass for DLCs (save for one), while almost all the bottom 10 players do. Because these players have no logged playtime with the DLC content, however, the effect can be attributed to a latent variable and not the DLC content.

Finally, the order of players based on predicted presence is depicted in Figure (d)d

. The order of the feature vectors appear to be uniformly distributed when ranked by

presence as there are not many observable features that differ between the top and the bottom players. Of note is that higher tier players show more involvement in the social activities of the game, mostly in terms of time spent in groups. Moreover, lower tier players are more focused on progression and expansion content. Since the UPEQ questionnaire measures physical presence [14], it is not surprising that activities which require extra-game coordination such as interaction with other players can decrease this factor.

Across all motivation factors, the feature varying the most between the top and the bottom ranked players is the completion of Daily Missions, and PvP activity (either spending time in the Dark Zone or going Rogue). While early gameplay alone is rarely a strong predictor, game metrics in general seem to be appropriate predictors and useful for both analytics and player modelling efforts. It is also largely evidenced that even in cases of no obvious linear relationship between individual features and motivation factors, non-linear PL techniques can provide efficient methods for predicting motivation and offer an insightful qualitative tool for game design.

(a) Competence
(b) Autonomy
(c) Relatedness
(d) Presence
Fig. 12: Feature sets of the top and bottom 10 players ordered based on predicted motivation factors. Each row represents a player and each column is a different input feature. The colour shows the normalised distribution.

Vii Discussion

In this study we applied preference learning via SVMs to model psychological constructs that make up self determination theory and demonstrated the supreme efficiency and robustness of our models to predict motivation factors of players based solely on their in-game data. We showed that these models can be used directly in a games user research context to predict factors of players’ motivation based on aggregated data, collected through multiple play sessions, which can be the subject of further qualitative analysis. In particular, we collected and processed the data of players of Tom Clancy’s The Division and defined motivation quantitatively using the UPEQ questionnaire. We converted the Likert scores to ordinal values using a second-order data processing approach [15] that yielded data corpora of over samples, representing the differences between players’ UPEQ Likert scores in pairs. We predicted each player’s levels of competence, autonomy, relatedness and presence with more than accuracy on average, whereas the best models achieve accuracies of at least .

The presented research offers insights for game industry professionals and stakeholders, who aim to leverage and enhance the positive psychological effects of gameplay but also for researchers in affective computing and game user research. The paper offers a novel approach that utilises a machine (preference) learning approach to model psychological constructs such as motivation by treating the subjectively defined notions as an ordinal phenomenon. To the best of our knowledge, this is the first study, which attempted to quantitatively model and predict constructs of STD based on player behaviour in a commercial-standard game and with that level of success.

A core limitation of this work is posed by the processing of the motivation factors (i.e., the ground truth). In particular, we opted for a simple experimental design and took the aggregated UPEQ scores at face value; aggregating Likert scale scores in this way, however, poses a number of methodological issues. The most important of these limitations is the processing of ordinal values as interval data [63]. As the distance between ordinal points is arbitrary, their mean value is not necessarily meaningful and introduces subjective reporting bias to the measurements [55]. To address this issue, future research should focus on new ways of transforming Likert-like data to preference relations. One possible approach would be to observe how individual player’s baseline shifts through the survey and scoring the 4 observed factors depending on significant changes in the responses.

Another limitation is posed by the nature of the dataset. As the dataset contains aggregated gameplay data, we cannot observe clearly how each player’s motivation changes over time and thus the relationship between the player’s “lifetime” and a one-time post-experience survey is rather broad. Although this type of data is easier to handle in a qualitative analysis, machine learning models are able to handle larger datasets with a higher dimensional feature space. Future research could focus on collecting a new dataset, recording multiple sessions and questionnaires per participant, and creating models which can predict temporal changes in the player’s motivation.

It is important to note that the ordinal data processing method we propose in this paper yields large datasets which are generated through the pairwise comparison of all players. As such, the method is feasible in current game development settings since it only requires a small sample of player experience annotations—such as those usually available in quality assurance departments of game studios. Although SVMs already showcased high levels of efficiency in predicting motivation on sets of several thousand datapoints, it can be argued that they might not be as robust when faced with much larger datasets of that type. In such instances alternative methods derived from deep preference learning [74, 75] are directly applicable to the modelling task.

Viii Conclusion

In this paper we attempted to infer the computational mapping between gameplay data and aspects of player motivation. In particular we used the Ubisoft Perceived Experience Questionnaire to survey about the competence, autonomy, relatedness and presence of almost 300 players of Tom Clancy’s The Division. We also logged these players’ in-game data and we examined the degree to which such data can be a powerful predictor of players’ survey responses. For that purpose, we converted survey scores to ordinal values and applied preference learning to create efficient and robust support vector machine models of the four factors of motivation. The ordinal machine learning approach we took proved extremely successful in predicting such complex psychological constructs by eliminating the reporting bias of the questionnaires. Our core findings suggest that the mapping between high-level gameplay metrics and survey-based annotations of complex emotional and cognitive states is not only possible to infer but the inferred models have predictive capacities that reach certainty levels.


  • [1] S. Rigby and R. M. Ryan, Glued to games: How video games draw us in and hold us spellbound.   Praeger, 2011.
  • [2] G. N. Yannakakis and A. Paiva, “Emotion in games,” Handbook on affective computing, pp. 459–471, 2014.
  • [3] G. N. Yannakakis and J. Togelius, Artificial Intelligence and Games.   Springer, 2018.
  • [4] J. M. Weinhardt and J. B. Vancouver, “Computational models and organizational psychology: Opportunities abound,” Organizational Psychology Review, vol. 2, no. 4, pp. 267–292, 2012.
  • [5] A. Canossa, J. B. Martinez, and J. Togelius, “Give me a reason to dig minecraft and psychology of motivation.” Proceedings of the Conference on Computational Intelligence and Games (CIG), vol. 2013, pp. 1–8, 2013.
  • [6] J. Hamari and J. Tuunanen, “Player types: A meta-synthesis,” Transactions of the Digital Games Research Association, vol. 1, no. 2, 2014.
  • [7] A. Canossa, J. B. Badler, S. Tignor, and R. C. Colvin, “In your face(t) impact of personality and context on gameplay behavior.” in Proceedings of the International Conference on the Foundations of Digital Games (FDG), 2015.
  • [8] D. Melhart, “Towards a comprehensive model of mediating frustration in videogames,” Game Studies, vol. 18, no. 1, 2018.
  • [9] A. Drachen, A. Canossa, and G. N. Yannakakis, “Player modeling using self-organization in tomb raider: Underworld,” in Proceedings of the Symposium on Computational Intelligence and Games (CIG).   IEEE, 2009, pp. 1–8.
  • [10] S. Makarovych, A. Canossa, J. Togelius, and A. Drachen, “Like a dna string: Sequence-based player profiling in tom clancy’s the division,” in Proceedings of the Artificial Intelligence and Interactive Digital Entertainment Conference.   York, 2018.
  • [11] M. S. El-Nasr, A. Drachen, and A. Canossa, Game analytics.   Springer, 2016.
  • [12] G. N. Yannakakis, P. Spronck, D. Loiacono, and E. André, “Player modeling,” in Dagstuhl Follow-Ups, vol. 6.   Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2013.
  • [13] R. M. Ryan and E. L. Deci, “Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.” American Psychologist, vol. 55, no. 1, p. 68, 2000.
  • [14] A. Azadvar and A. Canossa, “Upeq: ubisoft perceived experience questionnaire: a self-determination evaluation tool for video games,” in Proceedings of the International Conference on the Foundations of Digital Games (FDG).   ACM, 2018.
  • [15] G. N. Yannakakis, R. Cowie, and C. Busso, “The ordinal nature of emotions: An emerging approach,” IEEE Transactions on Affective Computing, 2018.
  • [16] J. Fürnkranz and E. Hüllermeier, “Preference learning,” in Encyclopedia of Machine Learning.   Springer, 2011, pp. 789–795.
  • [17] G. N. Yannakakis and J. Togelius, “Experience-driven procedural content generation,” IEEE Transactions on Affective Computing, vol. 2, no. 3, pp. 147–161, 2011.
  • [18] N. Shaker, G. N. Yannakakis, and J. Togelius, “Towards automatic personalized content generation for platform games.” in Proceedings of the Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), 2010.
  • [19] C. Holmgård, A. Liapis, J. Togelius, and G. N. Yannakakis, “Evolving personas for player decision modeling,” in Proceedings of the Conference on Computational Intelligence and Games (CIG).   IEEE, 2014, pp. 1–8.
  • [20] C. Bauckhage, A. Drachen, and R. Sifa, “Clustering game behavior data,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 7, no. 3, pp. 266–278, 2015.
  • [21] A. Drachen, R. Sifa, C. Bauckhage, and C. Thurau, “Guns, swords and data: Clustering of player behavior in computer games in the wild,” in Proceedings of the Conference on Computational Intelligence and Games (CIG).   IEEE, 2012, pp. 163–170.
  • [22] H. P. Martínez and G. N. Yannakakis, “Mining multimodal sequential patterns: a case study on affect detection,” in Proceedings of the International Conference on Multimodal Interfaces.   ACM, 2011, pp. 3–10.
  • [23] G. Wallner, “Sequential analysis of player behavior,” in Proceedings of the Annual Symposium on Computer-Human Interaction in Play.   ACM, 2015, pp. 349–358.
  • [24] T. Mahlmann, A. Drachen, J. Togelius, A. Canossa, and G. N. Yannakakis, “Predicting player behavior in tomb raider: Underworld,” in Proceedings of the Symposium on Computational Intelligence and Games (CIG).   IEEE, 2010, pp. 178–185.
  • [25] J. Runge, P. Gao, F. Garcin, and B. Faltings, “Churn prediction for high-value players in casual social games,” in Proceedings of the Conference on Computational Intelligence and Games (CIG).   IEEE, 2014, pp. 1–8.
  • [26] Á. Periáñez, A. Saas, A. Guitart, and C. Magne, “Churn prediction in mobile social games: towards a complete assessment using survival ensembles,” in

    Proceedings of the International Conference on Data Science and Advanced Analytics (DSAA)

    .   IEEE, 2016, pp. 564–573.
  • [27] M. Viljanen, A. Airola, J. Heikkonen, and T. Pahikkala, “Playtime measurement with survival analysis,” IEEE Transactions on Games, vol. 10, no. 2, pp. 128–138, 2018.
  • [28] S. C. Bakkes, P. H. Spronck, and G. van Lankveld, “Player behavioural modelling for video games,” Entertainment Computing, vol. 3, no. 3, pp. 71–79, 2012.
  • [29] H. P. Martínez, M. Garbarino, and G. N. Yannakakis, “Generic physiological features as predictors of player experience,” in Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII).   Springer, 2011, pp. 267–276.
  • [30] V. Georges, F. Courtemanche, M. Fredette, P.-M. Léger, and S. Sénécal, “Developing personas based on physiological measures,” in Proceedings of the International Conference on Physiological Computing Sysytems, 2018, p. 131–136.
  • [31] N. Shaker, S. Asteriadis, G. N. Yannakakis, and K. Karpouzis, “Fusing visual and behavioral cues for modeling user experience in games,” IEEE Transactions on Cybernetics, vol. 43, no. 6, pp. 1519–1531, 2013.
  • [32] E. Camilleri, G. N. Yannakakis, and A. Liapis, “Towards general models of player affect,” in Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII).   IEEE, 2017, pp. 333–339.
  • [33] Z. Borbora, J. Srivastava, K.-W. Hsu, and D. Williams, “Churn prediction in mmorpgs using player motivation theories and an ensemble approach,” in Proceedings of the International Conference on Privacy, Security, Risk and Trust (PASSAT) & Proceedings of the International Conference on Social Computing (SocialCom).   IEEE, 2011, pp. 157–164.
  • [34] N. Yee, “Motivations for play in online games,” CyberPsychology & Behavior, vol. 9, no. 6, pp. 772–775, 2006.
  • [35] K. J. Shim, K.-W. Hsu, and J. Srivastava, “An exploratory study of player performance, motivation, and enjoyment in massively multiplayer online role-playing games,” in Proceedings of the International Conference on Privacy, Security, Risk and Trust (PASSAT) & Proceedings of the International Conference on Social Computing (SocialCom).   IEEE, 2011, pp. 135–140.
  • [36] M. V. Birk, D. Toker, R. L. Mandryk, and C. Conati, “Modeling motivation in a social network game using player-centric traits and personality traits,” in Proceedings of the International Conference on User Modeling, Adaptation, and Personalization.   Springer, 2015, pp. 18–30.
  • [37] S. Reiss and S. M. Havercamp, “Toward a comprehensive assessment of fundamental motivation: Factor structure of the reiss profiles.” Psychological Assessment, vol. 10, no. 2, p. 97, 1998.
  • [38] E. Deci and R. M. Ryan, Intrinsic motivation and self-determination in human behavior.   Springer Science & Business Media, 1985.
  • [39] E. L. Deci, R. J. Vallerand, L. G. Pelletier, and R. M. Ryan, “Motivation and education: The self-determination perspective,” Educational Psychologist, vol. 26, no. 3-4, pp. 325–346, 1991.
  • [40] M. Gagné and E. L. Deci, “Self-determination theory and work motivation,” Journal of Organizational Behavior, vol. 26, no. 4, pp. 331–362, 2005.
  • [41] M. Joussemet, R. Landry, and R. Koestner, “A self-determination theory perspective on parenting.” Canadian Psychology/Psychologie Canadienne, vol. 49, no. 3, p. 194, 2008.
  • [42] M. S. Hagger and N. L. Chatzisarantis, Intrinsic motivation and self-determination in exercise and sport.   Human Kinetics, 2007.
  • [43] R. M. Ryan, C. S. Rigby, and A. Przybylski, “The motivational pull of video games: A self-determination theory approach,” Motivation and Emotion, vol. 30, no. 4, pp. 344–360, 2006.
  • [44] A. K. Przybylski, C. S. Rigby, and R. M. Ryan, “A motivational model of video game engagement.” Review of General Psychology, vol. 14, no. 2, p. 154, 2010.
  • [45] C. L. Hull, “Principles of behavior: An introduction to behavior theory.” Journal of Philosophy, vol. 40, pp. 558–559, 1943.
  • [46] B. F. Skinner, Science and human behavior.   Simon and Schuster, 1953, no. 92904.
  • [47] R. M. Ryan and J. P. Connell, “Perceived locus of causality and internalization: Examining reasons for acting in two domains.” Journal of Personality and Social Psychology, vol. 57, no. 5, p. 749, 1989.
  • [48] E. L. Deci, R. Koestner, and R. M. Ryan, “A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation.” Psychological Bulletin, vol. 125, no. 6, p. 627, 1999.
  • [49] M. Lombard and T. Ditton, “At the heart of it all: The concept of presence,” Journal of Computer-Mediated Communication, vol. 3, no. 2, 1997.
  • [50] G. Calleja, In-game: From immersion to incorporation.   MIT Press, 2011.
  • [51] R. M. Ryan and E. L. Deci, “Intrinsic and extrinsic motivations: Classic definitions and new directions,” Contemporary Educational Psychology, vol. 25, no. 1, pp. 54–67, 2000.
  • [52] J. H. Brockmyer, C. M. Fox, K. A. Curtiss, E. McBroom, K. M. Burkhart, and J. N. Pidruzny, “The development of the game engagement questionnaire: A measure of engagement in video game-playing,” Journal of Experimental Social Psychology, vol. 45, no. 4, pp. 624–634, 2009.
  • [53] L. E. Nacke, C. Bateman, and R. L. Mandryk, “Brainhex: A neurobiological gamer typology survey,” Entertainment Computing, vol. 5, no. 1, pp. 55–62, 2014.
  • [54] B. Chen, M. Vansteenkiste, W. Beyers, L. Boone, E. L. Deci, J. Van der Kaap-Deeder, B. Duriez, W. Lens, L. Matos, A. Mouratidis et al., “Basic psychological need satisfaction, need frustration, and need strength across four cultures,” Motivation and Emotion, vol. 39, no. 2, pp. 216–236, 2015.
  • [55] G. N. Yannakakis, R. Cowie, and C. Busso, “The ordinal nature of emotions,” in Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII).   IEEE, 2017, pp. 248–255.
  • [56] A. R. Damasio, Descartes’ error: Emotion, rationality and the human brain.   New York: Putnam, 1994.
  • [57] B. Seymour and S. M. McClure, “Anchors, scales and the relative coding of value in the brain,” Current Opinion in Neurobiology, vol. 18, no. 2, pp. 173–178, 2008.
  • [58] H. Helson, Adaptation-level theory: an experimental and systematic approach to behavior.   Harper and Row: New York, 1964.
  • [59] R. L. Solomon and J. D. Corbit, “An opponent-process theory of motivation: I. temporal dynamics of affect.” Psychological Review, vol. 81, no. 2, p. 119, 1974.
  • [60] S. Erk, M. Kiefer, J. Grothe, A. P. Wunderlich, M. Spitzer, and H. Walter, “Emotional context modulates subsequent memory effect,” Neuroimage, vol. 18, no. 2, pp. 439–447, 2003.
  • [61] H. Martinez, G. Yannakakis, and J. Hallam, “Don’t classify ratings of affect; rank them!” IEEE transactions on Affective Computing, no. 1, pp. 1–1, 2014.
  • [62] G. N. Yannakakis and H. P. Martinez, “Grounding truth via ordinal annotation,” in Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII).   IEEE, 2015, pp. 574–580.
  • [63] G. N. Yannakakis and H. P. Martínez, “Ratings are overrated!” Frontiers in ICT, vol. 2, p. 13, 2015.
  • [64] D. Melhart, K. Sfikas, G. Giannakakis, G. N. Yannakakis, and A. Liapis, “A motivational model of video game engagement.” Proceedings of Machine Learning Research, 2018 IJCAI workshop on AI and Affective Computing, vol. 86, pp. 26–33, in print.
  • [65] A. Kapoor, “Machine learning for affective computing: Challenges and opportunities,” The Oxford Handbook of Affective Computing, p. 406, 2015.
  • [66] M. Junge and R. Reisenzein, “Indirect scaling methods for testing quantitative emotion theories,” Cognition & Emotion, vol. 27, no. 7, pp. 1247–1275, 2013.
  • [67] ——, “Metric scales for emotion measurement,” Psychological Test and Assessment Modeling, vol. 58, no. 3, p. 497, 2016.
  • [68] J. Fürnkranz and E. Hüllermeier, “Pairwise preference learning and ranking,” in Proceedings of the European Conference on Machine Learning.   Springer, 2003, pp. 145–156.
  • [69] T. Joachims, “Optimizing search engines using clickthrough data,” in Proceedings of the SIGKDD International Conference on Knowledge Discovery and Data Mining.   ACM, 2002, pp. 133–142.
  • [70] V. E. Farrugia, H. P. Martínez, and G. N. Yannakakis, “The preference learning toolbox,” arXiv preprint arXiv:1506.01709, 2015.
  • [71] C.-C. Chang and C.-J. Lin, “Libsvm: a library for support vector machines,” Transactions on Intelligent Systems and Technology (TIST), vol. 2, no. 3, p. 27, 2011.
  • [72] V. Vapnik, “Chapter 5 constructing learning algorithms,”

    The Nature of Statistical Learning Theory

    , pp. 119–157, 1995.
  • [73] C. Croux and C. Dehon, “Influence functions of the spearman and kendall correlation measures,” Statistical Methods & Applications, vol. 19, no. 4, pp. 497–515, 2010.
  • [74] H. P. Martínez and G. N. Yannakakis, “Deep multimodal fusion: Combining discrete events and continuous signals,” in Proceedings of the International Conference on Multimodal Interaction.   ACM, 2014, pp. 34–41.
  • [75] H. P. Martinez, Y. Bengio, and G. N. Yannakakis, “Learning deep physiological models of affect,” Computational Intelligence Magazine, vol. 8, no. 2, pp. 20–33, 2013.