Learning is a process that occurs over time (Knight et al., 2017)
: new knowledge is built upon existing knowledge, suggesting that we should incorporate a temporal dimension in the analysis of learning data. Here, we develop a framework to model and analyse temporal sequences of task completion and task-to-task transitions by students taking a course, and allows us to compare them to the expected (i.e., designed) task order in the course. Our work introduces novel data-driven metrics and a Bayesian probabilistic model for analysis and prediction of sequence data, and shows that the order of task completion at fine-grained resolution facilitates improved prediction of performance and can be used to inform task-level course design. Unlike methods that construct feature vectors, our temporal data sequence framework(Mahzoon et al., 2018) directly incorporates temporal relationships between tasks. In the paper, we present the mathematical formulation and structure of the temporal data sequence framework, and we show how the data-driven analyses and Bayesian model can be used to predict student success, to reveal student confidence at a task level, and, from a task-centric perspective, to aid with learning re-design. We illustrate our work through data of student task completion collected from an online course taken over one semester as part of a 2-year MBA programme.
The availability of time-stamped data has improved substantially with the advent of computer-based education platforms (Kuzilek et al., 2017), coupled with advances in educational data mining and learning analytics that provide new and innovative tools for automated analysis. However, despite this increase in available tools, temporal analyses are still understudied in education research (Knight et al., 2017; Barbera et al., 2015). It is therefore important to expand the toolbox of temporal methodologies that are constructed to extract actionable information directly from the observed data without over-simplifying the learning process. For instance, the superiority of distributed learning over massed learning inherently posits the benefits of a consistent learning behaviour over time instead of an irregular, syncopated approach to learning (Bloom and Shuell, 1981). However, the distinction into only two such broad behaviours is an over-simplification, and a number of subtly different temporal behaviours have been shown to exist in real data (Peach et al., 2019). In general, the wide availability of data opens the possibility to introduce unsupervised methodologies that do not reduce observations to predetermined, broad categories, and aim to extract more nuanced results and conclusions.
1.1 Temporal feature analysis
In educational data mining, the most common approach to temporal analysis is to describe each student through a feature vector composed of a small selection of aggregated features from time-series data (e.g., time-gap between sessions, participation time, number of sessions, post views) (Kapur et al., 2008; Cepeda et al., 2006; Wise et al., 2012)
. These feature vectors can be combined with additional non-temporal information, such as grades or demographic information. The position of each vector (i.e., each student) in feature space can be further analysed using supervised or unsupervised learning algorithms to make predictions or to investigate the structure of the data(Munch, 2017). For instance, (Kloft et al., 2015)
extracted features aggregated over the course of a week (video views and active learning days combined with features such as country of origin) to predict drop-out rates. Similarly,Halawa et al. (2014) selected four predictive features (video-skipping, assignment skipping, lag, and assignment performance) to predict drop out rates. Feature analyses have also been highly successful at predicting stop out one-week in advance (Taylor et al., 2014), yet the analysis of posting patterns of students on a forum over the course of a semester. did not reveal anything about learning outcomes and student engagement with forum threads (Haythornthwaite and Gruzd, 2012). Recently, unsupervised clustering of features from learners following a blended course on two platforms (face-to-face, online) (Carroll and White, 2017)
identified four behavioural groups according to their differing levels of engagement across the two platforms. Other examples of temporal feature extraction includeAguiar et al. (2014), who calculated the number of times a student logged into their account among other non-temporal features; Papamitsiou and Economides (2014), who extracted temporal trace data (e.g., time to answer correctly) to predict student performance; and Riel et al. (2018), who extracted the time to half-way completion and frequency of completion, to develop tools for course managers or learning designers to understand and visualise participant performance.
Whilst feature vector analyses can be predictive of drop-out or performance in particular scenarios, these approaches are limited by the choice of features (‘feature engineering’), itself limited by the granularity of the data. Indeed, extending the feature space to include higher resolution temporal features (e.g, timestamps of lecture viewings or start-dates of peer-graded assessments) has been shown to improve prediction accuracies for drop-out and final performance (Ye and Biswas, 2014; Ye et al., 2015). Despite such improvements, temporal feature vectors lead to a ‘temporally averaged’ analysis of student trajectories, as they do not capture longitudinal (dynamical) changes, and therefore may miss important events or temporal dependencies.
1.2 Theoretically-motivated temporal analyses
Beyond feature vector representations, there have been developments guided by theoretical considerations (Wise and Shaffer, 2015). For example, a temporal analysis of conversational data used segmentation (or ‘stanzas’) to construct a network of micro-scale elements to reveal connections across timescales (Lund et al., 2017). A study of communal knowledge in online knowledge building discourse used temporal analytics combined with graph theory to identify ideas and their mobility over time (Lee and Tan, 2017). Graph centrality has been used to explore temporal and contextual profiles of shared epistemic agency on discourse transcripts (Oshima et al., 2018), identifying pivotal points of discourse exchanges. Halatchliyski et al. (2014) emphasised the importance of temporality as the main component of learning analytics through main path analysis of Wikiversity domains to model the flow of ideas. The method they developed could be used to support teachers in coordinating knowledge building, yet the method is not suited to sequential completion of tasks and would instead be interesting when applied to courses with no distinct task order. Another approach of temporal analysis is to perform so-called micro-analyses, where singular ‘extreme’ learners are examined in depth to explore differing behaviours (Wise et al., 2012). Whilst such approaches provide insight into particular behaviours, the conclusions are difficult to generalise and justify statistically (Reimann, 2009).
Whilst each temporal analysis requires a theoretically justified model, there is clearly also a need for high-resolution data that can reveal nuanced longitudinal relationships. Considering this, Liu et al. (2018) used a coarse grained approach across knowledge components to identify focal points that were worthy of more in-depth analysis using high-resolution data and analysis. This method integrated the temporal and data resolution scales that often evade temporal analyses, however, this approach may ignore potentially important focal points that can only be identified using high resolution analysis from the start. Peach et al. (2019)
used dynamic time warping on high-resolution task data to calculate similarities between the raw time-series of a cohort of students undertaking an on-line degree, and implemented a quasi-hierarchical clustering algorithm to identify data-driven groupings of students that exhibited similar temporal behaviours. They behavioural clusters were shown to be more predictive of final performance than classifiers based on temporal feature extraction method.
1.3 Sequence based methods
More recently, sequence-based methods have been shown to provide meaningful insights. For example Andrade et al. (2017) used optimal sequence matching to analyse the sequence of hand movements in a multi-modal learning environment. They found that incorporating temporality into their analysis was crucial to find a correlation between sensorimotor coordination and learning gains. Another sequence-based method was implemented by Chen et al. (2017), who used Lag-sequential analysis (LSA) and Frequent Sequence Mining (FSM) for the purpose of discovering sequential patterns in collaborative learning. They found that both LSA and FSM were able to uncover productive threads for collaborative learning. Data sequence models were introduced by Mahzoon et al. (2018) to model within- and between-semester temporal patterns. Their model emphasised the importance of incorporating temporal relationships of heterogeneous data to more accurately predict success or failure of students relative to non-temporal models with the same data. Sequence analysis has also been used on learning processes to test student learning outcomes at multiple levels and timescales (Chiu and Khoo, 2005; Chiu and Lehmann-Willenbrock, 2016; Chiu, 2018). Chiu (2018) explored whether one or more temporal sequences of learning processes related to a later learning outcome relative to a random concatenation of learning processes.
1.4 Our contribution
The recent success of sequence data analyses, as described above, suggests that using a sequence data framework can provide insightful analyses into learning analytics. We also highlight that the temporal sequence data framework is flexible: it provides an opportunity to analyse student trajectories as well as course learning design. A study by Mendez et al. (2014) concluded that using analytical techniques, including temporal analyses such as sequence mining, could highlight interesting anomalies or irregularities that can be the object of a more focused investigation by individuals with deep knowledge of the course. Accordingly, we use the data sequence framework to take a task-centric perspective, and identify common irregularities in task sequences across students.
Despite the benefits of sequence data models, there is a need for sequence-based methods for the analysis of high-resolution temporal data(Ye and Biswas, 2014; Ye et al., 2015; Tang et al., 2016; Mendez et al., 2014; Liu et al., 2018). In this paper, we treat each node in our sequence data framework as a single event allowing us to analyse the ordering of tasks. We take this approach so that we can incorporate high-resolution temporal data and identify high-resolution temporal relationships between events given our access to high resolution Learning Management System (LMS) data, and inspired by the remark by Mahzoon et al. (2018) that ‘Access to LMS data makes finer-grained node segmentation possible, which may lead to more timely assessments of academic risk’. Here we show that high resolution (task-level) data can be informative of student performance.
In this paper, we leverage LMS data and a sequence data framework to introduce both novel data-driven analyses and a Bayesian probabilistic model for the purpose of analysing and modelling sequence data. The data-driven analyses provide insights into our learner engagement data; for example, we show that the temporal data resolution (task level vs. weekly coarse grained sequences) can lead to complementary but distinct conclusions about learner engagement. The Bayesian probabilistic model is used to model student trajectories and to predict student performance as a function of sequence length, i.e., how early can we predict student performance from their trajectory.
The paper is structured as follows: In Section 2 we discuss the mathematical notation and structure of the data sequence framework. We also introduce methods for analysing the data sequences of students and the Bayesian probabilistic model for modelling sequence trajectories. In Section 3 we explore the ensemble data sequence trajectories of a student cohort undertaking an on-line course. In Section 3.1 we consider the patterns and engagement behaviours of the ensemble. We then compare the sequence engagement behaviours of students at high-resolution (task level) and coarse-grained (weekly) and compare the engagement behaviours of high performance and low performance groups. In Section 3.2 we apply the Bayesian probabilistic model to high and low performance groups to learn sequence trajectories associated with performance. Finally, in Section 3.3 we take a task-centric perspective and look at how the sequence data model can be used to reveal quantitative information about individual tasks and their relationship with aggregate student performance.
In summary, our paper provides the following contributions:
Within the framework of sequence analyses in learning analytics, we construct a data sequence method that treats single tasks as nodes, thus allowing the exploration of temporal relationships between tasks.
We compute key quantities that describe the probability of learners transitioning between tasks, and use these metrics to identify anomalous learner behaviours.
Comparing high-resolution temporal data with coarse-grained temporal data, we show that the temporal resolution of the data can lead to distinct conclusions about student engagement behaviour and its relationship with performance.
We introduce a novel Bayesian probabilistic model that is capable of modelling trajectories, and combine this model with a Bayes’ classifier to predict student performance.
In contrast to student-focused temporal methods that extract temporal features about student completion to predict student success, we show how the data sequence framework can also introduce metrics to analyse course design.
In this section we first describe and summarise the high-resolution dataset used to exemplify our analytical methods. Secondly, we describe the structure of the sequence data framework which forms the basis of the analytical methods and models introduced in this paper for the analysis of temporal sequence data. We then highlight key data-driven quantities that can be extracted from our data sequence model representation for the analysis of high-resolution LMS data. We then introduce a Bayesian probabilistic model for modeling sequence trajectories of learners and use it for predicting performance. Finally, we show that sequence data framework can be used to perform task-centric analyses and introduce data-driven metrics to reveal insights into course learning design.
2.1 Course information
The subjects in this study were post-experience learners pursuing a 2-year post-graduate part-time Management degree at a selective research orientated institution, a summary of the data is displayed in Table 2. The cohort ranged in age from 28 to 53 years old (57 males, 24 females) and resided in 18 geographically disparate countries. Although the subjects met face-to-face at the start of each academic year, the course was studied completely online without any further physical interaction. We gathered computer interaction data of the learners undertaking the online management course ‘Corporate Finance’ during the second semester of the first academic year. The anticipated study load was 5 to 7 hours per week for the single course we analyzed. This course was run in parallel with a second course (not analyzed here) with a similar workload. An online session was delivered to the students each week, with the course running over 10 weeks (for a total of 10 sessions). The course was assessed via a combination of coursework and a final exam. Coursework was undertaken at three points during the course: Sessions 3, 4 and 9. The performance distribution of the entire cohort had tail to low performance with a median of 69.7 (Figure 2).
|Population size, N||81|
|Module name||Corporate Finance|
|Degree Course||Online MBA (2 year)|
|No. Tasks, T||123|
|Mean Grade (.std)||68.4% (9.3%)|
|Mean Grade bottom 25% (.std)||55.6% (4.6%)|
|Mean Grade top 25% (.std)||79.3% (3.3%)|
2.2 Sequence data framework
The 10 sessions of the course are made up of a total of tasks identified by their ’Task ID’, an integer from 1 to 123 representing the nominal order of the course layout, i.e., the order in which tasks appear to learners as they navigate the online course website. This nominal task order is represented as the ordered set . Each learner is described by the ordered set of completed tasks:
where is the task ID of the task completed by learner . Note that the sets may be of different lengths for different learners since task completion is not compulsory and the number of completed tasks may vary from individual to individual. Figure 3 displays some example sequences for students using the framework described here.
We can also coarse grain our sequence framework. Given that tasks are grouped into sessions we compute the transition probability , by substituting the task IDs for session IDs in the ensemble . We use all these quantities to analyze the behaviors and critical juncture tasks associated with the course.
2.3 Data-driven learner centric analysis
Given our data sequence framework we introduce two key methods for analyzing the sequence data to investigate learner behavious. The whole cohort of learners is then described by the ensemble of ordered sets of tasks, . From this ensemble, we can compute the following two key quantities:
The number of times that task appears as the element across the ensemble is denoted as . The probability that task appears as the completed task across is thus given by . These probabilities are compiled in the matrix .
To quantify the transitions between tasks, we count the number of times that task is immediately preceded by task across the ensemble , and normalise it across all transitions to obtain the probability that task is preceded by task across the ensemble: . To obtain the conditional probability, we need to normalize across all tasks: . These conditional probabilities are compiled in the matrix . It should be noted that this matrix provides a summary of transitions to any task given a previous task acquisition, but is not a stochastic transition matrix for task acquisition due to tasks being irreversibly acquired.
The sequence analysis methods were adapted from Greenbury et al. (2018).
2.4 Bayesian probabilistic model: a generative model for task sequences
In the description of a learner’s task sequence, no assumptions about an underlying model that may have generated the sequences. The most general probabilistic model assumes that task completion is determined by all the tasks previously completed. If we assume that the order in which previous tasks were completed can be ignored, this model is a first-order Markov chain with a hypercubic state space defined bystates and an associated edges representing the transition matrix between these states. As , the state space and associated parameter space of the transition matrix is too large to be tractable.
Here we introduce the application of HyperTraPS, a recent probabilistic Bayesian model described in Greenbury et al. (2018); Johnston and Williams (2016). We fit this model to derive parameterisations that can be used for student classification. The model reduces the full hypercubic state space to a tractable state space by assuming that the probability of the next task in the sequence is proportional to a basal rate of acquisition for that task and a independent contributions from tasks already completed. These are fitted from the aggregate set of observed transitions (the student task sequences).
As we have complete information on the order in which tasks are completed, the utility of a generative model in this case may be found in its use as a classifier. Utilising and extending the Naïve Bayes classifier introduced in (Johnston et al., 2019), we consider the probability that a learner belongs to the high performing group and the low performing group after completing tasks with task set
. We can write the odds ratio relating to this probability as:
where are posterior samples drawn from the HyperTraPS model fitted to a set of sequences from labeled group . In this paper, the two groups considered are the top 25% () and the bottom 25% () based upon their score in the course.
2.5 Data-driven task centric analysis
Given the sequence data framework we are able to computer compute statistical properties of every task: the frequency with which it is completed; the mean rank across all learner sequences; and the difference between across different sub-groups.
2.6 Confidence analysis
To investigate the effectiveness of our analytical methods and model, we use a student confidence analysis to introduce a ‘story-telling’ or causal inference aspect . At the end of each session, students are asked to reflect on the tasks they had been given to complete. There were given three options to choose from for each task: (a) ’Yes I feel confident I can do this’, (b) ’I need to revisit this’, and (c) ’I need more support’. For each task we calculate the proportion of students that felt confident (stated confidence) or did not feel confident (needed to revisit or requested support).
Using this data we identify when a student was not confident on an individual task, and we measure the confidence of a student across a collection of tasks,
where a larger value indicates a more confident student and a lower value indicated a less confident student. Using this additional data, we can associate anomalous sequence orderings with measures of student confidence at both individual and group levels.
3 Results and Discussion
3.1 Data-driven learner centric analysis
Ensemble task completion behaviors of student cohort relative to course design
As defined above, the completion sequences of learners over tasks can be summarized as a 2D-histogram (or matrix of numbers) , where each element corresponds to the probability that task was completed in the position across the ordered sequences of the cohort. Figure 4A displays this histogram for our learner cohort, with tasks ordered according to their nominal task order () along the vertical axis and plotted against the actual completion order in the learners sequences on the horizontal axis. The histogram for each task across each row summarizes the spread of the actual order of completion across the learners in the cohort. If all learners had completed all tasks in the nominal order , then
would have been the identity matrix, with all the probability located on the diagonal. Therefore, the off-diagonal spread signals departure from the nominal order across the cohort. For example, learners completed task 1 as their first task with high coincidence (77%,), yet, even for this first task, there was a sizeable proportion of learners (21%, ) that completed task 1 as their second task. Overall, the strong diagonal component of Fig. 4A suggests that learners generally follow the nominal task order, with deviations evidenced by the presence of such off-diagonal elements.
At the beginning of the course, we observe a cohesive start in which learners generally follow the expected course structure as evidenced by the sequential peaks of each histogram up to task 15 (Fig. 4B(i)). Yet, more in detail, we observe some deviation from the nominal course structure. For example, very early on task 2 is most commonly completed in third position, whilst task 3 is completed second (Fig. 4
B(i)). Considering these two tasks in more detail, task 2 requires the learner to complete a vast amount of reading whilst task 3 requires the learner to submit an estimate for a quantitative question. Task 3 is located on the next screen requiring the learner to actively leave the web page containing task 2 to access task 3. Since task 3 is inherently related to task 2 (both explore financial inter-mediation), it could be that learners did not feel they comfortably understood the material in task 2 until they completed task 3. Another deviation occurs between task 15 and task 16 (Fig.4B(i)), whereby task 16 is completed earlier than task 15 on average. Task 15 requires the learner to complete a quantitative quiz with several complex financial questions. Although the quiz did not contribute to their final course grade, the learners may have wanted to leave the quiz until the end of the learning session, or they might have believed that additional material may become available later to aid them in the quiz.
Beyond such small deviations from the nominal task order, we also identify much larger deviations. Fig. 4B(ii) highlights tasks that have been completed much earlier that expected from the nominal task order. Interestingly, these tasks are completed in sequential order within their session, signalling a jump-ahead from the learners to an out-of-order later session. Such jump-ahead deviations appear as diagonal streaks in the lower triangle of Fig. 4A, but note that similar deviations are also observed in the upper triangle of Fig. 4A indicating tasks that were completed later relative to the nominal task order.
There are several features of the cohort dynamics that become more prominent as task completion progresses. In Fig. 4B(iii), we observe an example of bimodality (two peaks) in task completion indicating that there are two groups of learners completing these tasks systematically, but at different points in their sequences (relative to the nominal task order). Note that the presence of bimodality is not simply an artifact of learners randomly missing tasks, if so we would observe a single broad peak, but the emergence of subgroups of learners following different strategies. Indeed, this instance of bimodality appears at the beginning of Session 7 (task 84–task 95) and becomes less pronounced after Session 7 is finished. After a more thorough analysis, we find that a large group of learners appear to skip both task 78 and task 81 whilst the remaining learners complete these two tasks, resulting in a branching of two sets of learners corresponding to left-hand and right-hand peaks. Both of task 78 and 81 are interactive tasks that required an application of learned knowledge, which may have caused the split between those that were able to complete the tasks and those that were not.
As task completion progresses, we observe a broadening of the task distributions (Fig. 4B(iv)). Such broadening is expected when completion of tasks is not mandatory; if a learner misses a task and continues to follow the nominal task ordering of the course they will have a shift in their actual completion sequence relative to other learners. The broadening is amplified towards the end of the course, as a consequence of learners skipping an increased number of tasks (variable across the cohort of learners), in order to finish the course before the final exam.
An examination of the deviation of the different learners from the nominal task order is presented in Fig. 5. Fig. 5A shows two distinct task sequences for two learners: learner 17, who follows closely the nominal course order almost all the way to completion, and learner 29, who completes only about half of the total tasks in a more idiosyncratic order, with several jumps between sessions and completing only a little over 50% of the tasks of the course. We compile this information for all 81 learners in Fig. 5B, where we show the (normalized) distance between the actual completion of tasks and the nominal task order for each learner ordered in descending order of performance from top to bottom. Tasks in blue were completed earlier in the sequence relative to the nominal order, whilst tasks in red indicates later completion in the sequence relative to the nominal order. The learners of Fig. 5A are indicated by dots (green for learner 17, purple for learner 29). The observed deviations of learner 29 are seen as blocks of tasks (in blue and red) completed out of sync with the nominal task order. In contrast, the profile of learner 17 is almost totally white, indicating a constant progression in accordance with the nominal order. Fig. 5B also shows that low performing learners tend to exhibit large deviations from the nominal task order more often than high performing learners (although certainly not exclusively) and a lower fraction of completed tasks (as evidenced by the larger black blocks). Note that the figure shows sequential blocks of tasks, commonly corresponding to a single session, which have been completed out of sequence relative to the nominal order. This indicates that analyzing sequences not only at the level of tasks, but also at the level of weekly sessions can provide a meaningful temporal basis for the analysis of learners’ behaviors (Lund et al., 2017), as we explore below.
In summary, the results in this section have shown that using our data sequence model we are able to analyse the engagement behaviors of students at a high-resolution level. We are able to identify both anomalous deviations from the designed course structure at both a cohort level and for individual students. We have shown that once we have identified anomalous patterns, such as that for learner 29, we are able to provide a more indepth analysis that may reveal causal reasons.
A comparison between high-resolution and coarse-grained temporal sequences
Above, we extracted quantities that revealed deviations from the designed course sequence for both individual students and the ensemble of students. In this section we explore the high resolution task-level data and compare it against a coarse grained version of the data (coarse grained by week).
Fig. 6A plots the task transition probability matrix defined above in two different (but equivalent) ways: as a transition graph (i) and as a transition matrix (ii). As expected, we find strong transitions that follow the nominal structure both within each session as well as clockwise from session to session in Fig. 6A(i). This is similarly shown in Fig. 6A(ii) by the strong concentration on the diagonal and upper diagonal, which indicates the high probability that learners follow the nominal order of the course. However, we also find a large number of non-sequential transitions between tasks as a consequence of learners jumping forwards or backwards in the course. These are shown as non-sequential edges in Fig. 6A(i) and off-diagonal elements in Fig. 6A(ii). Specifically, we observe a large number of non-sequential transitions within sessions 2, 3 and 5. For example, within session 3 there is a large probability that learners complete task 42 and then return to task 38. In this particular case, task 38 is a discussion post where learners were required to publicly write a response to a tutor question whereas task 42 is a quiz with various questions regarding equity markets and valuations. It appears that a large number of learners completed the quiz before the discussion post, perhaps because they did not feel confident completing the public discussion post until they had gained additional information or understanding from the quiz. To explore this further we analysed the percentage of students that felt confident undertaking this task. We found that only 60% (59/81) of students were confident in this task (self-assessed at the end of each session), whereas across the remaining tasks within session 3 the average number of students that were confident was higher (69%, std. 7%) and the average task confidence across the module was (68% std. 8%).
For comparison, we take a coarse-grained approach and calculate the transition probabilities between the 10 weekly sessions (Fig. 6B) represented as: the inter-session transition graph in (i) and the coarse-grained transition matrix in (ii). As in the task level analysis, there is a strong clock-wise sequential component in the graph (i) and a concentration of probability on the main and upper diagonal in the matrix (ii), indicating that learners generally follow the session sequence. However, there are obvious deviations. For example, we observe that there is a high probability that learners will transition back to session 3 from several sessions (6, 7, 8, 9, 10). Similarly, learners at session 10 (the final session) will often transition to previous sessions (sessions 8 and 9), maybe in an effort to revise and complete parts of the course they had skipped over while completing the course.
These results suggest that task-level (high resolution) and session-level (coarse-grained) sequence data provide true but alternative descriptions of the data that can lead to distinct but complementary conclusions. Within the high resolution analysis we are able to identify individual anomalous tasks (such as task 38), whilst within the coarse-grained analysis we identified an anomalous week (e.g. week 3). This section highlights the importance of using high-resolution temporal data coupled with an appropriate temporal model for analysis.
Comparing sequence completion patterns between high and low performance learners
Given the differences between the high resolution and coarse grained analyses, we perform a similar comparison whilst exploring student performance. As suggested by Fig. 5B, the pathways and patterns of task completion may differ between high and low performing learners. To explore this hypothesis in more detail, we divide our cohort of learners into two groups: those in the top 25% (Group 1) and bottom (Group 2) 25% of performers according to course grade. Although meaningful and more nuanced groups could be created, we opted for a split into the bottom 25% and top 25% to ensure a bimodal performance distribution.
We compute two separate transition matrices and from the sequences of learners in Group 1 (high performers) and in Group 2 (low performers), respectively, and obtain the difference between both: . In Fig. 7A we show the transition graph (i) and transition matrix (ii) where the high performers have larger probability (i.e., the elements of ). Similarly, Fig. 7B presents the transition graph (i) and transition matrix where the low performers have larger probability (i.e., the elements ).
We identify a number of differences in the transitions when comparing high and low performers. For example, low performers were more likely to transition directly from session 5 to session 9, which contained a coursework task that needed to be completed, and after completion of session 9 they were likely to transition back to session 6. In general, low performing learners have a higher probability of transitioning to a task within the same session (diagonal of Fig. 7B(ii)) but also a higher rate of transitions between non-sequential sessions (Fig. 7B(i)). These two views are only seemingly contradictory: low performers followed the nominal task order within sessions but did not follow the nominal session order within the course. We also observe that the only point where high performers are more likely to depart from the course session structure than low performers is in the jump back transitions form session 10, an indication of an effort to revise and complete missing tasks just before the end of the course. These results reinforce the need for high-resolution data and analysis. Using a coarse-grained approach we would observe low performing students less strictly following the course session structure, however, we would not observe their higher probability of following the nominal task order.
Following on from the confidence analysis in Section 3.1
, we found that the low performance group were significantly (paired t-test, p=0.041, alpha=0.05) less confident (no. tasks confident, mean=0.65, std=0.27) than the high performance group (no. tasks confident, mean=0.76, std=0.19) across the entire module. At the group level these confidence values correlate with performance as we would expect. However, these results suggest that less confident students are less likely to follow the nominal session order of the course.
3.2 Bayesian model for sequence trajectories
In this section, we use the probabilistic generative model to examine whether we can predict the group (top 25%, or bottom 25%, ) to which a student belongs after completing tasks with task set . We perform two experiments to consider the plausibility of such a model for the data.
In our first numerical experiment, we test whether the probabilistic model can learn parameterisations that can distinguish two groups. We draw posterior samples and from HyperTraPS for the entire set of and . For each learner in the training datasets and , we plot the probability that they belong to their respective groups. Fig. 8 (top) shows the result of this when the test learners that we are trying to predict group identity are including in the training dataset. In this case, the left-hand plot (for the top 25% learners) and right-hand plot (for the bottom 25% learners) show strong predictive ability after only a few tasks have been completed in the set of all tasks with the probability going to unity and zero for the groups respectively after around 30 tasks. This indicates the probabilistic model is able to learn differences between the two groups in its parameterisations , an important prerequisite for establishing whether learners task set in conjunction with parameterisations of a generative model can be used to predict performance.
In our second numerical experiment, we test whether the probabilistic model can use learned parameterisations to predict which group unseen learners belong to. We do this by splitting our groups and into a 70% training and 30% test split. We draw posterior samples from and , and then perform the same method to calculate the probability that learners from the test split belong to and (given we know their true labels). Fig. 8 (bottom) illustrates the results for the test split of learners (split by their true label on left-hand side and on right-hand side). The results indicate much less certainty for individual learners, with misclassification occurring. However, the averages across learners in each group indicate some predictive power with average values mainly above 0.5 for and below 0.5 for . Additionally, the region of strongest predictive power where the averages are farthest from 0.5 is around the region where we have observed learners being particularly divergent in their choice of task suggesting the results may be robust. Given such small dataset sizes of only around 15 learners for training, this result would be able to be validated on courses with many more learners available.
3.3 Data-driven learner centric analysis
Using a data sequence framework, we have shown that we can analyse individual or ensemble trajectories of students. However, one of the benefits of a data sequence framework is that it can be used for task centric analysis, i.e. each individual task can be analysed with respect to the sequences of a group of students to study particular outcomes such as performance. In this section we introduce task centric quantities that can be extracted using a data sequence framework. In particular, we explore the differences between high and low performance in completion behaviour for individual tasks and task types.
The tasks are of six types: Coursework; Reading/Video; Quiz; G-chart task, Multi-response poll, and Discussion post. For each task, we calculate: (i) the difference in the frequency of completion between high and low performers; (ii) the difference in the mean position in the completion sequence between high and low performers. The results for all tasks are shown in Fig. 9 identified by their Task ID and colored by their task type. Deviations from the (0,0) point in the centre of Fig. 9 correspond to differences between low and high performers. If a task appears in the upper half, the task has been completed earlier on average by the group of high performing learners. If a task is located on the right half, the task has been completed more frequently by the group of high performing learners. Therefore, a task located in the upper right quadrant is completed earlier (in the learner sequence - not necessarily in time) and more frequently by high performance learners.
In Fig. 9 we also show the median and interquartile range for each task type. The median for the ’Reading/Video’ tasks appears almost directly at the centre of the plot, indicating that completing a video or a reading task is not correlated with high or low performance. In contrast, other task types show significant deviations between the two groups. Specifically, discussion posts or interactive tasks such as multi-response submissions (usually requires a learner to make a calculation) and G-chart tasks (learners must submit a choice from a selection of answers) have a median that is located in the upper right quadrant, i.e., they are completed more frequently and earlier by high performers. In addition, coursework tasks appear significantly earlier in the sequences of low performers. Although this might seem counterintuitive at first sight, it is consistent with the fact that low performers tend to skip more tasks but do complete courseworks.
Beyond general task types, we can identify specific tasks that may be correlated with high performance. For example, tasks 64 and 78 (highlighted in Fig. 9) exhibit high frequency and early completion by the high performing learners. Task 64 was an open discussion where learners were asked to perform two calculations and then discuss the results; task 78 required learners to make a decision about picking stocks for a portfolio. Both of these tasks require the learner to understand the previous content. The public nature of the answers among their peers might put off learners that lack confidence in their answers. In the case of discussion tasks, where a learner must submit a public answer to a tutor’s question, 19/20 discussion tasks are found in the right half, where high performers complete these tasks more frequently, and 15/20 are found in the upper right quadrant, where they also appear earlier in the sequence of high performers.
The bottom half contains a number of tasks such as the coursework tasks 36, 54, 55 and 115. As discussed above, coursework tasks appear significantly earlier in the sequence of low performers as a reflection of the fact that they have completed fewer tasks by the time the coursework is due for submission. Hence low performance learners skip ahead to reach the coursework. Other tasks completed earlier and more often by low performing students include a number of tasks from session 2 (tasks 17, 20, and 29) and session 9 (tasks 108 and 109) which appear because low performing students skipped the beginning of both sessions.
The results in this section, again, highlight the need for high-resolution temporal data that can differentiate between individual tasks. At the session-level we would be unable to differentiate clearly between task types. We show that the sequence data framework and the associated analyses are able to highlight tasks that are associated with anomalous engagement behavior. This provides learning designers and educators with an appropriate method for analyzing the course design and evaluating any parts that may require re-design.
In general, current temporal analyses of student data tend to obfuscate the longitudinal relationships between events through averaging, coarse graining or feature extraction. In this paper, we have used a data sequence framework to form the basis of a set of data-driven analyses and a Bayesian sequence model to investigate high-resolution task completion data of learners. We also show that using this framework we can introduce a task-centric analysis which is capable of informing learning design. In general we identify aggregate and individual learner behaviors; explore the relationship between course structure and learner trajectories; reveal the connection between completion patterns and task types; and identify points in the course where learners might benefit from intervention.
Summary of paper
The first section used the data sequence model to analyze the ensemble of learner trajectories relative to the nominal task order. We highlighted the fact that although learners generally follow the nominal course structure, a number of exceptions occur, for different reasons, at different junctures in the course. We identified various behaviors such as bimodality and branching from nominal task ordering that occurred as a consequence of groups of learners taking alternative approaches to task completion. Our comparison between learner trajectories and nominal course structure revealed large deviations in different parts of the course and increasingly as the course progresses. Whilst not given here, analyzing the causal effects of such behaviors could provide the educator with an improved understanding of their course design, and about individual learners undertaking the course.
In the second section, we compared a high-resolution approach with a coarse-grained approach to temporal analysis. The transition probability between tasks and sessions was used to study the deviations when learners transition between non-sequential tasks. The analysis at the coarse-grained level of inter-session transitions revealed specific sessions in the course structure where deviations from the nominal course structure occurred, highlighting the importance of an analysis at different levels of time resolution to understand the interactions of learners with the material (Lund et al., 2017).
In the third section, we furthered our comparison of high-resolution and coarse-grained approaches to evaluate differences between high and low performance learners. We began our exploration of the differences in completion behaviors between high and low performing learners. Having split the learners into two groups according to the median grade, we showed that low performing learners followed more strictly the task structure within a session but were less likely to follow the session structure throughout the course. The analysis also showed that low performers tend to skip a number of sessions in order to complete coursework tasks.
In the penultimate results section, we introduced a Bayesian probabilistic model to learn the trajectories of task completion sequences. Using the top 25% and bottom 25% groups of students (a bimodal performance distribution) we showed that the task completion sequences could be learned and used for predicting student performance.
In the final section, we showed that our data sequence model could be used for task-centric analyzes. We examined the link between the completion of individual tasks and learner performance. Our analysis showed that completion of task types related to active learning (such as discussion posts, multi-response polls, or G-chart tasks) were correlated with above median performance. The completion of such tasks, some of which are also visible to the peer learners, might indicate a level of understanding and confidence in the knowledge of the material around that task. An educator may consider encouraging the completion of some of those key tasks (or making them compulsory) in order to encourage the learner to learn the material in order to complete that task.
We found a clear distinction between high and low performers as regards to the types of tasks, how frequently they were completed, and at what point in their sequence. These findings could be considered in the context of Bloom’s Taxonomy of Learning (Bloom et al., 1956). For example, discussion-based tasks require skills toward the higher end of the taxonomy, whereas tasks that comprise reading texts and watching videos require skills towards the lower end. It is then not surprising that tasks in the former category were more commonly completed and earlier in the study sequence of high performing students. Identifying these types of tasks could help tutors structure the courses to reinforce the importance and build-up towards such tasks, as well as identifying differences between groups of learners. These results exemplify through a small illustrative example the power of relatively simple sequence analysis methods applied to learning data, and the potential for their use on available data to reveal the connections between learning patterns, performance, and course design.
This paper has provided: (1) methodological contributions (data-driven metrics and sequence modelling); (2) insights into the resolution of temporal data; (3) an evidence base to produce an actionable software package within our LMS.
The methodological contributions of this paper are three fold: (i) We show that a data sequence framework provides an appropriate format for investigating the temporal relationships of learning data; (ii) We introduced new data-driven analytical metrics for investigating sequential data; (iii) Using a Bayesian sequence model we are able to learn the trajectories of learners and show that it can be used for predicting performance.
The research in this paper has also provided necessary contributions within our Business School. We are currently developing a software module that incorporates the data sequence model into our LMS for two main purposes: (i) To identify anomalous student engagement behaviours; (ii) To evaluate the course design and identify areas that may require re-design.
Whilst the majority of on-line courses have a designed linear structure, it is clear that cognitive development does not progress through a fixed sequence of events. The alternative is to present each course component in parallel, without a displayed intended structure, and students can begin at any point and transition between any components of the course. In such a system, our Bayesian sequence model could still be applied given that it is able to characterise the complete state-space of possible trajectories and isn’t necessarily dependent on the ground truth course structure. Given this, we intend to pursue the Bayesian model further.
A second direction would be to look at higher order network models which don’t make Markovian assumptions. Considering the full trajectories of each student could yield higher order temporal dependencies that may better model the transitions between tasks.
Declaration of Conflicting Interest
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
- Aguiar et al. (2014) Aguiar, E., Ambrose, G. A. A., Chawla, N. V., Goodrich, V., and Brockman, J. (2014). Engagement vs Performance: Using Electronic Portfolios to Predict First Semester Engineering Student Persistence. Journal of Learning Analytics.
- Andrade et al. (2017) Andrade, A., Danish, J., and Maltest, A. (2017). A Measurement Model of Gestures in an Embodied Learning Environment: Accounting for Temporal Dependencies. Journal of Learning Analytics.
- Barbera et al. (2015) Barbera, E., Gros, B., and Kirschner, P. (2015). Paradox of time in research on educational technology. Time & Society.
- Bloom et al. (1956) Bloom, B. S., Englehard, M., Furst, E., Hill, W., and Krathwohl, D. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals.
- Bloom and Shuell (1981) Bloom, K. and Shuell, T. J. (1981). Effects of massed and distributed practice on the learning and retention of second-language vocabulary. Journal of Educational Research, 74(4):245–248.
- Carroll and White (2017) Carroll, P. and White, A. (2017). Identifying Patterns of Learner Behaviour: What Business Statistics Students Do with Learning Resources. INFORMS Transactions on Education, 18(1):1–13.
- Cepeda et al. (2006) Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., and Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin.
- Chen et al. (2017) Chen, B., Resendes, M., Chai, C. S., and Hong, H. Y. (2017). Two tales of time: uncovering the significance of sequential patterns among contribution types in knowledge-building discourse. Interactive Learning Environments.
- Chiu (2018) Chiu, M. M. (2018). Statistically modelling effects of dynamic processes on outcomes: An example of discourse sequences and group solutions. Journal of Learning Analytics, 5(1):75–91.
- Chiu and Khoo (2005) Chiu, M. M. and Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multilevel analysis. Small Group Research, 36(5):600–631.
- Chiu and Lehmann-Willenbrock (2016) Chiu, M. M. and Lehmann-Willenbrock, N. (2016). Statistical discourse analysis: Modeling sequences of individual actions during group interactions across time. Group Dynamics: Theory, Research, and Practice, 20(3):242.
- Greenbury et al. (2018) Greenbury, S., Barahona, M., and Johnston, I. (2018). Inferring high-dimensional pathways of trait acquisition in evolution and disease. To appear in Cell Systems, BioRxiv, pages 1–27.
- Halatchliyski et al. (2014) Halatchliyski, I., Hecking, T., Goehnert, T., and Hoppe, H. U. (2014). Analyzing the Main Paths of Knowledge Evolution and Contributor Roles in an Open Learning Community. Journal of Learning Analytics.
- Halawa et al. (2014) Halawa, S., Greene, D., and Mitchell, J. (2014). Dropout Prediction in MOOCs using Learner Activity Features. eLearning Papers.
- Haythornthwaite and Gruzd (2012) Haythornthwaite, C. and Gruzd, A. (2012). Exploring patterns and configurations in networked learning texts. In Proceedings of the Annual Hawaii International Conference on System Sciences.
- Johnston et al. (2019) Johnston, I. G., Hoffmann, T., Greenbury, S. F., Cominetti, O., Jallow, M., Kwiatkowski, D., Barahona, M., Jones, N. S., and Casals-Pascual, C. (2019). Precision identification of high-risk phenotypes and progression pathways in severe malaria without requiring longitudinal data. npj Digital Medicine, 2(1).
- Johnston and Williams (2016) Johnston, I. G. and Williams, B. P. (2016). Evolutionary inference across eukaryotes identifies specific pressures favoring mitochondrial gene retention. Cell systems, 2(2):101–111.
- Kapur et al. (2008) Kapur, M., Voiklis, J., and Kinzer, C. K. (2008). Sensitivities to early exchange in synchronous computer-supported collaborative learning (CSCL) groups. Computers and Education.
Kloft et al. (2015)
Kloft, M., Stiehler, F., Zheng, Z., and Pinkwart, N. (2015).
Predicting MOOC Dropout over Weeks Using Machine Learning Methods.
- Knight et al. (2017) Knight, S., Wise, A. F., and Chen, B. (2017). Time for change: Why Learning analytics needs temporal analysis. Journal of Learning Analytics, 4(3):7–17.
- Kuzilek et al. (2017) Kuzilek, J., Hlosta, M., and Zdrahal, Z. (2017). Open University Learning Analytics dataset. Scientific Data.
Lee and Tan (2017)
Lee, A. V. Y. and Tan, S. C. (2017).
Promising Ideas for Collective Advancement of Communal Knowledge Using Temporal Analytics and Cluster Analysis.Journal of Learning Analytics, 4(3):76–101.
- Liu et al. (2018) Liu, R., Stamper, J., and Davenport, J. (2018). A novel method for the in-depth multimodal analysis of student learning trajectories in intelligent tutoring systems. Journal of Learning Analytics, 5(1):41–54.
- Lund et al. (2017) Lund, K., Quignard, M., and Shaffer, D. W. (2017). Gaining Insight by Transforming between Temporal Representations of Human Interaction. Journal of Learning Analytics.
- Mahzoon et al. (2018) Mahzoon, M. J., Maher, M. L., Eltayeby, O., and Dou, W. (2018). A Sequence Data Model for Analyzing Temporal Patterns of Student Data. Journal of Learning Analytics, 5(1):55–74.
- Mendez et al. (2014) Mendez, G., Ochoa, X., Chiluiza, K., and de Wever, B. (2014). Curricular Design Analysis: A Data-Driven Perspective. Journal of Learning Analytics.
- Munch (2017) Munch, E. (2017). A User’s Guide to Topological Data Analysis. Journal of Learning Analytics.
- Oshima et al. (2018) Oshima, J., Oshima, R., and Fujita, W. (2018). A Mixed-Methods Approach to Analyze Shared Epistemic Agency in Jigsaw Instruction at Multiple Scales of Temporality. Journal of Learning Analytics, 5(1):10–24.
- Papamitsiou and Economides (2014) Papamitsiou, Z. and Economides, A. A. (2014). Temporal Learning Analytics for Adaptive Assessment. Journal of Learning Analytics.
- Peach et al. (2019) Peach, R. L., Yaliraki, S. N., Lefevre, D., and Barahona, M. (2019). Data-driven unsupervised clustering of online learner behaviour. npj Science of Learning.
- Reimann (2009) Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning.
- Riel et al. (2018) Riel, J., Lawless, K. A., and Brown, S. W. (2018). Timing Matters : Approaches for Measuring and Visualizing Behaviours of Timing and Spacing of Work in Self-Paced Online Teacher Professional Development Courses. Journal of Learning Analytics, 5(1):25–40.
- Schmidhuber (2015) Schmidhuber, J. (2015). Deep Learning in neural networks: An overview.
- Tang et al. (2016) Tang, S., Peterson, J. C., and Pardos, Z. A. (2016). Deep neural networks and how they apply to sequential education data. In L@S 2016 - Proceedings of the 3rd 2016 ACM Conference on Learning at Scale.
- Taylor et al. (2014) Taylor, C., Veeramachaneni, K., and O’Reilly, U.-M. (2014). Likely to stop? Predicting Stopout in Massive Open Online Courses.
- Wise et al. (2012) Wise, A. F., Perera, N., Hsiao, Y. T., Speer, J., and Marbouti, F. (2012). Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course. Internet and Higher Education Wise.
- Wise and Shaffer (2015) Wise, A. F. and Shaffer, D. W. (2015). Why Theory Matters More than Ever in the Age of Big Data. Journal of Learning Analytics.
- Ye and Biswas (2014) Ye, C. and Biswas, G. (2014). Early Prediction of Student Dropout and Performance in MOOCs using Higher Granularity Temporal Information. Journal of Learning Analytics.
- Ye et al. (2015) Ye, C., Kinnebrew, J. S., Biswas, G., Evans, B. J., Fisher, D. H., Narasimham, G., and Brady, K. A. (2015). Behavior Prediction in MOOCs using Higher Granularity Temporal Information. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S ’15, pages 335–338.