Learn-Apply-Reinforce/Share Learning: Hackathons and CTFs as General Pedagogic Tools in Higher Education, and Their Applicability to Distance Learning

by   Tom Goodman, et al.
University of Birmingham

This paper lays out two teaching/learning methods that are becoming increasingly prevalent in computer science - hackathons, and Capture the Flag (CTF) competitions - and the pedagogic theory that underpins them. A case study of each is analysed, and the underpinning similarities extracted. The frameworks are then generalised to Learn-Apply-Reinforce/Share Learning - a social constructivistic method that can be used subject-independently. The applicability of this new method to distance learning is then investigated - with a mind to potential necessity to work from home - both due to increasing demand in the Higher Education sector, but also the devastating impact of crises such as the ongoing COVID-19 pandemic. Finally, a few potential extensions and future applications are discussed - including the possibilities of pivoting the method to be more research-driven, or indeed, to drive research.


page 2

page 3

page 4

page 5

page 7

page 9

page 11

page 13


COVID-19 and Online Learning Tools

Distance education has a long history. However, COVID-19 has created a n...

A UK Case Study on Cybersecurity Education and Accreditation

This paper presents a national case study-based analysis of the numerous...

SARS-CoV-2 Impact on Online Teaching Methodologies and the Ed-Tech Sector: Smile and Learn Platform Case Study

In the past few years, the use of new technologies and digital, educatio...

E-learning model for art education: Case study in Iran

According to the characteristics of art, art education according to Bloo...

Teaching Continuity in Robotics Labs in the Age of Covid and Beyond

This paper argues that training of future Roboticists and Robotics Engin...

A Survey of Learning Management Systems and Synchronous Distance Education Tools

Although compelling assessments have been quite frequently examined in r...

Ideals and Virtual Realities

A main step for world progress is to keep sharing ever-present Ideals fo...

1 Introduction

Hackathons and CTFs have long been a staple of Computer Science, with the first software hackathon being held by Sun Microsystems at their JavaOne conference in June 1999 [1], and the notion of security CTFs (or wargames) being around since time immemorial. Over time, they have become increasingly popular, with significant uptake not only by students111https://mlh.io/about,222https://ctftime.org/ctfs, but also in industry. Not only do these constructs offer invaluable frameworks for Problem-Based Learning (PBL), they also pose a potential solution to inspiring and attracting a diverse range of applicants to Higher Education - not least in subjects like computer science where this is especially pertinent [25].

With the rise of distance learning in Higher Education institutions - both due to overall demand, and situations such as the ongoing COVID-19 crisis - it is becoming increasingly necessary to consider a wide variety of innovative and engaging methods for delivering teaching. This paper, therefore, delves deeper into the essence of both hackathons and CTFs, and aims to generalise them such that they could be applied in a non-discipline-specific way. Furthermore, it is believed that these approaches could be altered further to incorporate a more research-driven angle - making them of particular value to research-intensive institutions.

1.1 Hackathons

Hackathons are ‘creative marathons’ where people (in the context of this paper, students) come together in teams to collaborate - learning new ideas, creating something innovative or novel, and sharing these inventions to the other participants. By pulling together three tenets (learn, build, and share), they foster a truly unique learning environment where participants build up not only their hard skills, but also their soft skills (i.e. through presentation, communication, etc.). Traditionally participants work (usually in teams of 4) towards broad challenges set by sponsors (often companies) such as ‘the project that could do the most good for society’, or ‘the best use of some specific technology. As discussed in Section 5, the framework provided by hackathons (Figure 1) can be generalised (along with that of CTFs) to teaching in Higher Education on the whole.

Hackathons are clear examples of PBL [35] – with sponsors’ challenges broadly acting as the problems being solved, and reinforced by the various technical problems faced in the conception and building of solutions. This teaches not only the content (in this case, about the ‘challenges’ set), but also about the thinking strategies and technologies that can be used when solving them (and similar problems encountered in the future) [18]. Further, the collaborative, community-focused environment often fostered at hackathons encourages significant peer mentoring - even between ‘competing’ teams [23]. By virtue of their underpinning nature, hackathons additionally exhibit features of Project-Based Learning (PjBL) [5, 22]

. By bringing together such a diverse range of learning vectors, hackathons are able to stimulate curiosity in learners, and engage diverse ranges of learners.

Figure 1: Diagram showing the overall framework of a general hackathon. Interactions between entities are depicted as arrows.

1.2 CTFs

Historically, Capture the Flag competitions have existed within the context of cyber security, whereby teams333Generally, CTFs are played in teams. However, individual CTFss exist as well, for example, as qualifiers for an in-person finale (e.g. C2C 2017). play against each other to obtain as many flags as possible, by solving challenges, set by the organisers. As an analogy to traditional teaching, the challenges can be viewed as exercises or assignments set by teachers or lecturers. There are two flavours of CTF - attack/defence, or jeopardy style - with the latter being the most popular. In an attack/defence CTF, teams have to protect a (purposefully vulnerable) host, while at the same time attacking other teams’ hosts, which have the same vulnerabilities. In a jeopardy CTF, participants solve a number of stand-alone challenges, across a range of categories (e.g. web, cryptography, binary exploitation, reverse engineering, forensics, networking, etc).

CTFs illustrate best the combination of PBL and online learning. Most CTFs are held online444Some CTFs hold online qualifiers, and in-person finales. The online stage is open to everyone, while the top n teams are invited for the finale, where team sizes are limited, and some financial support is provided for transport and accommodation., and teams can be made up of individuals from across the world, thereby shattering any geographical restrictions in-person events have. In terms of learning outcomes, CTFs provide a good opportunity for participants to sharpen existing skills by focusing on categories they are familiar with, or acquire new knowledge by engaging with new categories, as challenges vary in difficulty levels within categories. Solving CTF challenges means exploiting vulnerabilities in software/hardware and, therefore, this type of competition has a particularly close relationship with the state-of-the-art in the cyber security field. Furthermore, CTFs informally address some of the key factors identified by the Government in the National Cyber Security Strategy 2016-2021: the lack of young people entering the profession, and insufficient exposure to cyber and information security concepts in computing courses [16].

Figure 2 presents an overall framework of a CTF. Organisers (and sometimes sponsors) create a set of challenges (prior to the CTF itself), which teams then work together to solve. Various tools and technologies can aid participants in solving the challenges (e.g. the Kali Linux555https://www.kali.org/ operating system comes preloaded with an extensive suite of software useful for CTFs). Some challenges require programming skills, while others can be solved by conventional methods (e.g. cryptography challenges could require pen-and-paper solutions). Solving a challenge yields a flag, which is then submitted to a scoreboard (omitted from diagram for simplicity purposes), in return for points. This allows participants to have immediate feedback on the correctness of their solution. After the competition ends, teams often write up their solutions for challenges and share them with the wider online community. Learning continues through participants being able to look up challenges which they have not have been able to solve during the CTF, and learn from the write-ups of their peers.

Figure 2: Diagram showing the overall framework of a general CTF. Interactions between entities are depicted as arrows.

While CTFs are generally short events, (they last from a few hours to ~3 days), they have also been adapted and used as tools for teaching and learning in Higher Education, over greater periods of time. Mirkovic and Peterson [27] describe how to adapt CTFs to ‘Class-CTFs’, classroom-based exercises, such that learning is done over a longer period of time (over the length of a course) and all students reach their learning objectives. Flushman et. al. [13] have developed a full introductory computer science course for first-year undergraduates, based on CTFs and linear alternate reality games. Chothia et. al. [8] explain how they used CTF-style exercises for an 11-weeks cyber security course, adding a story throughout the assignments, which increased engagement and attainment levels in students. CTFs have also been used at Further Education level (high-school), in order to try and address the shortage of students who are interested in studying cyber security during their university studies [12]. Moreover, a number of online learning platforms [15, 19, 39] have appeared over the last years, generally aimed at being ‘training grounds’ for CTFs. These platforms could also be used as resources for Distance Learning (DL).

2 HackTheMidlands

Figure 3: A shot from HackTheMidlands 4.0 showing some of the scale - both of sponsors and participants at the event.
Figure 4: Top Left: Learn - a small-scale workshop where attendees are learning about the Twilio API; Top Right: Build - a team of attendees working hard on their project; Bottom: Share - a team of four showing off their project (an automatic selective outliner for pixel art) on an IMAX screen to over 300 people.

HackTheMidlands (HTM)666https://hackthemidlands.com/ is a yearly-hosted hackathon in Birmingham, UK. Founded in 2016, the distinguishing factor from most other hackathons (particularly MLH-partnered ones) is that it is open to everyone. This includes non-students of all ages, and under-18s. This diversity encourages vastly wider interaction between traditionally distanced groups, and facilitates the sharing of a wide range of experiences, knowledge, and opinions, and promotes unlikely collaborations. This diversity of makeup helps to foster creativity and innovation in much the same way Bercovitz and Feldman observed of researchers [3].

Operationally, HTM functions in much the same way as other hackathons. Attendees congregate in a single physical location (allowing a strong community feel to be fostered), and work together over the span of ~24 hours to ‘hack together’ solutions to often broad and real-world challenges or problems set by the sponsors (Figure 3). Generally the sponsors are companies (large or small), but they have also been more charitable or academic institutions such as the University of Birmingham or the BCS777https://bcs.org/. In addition, HTM partners with a number of organisations each year - especially to improve outreach, such as work with Arkwright888https://arkwright.org.uk/ and the Social Mobility Foundation999https://www.socialmobility.org.uk/.

On the first of the two days (preceded by the opening ceremony, in which sponsors pose their challenges) the organisers put on what amounts to a mini-conference - two tracks (one entry-level, one more advanced) of talks and workshops to equip attendees with relevant skills. In addition to this, a mentor scheme - with mentors consisting both of previous attendees and industry experts - helps attendees to access help with little preventative barrier. Whilst the mentor scheme and workshops support the ‘learn’ and ‘build’ aspects of the hackathon, the ‘share’ part is predominantly propped up by the volunteer and organiser teams (Figure 4). They actively encourage all attendees to not only present what they have made, but also what they have learned, and the experience they have had. This in turn means that all learners are reassured that they have learned or built something valuable, which is worth sharing. Given that presenting in front of a 300+-strong crowd is daunting at the best of times, these reassurances lead to a significant proportion of participants giving presentations.

Further, HTM has made a concerted effort to be an informal (as described by Nandi and Mandernach [28]), and approachable learning environment. This not only encourages a diverse range of participants (especially from disadvantaged or underrepresented groups) to take part, but also engages with those in Higher Education who are typically disinterested by traditional teaching environments such as lectures. In addition, by leveraging the concept of delight [21] (simply put, going above and beyond for the learner), it is possible to augment the learning experience by making the students more conducive to learning. This has the additional effect of creating a positive and supportive culture at the event, which leads to peer mentoring, and a willingness to ask for help more readily. This not only improved the overall learning that participants experience, but also helps to improve diversity at the event - a particularly clear issue in computer science. In comparison to the figures set out by Briscoe [7], HESA [17], and Bennaceur et. al. [2] HTM reports a higher proportion of female and non-binary attendees in all but their first year - likely down to the additional steps they took to reach out to and include underrepresented groups (Figure 5).

Figure 5: Graph showing female and non-binary attendance of HackTheMidlands and WhatTheCTF?! compared to general levels set out by [7], [17], and [2].

Participant feedback is at the core of HTM, as it is imperative to act on it to upkeep delight and satisfaction (and thus, maintain a high quality of teaching). To this end, HTM surveyed attendees post-event to ascertain what was positive and negative about the experience, and moreover, what could be improved on the whole. Some of these questions were shared with ‘WhatTheCTF?!’ in order to facilitate comparisons between the two (Section 4).

3 WhatTheCTF?!

WhatTheCTF?! (WTCTF)101010https://wtctf.afnom.net is a yearly-hosted CTF at the University of Birmingham. Its audience is predominantly computer science students, however the event is open to all students of the university. The first iteration of WTCTF was in 2019, when a group of members of the Ethical Hacking Club AFiniteNumberOfMonkeys (AFNOM)111111https://afnom.net, having experienced a number of CTFs [30], decided it was an opportune time to organise such an event for the University. They set out with the goal of creating a beginner-friendly CTF, such that all students can successfully participate, learn new skills and build up their confidence in an area they might not know much about.

One of the aims was to entice people that did not generally have an interest in cyber security, with the hope that they would start considering such a path for their studies or career afterwards. The set of challenges created covered a wide range of topics: web, cryptography, forensics, binary reverse engineering and exploitation, forensics, steganography, and password cracking. Participants could approach the challenges in a ‘horizontal’ way, trying out a small number of challenges from all categories, in order to form a broad idea of what topics cyber security covers, or could use a ‘vertical’ approach, diving deep within one or two categories. In each category, the difficulty of the challenges was signalled by the amount of points assigned to them. Some challenges employed the scaffolding method of learning, whereby solving one challenge unlocked a new, very similar one, with a slightly increased difficulty.

WTCTF ran over seven hours, with a break for lunch, when all challenges were unreachable to participants. This encouraged people to network, exchange ideas and, simply take a break from the grip of the competition. In terms of attendees, WTCTF had a lower number of participants (~35) than HTM, both due to the specialised nature of the event, as well as being open only to University of Birmingham students. Figure 5 shows the percentage of female attendees, as compared to the percentage of women following a cyber security career [32] and percentage of women currently studying Computer Science at degree level [17].

During WTCTF, the main learning vector is collaborative learning, done within teams, and significant amounts of self-directed learning from the participants is also required. The learning and solving components, as described in the general CTF framework (Figure 2) are tightly coupled together, and happen simultaneously. Studies have shown that collaborative learning improves learning outcomes for students (participants), with respect to academic achievement, quality of inter-personal interactions, self-esteem and perceived social support [29, 20, 37]. Moreover, collaborative learning appears to improve retention of traditionally under-represented groups [14, 4]. One of the soft skills most improved by WTCTF are communication skills, as the time limited nature of the event means information must flow quickly and accurately between teammates. Furthermore, the organisers provide direct support during the competition, as participants can just walk up to them and ask questions about challenges. Organisers can choose to provide hints or point participants to specific resources. Due to the smaller scale of the event, organisers can also closely monitor teams, identify any that are struggling, and provide additional help. One of the aims of WTCTF is to make sure newcomers gain new skills in cyber security, and the ability to provide close support is core to achieving this.

Figure 6: Left: Learn & Solve - Attendees work together to solve a challenge in an attempt to elicit the flag; Right: Share - One of the many online write-ups (https://jedevc.com/blog/dctf-secret-writeup/).

4 Comparison

To aid with the generalisation of the two similar frameworks (hackathons and CTFs), and elicit important aspects of these learning environments it is important to consider insights from attendee feedback (Figures on pages 7 and 8) - particularly the shared questions at HTM 4.0 and WTCTF 2020. Though this feedback comes from two specific implementations, or instances, of the frameworks, the lessons learned from each are still invaluable, and help with the abstraction of the underlying concepts. To this end, the responses to each of the seven statements (where X corresponds to either ‘HTM’ or ‘WhatTheCTF!?’),

  1. [label=Q0.]

  2. I met new people at X;

  3. I felt welcome at X;

  4. I learned something new at X;

  5. I enjoyed X;

  6. X has inspired me to attend similar events;

  7. When I needed help, I was able to get it easily; and

  8. I got help from attendees as well as organisers;

on a 5-point Likert scale (Strongly Disagree; Disagree; Normal; Agree; Strongly Agree) will be analysed, with (Q2) and (Q4), and (Q6) and (Q7) considered jointly under broader headings.

4.1 Q1: I met new people at X

Although people are less likely to mingle outside of their team at CTFs than hackathons (likely due to their heightened competitive nature), this can both be alleviated (through enforced ‘no work’ lunch breaks), and the general trend shows that both hackathons and CTFs offer effective environments for networking. Though in Higher Education this may serve a somewhat lesser purpose - as the cohort will likely already know one another quite well - this could help to form connections between groups of peers, and certainly encourages peer-mentoring at the events themselves. This is in large part down to the strong sense of community that is often fostered in these environments, and it is precisely this community feel and welcoming nature that lies core to the effective learning that they offer. This is corroborated in Q2 and Q7, which both highlight the incredibly supportive and inclusive spaces that both HTM 4.0 and WTCTF created.

4.2 Q2/Q4: Engagement/Learning Environment

As mentioned in Sections 2 and 3, the organisers of these events dedicate significant amounts of time during the event to making the participants feel included, and making sure that no one leaves without learning new skills. As explored in the analysis of Q1 above, both hackathons and CTFs aim to foster welcoming environments for their participants. Whether through the normalisation of asking for help (both from ‘teachers’ and peers), peer-mentoring (Q7), or a variety of breakout spaces and side events, they should work to encourage participants to actively engage with one another, and instil in them a sense of belonging. This, along with nurturing the eponymous idea of ‘hacking together’ a functional yet imperfect solution, helps to curtail the significant Impostor Syndrome [34] that many newcomers and ‘veterans’ experience alike, and encourages the sharing of failure as well as success. This is a particularly pertinent issue in Higher Education in the form of reporting bias [11].

Student engagement is paramount to effective learning [26, 38], and the attendee responses to Q4 are testament to the immense engagement that hackathons and CTFs can offer. Despite being learning environments, their departure from traditional lecture-based settings completely radicalises student perception of them. Notably, from a Transactional Analysis (TA) [10] perspective, this represents a shift from Parent-Child transactions in lectures (and other similar settings) to Adult-Adult transactions - treating the students as equals, and eliciting more natural, candid, and unbounded discussion.

4.3 Q3: I learned something new at X

Perhaps key to all learning environments (by definition) is that the students in them learn something new, and the responses to Q3 affirm that this was the case both at HTM and at WTCTF. The overwhelming majority of respondents not only agreed with - but strongly agreed with - the statement. This reinforces the perceived efficacy of these environments as pedagogic tools.

4.4 Q5: X has inspired me to attend similar events

Learning is an ongoing process, and one that - especially as a student in Higher Education - lasts for a prolonged period of time. For a method of teaching to be effective, it must be able to consistently engage with students over this period. If students are deterred by a certain method, their engagement will be lessened, and thus, teaching becomes less effective on the whole. Responses show that not only are the vast majority of students engaged by this type of teaching (Q4), but they are also encourages or inspired to actively seek out similar learning opportunities for themselves (Q5). In the present day, it is quite common for undergraduate computer science students to be willing to travel to a hackathon or CTF most weekends (alongside their studies) - both for the social aspects of the events, but also to learn new skills, build projects, and share what they eventually make (or don’t make, as the case may be). Of course, this isn’t to suggest that all students are, or possibly could be, this engaged, but by removing the barrier of distance - i.e. by running these sessions locally, similar to lectures, it seems likely that this engagement would be much higher.

4.5 Q6/Q7: Help/Support

A robust system of support for attendees is critical for both hackathons and CTFs. Especially in the case that material or technologies are being largely self-taught during the event, attendees often find it hard to move past what may otherwise seem to be trivial obstacles - contributing significantly to feelings of Impostor Syndrome. It is therefore the role of the teacher(s) (organisers/volunteers) to mentor, and students (attendees) themselves to support one another. Without this support net of like-minded individuals in a similar situation, it is unlikely that as much material could be properly learned or applied in the intensely-short time frame over which such events often run. The responses to Q6 and Q7 show that most attendees at both HTM and WTCTF were not only able to access help when it was required, but for the most part, they were also able to access that support from their peers. This not only provides a more supporting and engaging environment for the students, but also equips them with important interpersonal skills, and helps to reinforce their own learning [36, 33].

Figure 7: The first four (of seven) questions posed to participants.
Figure 8: The final three questions posed to participants.

5 Generalisation

Figure 9: The generalised framework, centered around Learn-Apply-Reinforce/Share. Interactions between entities are depicted as arrows.

Though traditionally applied in a computer science (or STEMM) context, it is unequivocally the case that similar approaches can be utilised throughout Higher Education (and potentially further afield). By generalising the two frameworks (set out in Figures 1 and 2) for hackathons and CTFs respectively, it is possible to consider an approach focused on three main stages - Learn, Apply, and Reinforce/Share (Figure 9). Here, a clear parallel can be drawn to the “Learn, Build, Share" mantra of hackathons, or that of “Learn, Solve, Share" for CTFs. This framework can be considered both in terms of a short, high-intensity ‘one-off’ event, or as a project over a longer period of time. The importance of this distinction becomes clearer when considering distance learning (Section 6).

It is also important to consider these ‘phases’ as largely asynchronous (with the exception of sharing work to the entire cohort). Students will regularly move between learning new material - especially that which is self-taught - and using that material to develop their project. Further, through peer mentoring, the ‘Reinforce’ aspect is also prevalent throughout. This fluidity between stages results in a creation/development process very similar to the agile notion of Rapid Application Development (RAD) [24] (albeit applied more broadly, and without a focus on software development), with the students rapidly prototyping aspects of their project (be that pieces of code, paragraphs, sections, or whatever), and using their team as a support network to gather and implement feedback iteratively.

In the ‘Learn’ part of the framework, students learn new materials, skills or technologies, that will be applicable to the challenges or problems that they’ll work towards solving. This could be taught by the teacher(s) in a more traditional way, through talks or workshops ran by the teacher(s) or students, self-taught by the students, or in any other effective way. Further, peer-mentoring plays a significant part in the ‘Learn’ and ‘Apply’ stages, and should be encouraged.

Within ‘Apply’, the students take the knowledge and skills that they’ve gained, and use them to tackle a specific problem. This could be guided or set by the teacher(s) (similar to organisers and sponsors setting challenges in hackathons or CTFs), or the students could be left to define their own. Regardless, it’s generally important to have a broad theme to unobtrusively guide their work. This initial broadness is necessary, so as to not stifle the creativity or imagination of the students. By effectively engaging the students as co-creators [6], it is possible to engage them much more effectively, as they’re likely to be significantly more invested in creating and developing their project. One of the additional benefits around this self-definition of a problem (and thus, engagement with it), is a reduction in the urge to rationalise their creation post-hoc (i.e. fitting a solution to a problem), which in turn encourages students to focus on a much more goal-oriented approach.

The ‘Reinforce/Share’ stage encapsulates the presentation of students work to one another on an individual basis, but also to the whole cohort at the end of the learning. To many students, this may seem daunting, and in order to combat this, it is crucial to reassure them that what they have created need not be a full or perfect solution, and further, that they should talk about both their successes, but also their failures, and obstacles that they faced in their teams. By normalising this honesty and openness, the students are encouraged to learn from these failures, and often gain confidence by seeing that their peers all encountered challenges too. This can prove invaluable when tackling Impostor Syndrome within cohorts. Drawing on inspiration from CTFs, it is also positive to encourage students to create a write-up, not only of their project, but also of their experience and learnings. This can then be shared either online or offline, and in some cases, could contribute more widely to the community.

Though not depicted in the diagram of the framework (for simplicity), feedback is universally important within it - and in all directions - that is,

  • Student Student - through peer-mentoring and collaboration within teams;

  • Teacher Student - through guidance and mentoring;

  • Student Teacher - in the form of feedback about the learning environment;

  • Teacher Teacher - with the teacher acting as a reflective practitioner.

Through these various feedback vectors, the learning environment (and learning experience of the students) is enhanced - especially when the Student Teacher and Teacher Teacher feedback is internalised, and implemented in future practice.

6 Using the Framework in a DL Context

Distance Learning (DL) is increasingly becoming a priority in the Higher Education sector, with increasing demand, and a variety of research being conducted into it [31, 9]. Particularly during the ongoing COVID-19 crisis, it has been brought to the forefront of university education, with institutions having to rapidly develop and deliver online teaching to millions of students worldwide. Within DL, as within regular learning, it is imperative to maintain student engagement, but the state of such engagement is significantly different within this context. Whereas it may be possible to engage students in high-intensity short events like hackathons when held in person, it is much harder to sustain that level of intense focus and engagement at distance. Thus, it is important to consider how best to alter the framework (or at least specific implementation of it) to better-suit these evolving needs.

The vast majority of CTFs are hosted online. In these cases, the onus of collaboration falls on the individual participants, who must form teams, and establish desired channels of communication with one another prior to the event. For the large part, the community aspect of these online CTFs comes both from the camaraderie within teams, and post-hoc from contributing to the broad online community through write-ups. This makes CTFs relatively well-adapted to distance learning, but does place a significantly more active role on participants - who must more proactively engage with the CTF in advance - to sign-up, form teams, decide on communication, and other tasks. When looking to transition the minority of CTFs that are held in-person to an online delivery, however, it may not be as simple as to follow the example set by their online counterparts - the lack of persistent community throughout the event would pose a fundamental change. To this end, additional provision should be made to help participants engage with one another, and to encourage and support typically underrepresented or disadvantaged participants to take part.

In contrast, the vast majority of hackathons are held in person. This is largely because of the aforementioned relative ease of engagement in that format, but also due to the underlying necessity for a community in which it is possible to easily seek help, share your ideas and work, and generally feel welcome. It is clearly not always feasible to run this kind of event in person, however - as highlighted quite pertinently by the ongoing pandemic. Thus, it is essential to adapt the approach such that it may effectively function online. When considering this kind of transition to online delivery (though not necessarily just in the context of hackathons), it is imperative to maintain the community. This requires a firm grasp of the appropriate technology for the job, as it is a relatively delicate art, and can prove easy to get wrong. Compared to community interactions in ‘real life’, there are many more hidden obstacles that students may face when trying to engage with this online community, and if the resulting barrier to entry is too high, it is much more common for students to simply not engage.

One such tool is Discord121212https://discord.com/ - a tool originally aimed towards gamers. Its intuitive design, extensibility (through bots), and flexibility make it a great candidate for cultivating the community required to underpin this kind of learning environment. On Discord, persistent (i.e. present 24/7) communities are called ‘Servers’, which can be set up by anyone. Members can join these servers through an invite link (which makes them easy to disseminate, and reduces the barrier to entry), and engage with one another in both text channels, and voice/video channels. In addition, members can also be assigned ‘roles’ such as ‘Teacher’, ‘Student’, or ‘Mentor’, and these can be customised to grant various permissions. In addition, Discord have recently put together a magnificent step-by-step guide for teachers in all settings - helping them to set up virtual classrooms with ease 131313https://blog.discord.com/how-to-use-discord-for-your-classroom-8587bf78e6c4.

Figure 10: A snippet of HackQuarantine’s mentoring system - allowing participants to self-define as mentors, and contact one another easily for help.

Though Discord is sometimes used at hackathons such as HTM to supplement the in-person learning, they are much less-often used to facilitate one that is entirely online. In response to the COVID-19 pandemic, however, a hackathon called HackQuarantine 141414https://hackquarantine.com/ was set up and ran almost entirely on Discord. Key to much of this was their utilisation of an innovative mentoring system that not only allowed participants to quickly seek help, but also allowed them to self-define as a mentor for specific skills (in their case, programming languages). By using the Discord bot, ‘Zira’ 151515https://zira.gg/, they were able to automatically assign specific Discord roles to users that clicked on specific reactions (Figure 10) under a pinned message. Each of these roles could then be specifically mentioned in the text chat when someone needs help (eg. @Python), which directly facilitated mentoring, and made it much easier for people to ask for help.

Thus, given the right technology, and a little momentum within a cohort, it is clear that the Learn-Apply-Reinforce/Share framework of learning is not only applicable to, but efficacious for, online delivery - with HackQuarantine engaging over 3500 participants, and ‘outputting’ just shy of 250 projects 161616https://hackquarantine.devpost.com/submissions/. One important factor to consider, however, is the time frame over which to run these ‘sessions’. Though in-person hackathons and CTFs tend to last between 8 and 48 hours, it is worth noting that this works largely due to the potential to create a much more focused environment. When students are working from home, for example, it is likely that there will be far more distractions (and thus a lack of focus).

This, coupled with the dispersion of students over a diverse range of time zones, means that the sessions must be run over a longer period of time. HackQuarantine, for example, ran over three weeks, which enabled students to better-engage with the learning. In order to maintain good levels of engagement, it is worth running ‘checkpoints’ throughout this longer period, where students can informally share their current progress and obstacles or challenges with one another - interweaving the ‘Reinforce/Share’ aspect of the framework, and providing a more active feedback loop for the students. Further, a range of ‘side events’ will prove invaluable in maintaining this engagement, and can be used to mimic the breakout spaces provided at in-person events.

7 Conclusion & Future Work

This paper has presented a general pedagogic framework inspired by both hackathons and CTFs. Though these events are typically aimed at computer science or STEMM students, the framework is universally applicable. Split into three distinct sections, ‘Learn’, ‘Apply’, and ‘Reinforce/Share’, it builds on a social constructivistic philosophy and a number of agile principles to create engaging and fast-paced learning environments with a goal-oriented and project-based focus. The applicability of this framework in a distance learning capacity was also explored, with some best-practice technology suggested.

Moving forwards, this framework shows promise, not just from a pedagogic standpoint, but also for research - with bodies such as the Alan Turing Institute hosting ‘Data Study Groups’ 171717https://www.turing.ac.uk/collaborate-turing/data-study-groups/ - week-long ‘collaborative hackathons’ for researchers. It could be possible to organise such events within individual departments or institutions to conduct and output small research projects or studies. Further, there is the potential for them to improve knowledge transfer between experienced and early-career researchers if the groups were formed of researchers at a variety of points in their career.

To conclude, this broadly-applicable framework is currently of particular pertinence - both with growing demand for distance learning or bimodal delivery of material in the Higher Education sector, and in the midst of the COVID-19 pandemic. It is hoped that it will prove a useful resource for educators in Higher Education, and offer a wider outlook on potential tools and technologies to use in the current climate.


Tom is one of two founders of HackTheMidlands, and was the ‘Science Lead’ for HackQuarantine. Andreea runs WhatTheCTF!? as part of the AFNOM Ethical Hacking Society.


  • [1] M. H. Aviram (1999-08) JavaOne’s palm-sized winner. JavaWorld. External Links: Link Cited by: §1.
  • [2] A. Bennaceur, A. Cano, L. Georgieva, M. Kiran, M. Salama, and P. Yadav (2018) Issues in gender diversity and equality in the uk. In Proceedings of the 1st International Workshop on Gender Equality in Software Engineering, GE ’18, New York, NY, USA, pp. 5–9. External Links: ISBN 9781450357388, Link, Document Cited by: Figure 5, §2.
  • [3] J. Bercovitz and M. Feldman (2011) The mechanisms of collaboration in inventive teams: composition, social networks, and geography. Research Policy 40 (1), pp. 81–93. Cited by: §2.
  • [4] L. Berry Jr (1991) Collaborative learning: a program for improving the retention of minority students.. Cited by: §3.
  • [5] P. C. Blumenfeld, E. Soloway, R. W. Marx, J. S. Krajcik, M. Guzdial, and A. Palincsar (1991) Motivating project-based learning: sustaining the doing, supporting the learning. Educational Psychologist 26 (3-4), pp. 369–398. External Links: Document, Link, https://doi.org/10.1080/00461520.1991.9653139 Cited by: §1.1.
  • [6] S. Brand and L. Millard (2019) STUDENT engagement in quality in uk higher education. Student Engagement and Quality Assurance in Higher Education: International Collaborations for the Enhancement of Learning, pp. 35. Cited by: §5.
  • [7] G. Briscoe (2014) Digital innovation: the hackathon phenomenon. Cited by: Figure 5, §2.
  • [8] T. Chothia, C. Novakovic, A. Radu, and R. J. Thomas (2019) Choose your pwn adventure: adding competition and storytelling to an introductory cybersecurity course. In Transactions on Edutainment XV, pp. 141–172. Cited by: §1.2.
  • [9] J. T. Clark (2020) Distance education. In Clinical Engineering Handbook, pp. 410–415. Cited by: §6.
  • [10] W. F. Cornell (2018) Into ta: a comprehensive textbook on transactional analysis. Routledge. Cited by: §4.2.
  • [11] P. Dawson and S. L. Dawson (2018) Sharing successes and hiding failures: in learning and teaching research. Studies in Higher Education 43 (8), pp. 1405–1416. External Links: Document, Link, https://doi.org/10.1080/03075079.2016.1258052 Cited by: §4.2.
  • [12] W. Feng (2016) A" divergent"-themed ctf and urban race for introducing security and cryptography. In 2016 USENIX Workshop on Advances in Security Education (ASE 16), Cited by: §1.2.
  • [13] T. Flushman, M. Gondree, and Z. N. Peterson (2015) This is not a game: early observations on using alternate reality games for teaching security concepts to first-year undergraduates. In 8th Workshop on Cyber Security Experimentation and Test (CSET 15), Cited by: §1.2.
  • [14] E. Fredericksen (1998) Minority students and the learning community experience: a cluster experiment.. Cited by: §3.
  • [15] HackTheBox. Note: https://www.hackthebox.eu/ Cited by: §1.2.
  • [16] P. Hammond (2016) National cyber security strategy 2016 to 2021. London: Her Majesty’s Government. Cited by: §1.2.
  • [17] HE student enrolments by subject area and sex, 2018/2019. Note: https://www.hesa.ac.uk/news/16-01-2020/sb255-higher-education-student-statistics/subjects Cited by: Figure 5, §2, §3.
  • [18] C. E. Hmelo-Silver (2004) Problem-based learning: what and how do students learn?. Educational psychology review 16 (3), pp. 235–266. Cited by: §1.1.
  • [19] Immersive Labs. Note: https://dca.immersivelabs.online/ Cited by: §1.2.
  • [20] D. W. Johnson, R. T. Johnson, and K. A. Smith (1998) Cooperative learning returns to college what evidence is there that it works?. Change: the magazine of higher learning 30 (4), pp. 26–35. Cited by: §3.
  • [21] M. Kim, C. A. Vogt, and B. J. Knutson (2015) Relationships among customer satisfaction, delight, and loyalty in the hospitality industry. Journal of Hospitality & Tourism Research 39 (2), pp. 170–197. Cited by: §2.
  • [22] D. Kokotsaki, V. Menzies, and A. Wiggins (2016) Project-based learning: a review of the literature. Improving Schools 19 (3), pp. 267–277. External Links: Document, Link, https://doi.org/10.1177/1365480216659733 Cited by: §1.1.
  • [23] M. Lara and K. Lockwood (2016) Hackathons as community-based learning: a case study. TechTrends 60 (5), pp. 486–495. Cited by: §1.1.
  • [24] H. Mackay, C. Carne, P. Beynon-Davies, and D. Tudhope (2000) Reconfiguring the user: using rapid application development. Social studies of science 30 (5), pp. 737–757. Cited by: §5.
  • [25] J. B. Main and C. Schimpf (2017) The underrepresentation of women in computing fields: a synthesis of literature using a life course perspective. IEEE Transactions on Education 60 (4), pp. 296–304. Cited by: §1.
  • [26] F. Martin and D. U. Bolliger (2018) Engagement matters: student perceptions on the importance of engagement strategies in the online learning environment.. Online Learning 22 (1), pp. 205–222. Cited by: §4.2.
  • [27] J. Mirkovic and P. A. Peterson (2014) Class capture-the-flag exercises. In 2014 USENIX Summit on Gaming, Games, and Gamification in Security Education (3GSE 14), Cited by: §1.2.
  • [28] A. Nandi and M. Mandernach (2016) Hackathons as an informal learning platform. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pp. 346–351. Cited by: §2.
  • [29] M. Prince (2004)

    Does active learning work? a review of the research

    Journal of engineering education 93 (3), pp. 223–231. Cited by: §3.
  • [30] A. Radu and S. L. Thomas (2015) Organising Monkeys or How to Run a Hacking Club. In Workshop on Cybersecurity Training & Education (VIBRANT15), Cited by: §3.
  • [31] L. Rai (2020) The shifting landscape of open and distance learning. Open Learning: The Journal of Open, Distance and e-Learning 35 (1), pp. 1–3. External Links: Document, Link, https://doi.org/10.1080/02680513.2019.1704233 Cited by: §6.
  • [32] J. Reed, Y. Zhong, L. Terwoerds, and J. Brocaglia (2017) The 2017 global information security workforce study: women in cybersecurity. Frost & Sullivan White Paper. Cited by: §3.
  • [33] C. Reeves, R. Kiteley, K. Spall, and L. Flint (2019) Working with students as partners: developing peer mentoring to enhance the undergraduate student experience. In Mentorship, Leadership, and Research, pp. 27–45. Cited by: §4.5.
  • [34] A. Rosenstein, A. Raghu, and L. Porter (2020) Identifying the prevalence of the impostor phenomenon among computer science students. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE ’20, New York, NY, USA, pp. 30–36. External Links: ISBN 9781450367936, Link, Document Cited by: §4.2.
  • [35] J. R. Savery and T. M. Duffy (1995) Problem based learning: an instructional model and its constructivist framework. Educational technology 35 (5), pp. 31–38. Cited by: §1.1.
  • [36] M. Snowden (2019) Peer mentoring: guaranteed gold?. In Mentorship, Leadership, and Research: Their Place within the Social Science Curriculum, M. Snowden and J. P. Halsall (Eds.), pp. 117–128. External Links: ISBN 978-3-319-95447-9, Document, Link Cited by: §4.5.
  • [37] L. Springer, M. E. Stanne, and S. S. Donovan (1999) Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: a meta-analysis. Review of educational research 69 (1), pp. 21–51. Cited by: §3.
  • [38] J. H. Tai, R. Bellingham, J. Lang, and P. Dawson (2019) Student perspectives of engagement in learning in contemporary and digital contexts. Higher Education Research & Development 38 (5), pp. 1075–1089. External Links: Document, Link, https://doi.org/10.1080/07294360.2019.1598338 Cited by: §4.2.
  • [39] TryHackMe. Note: https://tryhackme.com/ Cited by: §1.2.