Capture the Flag for Team Construction in Cybersecurity

by   Sang-Yoon Chang, et al.

Team collaboration among individuals with diverse sets of expertise and skills is essential for solving complex problems. As part of an interdisciplinary effort, we studied the effects of Capture the Flag (CTF) game, a popular and engaging education/training tool in cybersecurity and engineering, in enhancing team construction and collaboration. We developed a framework to incorporate CTF as part of a computer-human process for expertise recognition and role assignment and evaluated and tested its effectiveness through a study with cybersecurity students enrolled in a Virtual Teams course. In our computer-human process framework, the post-CTF algorithm using the CTF outcomes assembles the team (assigning individuals to teams) and provides the initial role assignments, which then gets updated by human-based team discussions. This paper shares our insights, design choices/rationales, and analyses of our CTF-incorporated computer-human process framework. The students' evaluations revealed that the computer-human process framework was helpful in learning about their team members' backgrounds and expertise and assigning roles accordingly made a positive impact on the learning outcomes for the team collaboration skills in the course. This experience report showcases the utility of CTF as a tool for expertise recognition and role assignments in teams and highlights the complementary roles of CTF-based and discussion-based processes for an effective team collaboration among engineering students.


page 2

page 6


Who does what? Work division and allocation strategies of computer science student teams

Collaboration skills are important for future software engineers. In com...

A network approach to expertise retrieval based on path similarity and credit allocation

With the increasing availability of online scholarly databases, publicat...

Exploring motivation and teamwork in a large software engineering capstone course during the coronavirus pandemic

In the spring of 2020, the Department of Informatics covered a 20 ECTS c...

TAIP: an anytime algorithm for allocating student teams to internship programs

In scenarios that require teamwork, we usually have at hand a variety of...

Strategic Voting in the Context of Stable-Matching of Teams

In the celebrated stable-matching problem, there are two sets of agents ...

Une approche pour mieux appréhender l'altérité en SIC

In this paper, we propose a novel approach that aims: to facilitate the ...

Bioblox 2.5D – Developing an Educational Game Based on Protein Docking

We present the development process of Bioblox2-5D, an educational biolog...

1. Introduction

Team collaboration skills are becoming one of the most desired competencies in the contemporary workforce and considered fundamental to engineering professions (Cabrera et al., 2017). Team collaboration skills are particularly relevant for cybersecurity professionals, whose work requires coordination of collective efforts involving individuals with differing sets of knowledge, skills, and abilities when developing solutions for complex cybersecurity threats (Yoon and Chang, 2021). At the core of effective team collaboration is team members’ abilities to recognize each other’s strengths and fully utilize them by delegating responsibilities to each other in line with each member’s areas of expertise (Hollingshead et al., 2011).

This experience report describes our study of the impact of a computer-human process on enhancing team collaboration. Specifically, we developed a two-pronged team-knowledge sharing process designed to help teams configure their members’ functional roles in a way that maximizes the utilization of individual members’ knowledge. The first step of the process was a Capture the Flag (CTF) exercise in which team members solved challenges in disparate functional areas in the field of cybersecurity and demonstrated their competency in each area. Our framework then used the individual CTF performance data to optimize team assembly and individual role assignment. The second step of the process involved teams’ collective discussion in which team members further shared their professional and academic experiences/expertise beyond what they demonstrated in the CTF exercise and adjusted the initial algorithmic role assignment. This two-pronged process for role assignment illustrates a unique approach integrating both algorithm-based and human-based mechanisms for improving team construction including team assembly and role assignments.

2. Related Work

(a) CTF role categories as shown to the participants
(b) The participant scores summed across all roles/skills (in different curves) vs. time as progressed in the CTF game
Figure 1. CTF game, as shown to the participants during the game

2.1. Team Collaboration and Role Assignment

Research on team collaboration integrates a variety of disciplinary perspectives including psychology, communication, and management. One of the dominant streams of research on team collaboration is concerned with knowledge sharing processes and offers robust theoretical and empirical grounds showing that knowledge sharing processes are a key determinant for effective team collaboration. A theoretical lens we ground our current research on is Transactive Memory Systems (TMS) theory, which explains that groups develop a shared memory among its members regarding who is knowledgeable about what and use the memory system for delegating responsibilities to individual members (Wegner, 1987). Communication and interaction among group members serve as a critical vehicle through which group members learn about each other’s expertise and develop their collective memory system. The more accurately group members recognize each other’s knowledge base, the more likely they align each member’s expertise with his/her roles and responsibilities and share knowledge more effectively (Brendon and Hollingshead, 2004; Hollingshead et al., 2011). Empirical evidence has shown a robust positive relationship between TMS and team performance in a variety of task environments (Ren and Argote, 2011).

An accurate recognition of team members’ expertise requires accumulated shared experiences through which one’s knowledge is demonstrated, performed, or evaluated over time. However, when a team is newly formed, its members rely on initial perceptions and judgements based on limited interactions to gauge each other’s expertise, which often leads to stereotyping or inaccurate assessment of one’s abilities (Yoon and Hollingshead, 2010). Also, individuals can strategically reveal, hide, or exaggerate one’s expertise to meet a variety of individual goals (Yoon et al., 2019). Therefore, it is challenging to ensure accurate recognition of expertise in newly formed teams due to the psychological and social processes that are prone to errors. Although recent empirical studies identify the ways to enhance the accuracy of expertise recognition in teams (Yoon and Zhu, In Press), it is still limited in offering tangible interventions for teams to implement. To address this gap, our current research developed a CTF exercise for teams that could serve as expertise cues based on individual performance data and used an algorithmic mechanism to assemble teams and assign roles for them. We argue that, in combination with team’s verbal discussion, a performance-based algorithm enhances the accuracy of expertise recognition and the alignment between roles and expertise.

2.2. CTF in Education

CTF is popularly used in cybersecurity and engineering, including in classrooms and labs (Beltran et al., 2018; Prabawa et al., 2017), for training security professionals (Cherinka, 2019), for pedagogical research based on live CTF competition (Katsantonis et al., 2017), and for taxonomy framework research for training and education (Knüpfer et al., 2020). While existing practices incorporate CTF for talent acquisition and job placements, e.g., National Cyber League and MITRE (Cherinka, 2019), the analyses of the impact of these methods are lacking, which motivated our work. In particular, we measure the impact of the CTF exercise on knowledge sharing in teams and examine how it serves as a tool for facilitating expertise recognition.

3. Background and Context

Our study was conducted in a three-week long online course ”Virtual Teams”, designed for cybersecurity students (pursuing degrees in cybersecurity in the engineering college) enrolled in a NSF-funded scholarship program111Colorado-Washington Security Scholars Program (CWSSP) is a NSF-funded CyberCorps Scholarship for Service (SFS) program for cybersecurity students at bachelor’s, Master’s, and PhD levels hosted by University of Colorado Colorado Springs and University of Washington Tacoma (Bai et al., 2021).. The scholarship program emphasizes the importance of teamwork skills for future cybersecurity professionals and requires all student scholars enrolled in the program to take the Virtual Teams course. One of the assignments in the course requires students to coordinate and develop areas of responsibilities as a team in a hypothetical situation where a team of cybersecurity professionals needs to develop a security measure for a local mass transit system. This team assignment was designed to offer students an opportunity to apply the theoretical concepts related to team knowledge sharing and expertise recognition to a real-life situation and to experience how best to recognize team members’ expertise and assign roles to each member as a team.

The team assignment required each student team to share each other’s academic, professional backgrounds, research abilities, software and hardware skills, communication skills, and other expertise and using it to assign a functional role to each person. Before integrating the CTF game exercise in our study in the past offerings of the Virtual Teams course, the student teams relied solely on their team discussion when learning about each other’s expertise and decided on who would be most capable of performing which functional role. By adding the CTF exercise, student teams were offered an additional data point regarding their team members’ abilities through the CTF exercise and had an opportunity to combine the CTF performance data with their team’s discussion to determine each member’s functional roles. To test the effect of the CTF exercise on their team knowledge sharing and role assignment, we compared student teams with and without the CTF exercise in terms of their response to a questionnaire that measured their team collaboration efficacy.

Participants      The participants were university students in the engineering college pursuing security degrees. In 2021, ten students (nine males and one female; one Ph.D. student, four Master’s, and five undergraduates) participated in the Virtual Teams course and the CTF exercise. In 2020 where the CTF exercise was not implemented, seven students (six males and one female; one Ph.D. student, two Master’s, and four undergraduates) participated in the Virtual Teams course.

Because our CTF’s main purpose is for education and teaching, our research did not require approval from the Institutional Review Board (IRB).

Figure 2. Overview including the CTF and team discussion for team assembly (top row) and the role assignment (bottom row). The CTF score matrix is used for team assembly, resulting in including the team information as its elements. The initial role assignment using the CTF scores () and the team assembly () results in , followed by the team discussion to update the role assignment in ; both and include the role information as their elements.

4. CTF Game Design and Build

We designed and built a CTF game in order to use it for our team assembly and the initial role assignments. The CTF game performances across different areas of functional role provided us with the information of the participants’ strengths in those areas. The six role/skill categories were designed for the team project required in the class and are shown in Figure 0(a). Capturing flags and earning the corresponding scores involved both knowledge/Internet search engine searching and hands-on software-based tasks, ranging from identifying a command to analyzing code or data.

We built on the open-source CTFd platform because of its feature-rich design and easiness to develop and use 

(Karagiannis et al., 2020). Our CTF game has also been alpha-tested by our team and beta-tested by four participants not involved in the research, development, and execution of this project. The participants played the CTF game for one hour (while the CTF is designed for 3-4 hour duration) on the first day of the Virtual Teams course. Due to the short duration, the CTF has been designed for a minimal learning curve from the participants and only required a computer, Internet connectivity/browser, a video-conferencing platform (Zoom for real-time interactions and coordination for the Coordination flags), and a camera (for QR code for a steganography problem).

While it was an individual game (i.e., the participants had their own scores), the CTF was designed for better engagement and real-time coordination and interactions between the participants. Specifically, the Coordination category had flags involving real-time interactions among the participants (e.g., using their pre-shared codes to collectively solve a flag or finding a participant satisfying a condition). The CTF platform also showed a real-time scoreboard (the final scoreboard, excluding the identifiers/nicknames, is shown in Figure 0(b)) and a compressed version of the real-time scores and top performers in their main screen accessing and solving the challenges. In order to encourage trying and tackling multiple role/skill categories, the CTF unlocked more advanced flags if easier flags were solved and offered bonus incentive points for solving a greater number of role/skill categories (these incentive points have not been used in our algorithms for team assembly and initial role assignments in Section 5). Our previous publication (Zhang et al., 2022) describes in greater details about our design of the CTF to incorporate interaction and coordination on the CTF flags, which is a novel approach and design for the CTF games; this paper builds on our previous work but focuses on the CTF application for the team construction and facilitation after the CTF design/build, and the rest of the section does not overlap with our previous publication.

5. CTF for Team Assembly and Initial Role Assignment

Given that the CTF game can serve as an objective method for observing and measuring individuals’ skills and expertise levels, we used the CTF performance data to assemble teams and assign the functional roles within each team. For the role assignment, the CTF provided a starting point, from which the team members could further discuss (e.g., share other inputs and relevant information such as preferences) and revise each member’s roles and deliverables (Section 6).

5.1. Framework for Using CTF for Team Assembly and Role Assignment

Below we present a mathematical framework for using the CTF for team assembly and role assignment. Participants: There is a set of number of participants . We use an integer index to identify the participants, as opposed to the participants’ names/nicknames, for the participant anonymity, e.g., if there are fifteen participants (), then . Roles: There is a set of functional roles . For example, in our CTF game, is a subset of { IN, DE, AN, IM, TE, CO }, since the CTF game implementation designed for engineers and cybersecurity students included six roles of Investigation (IN), Design (DE), Analysis (AN), Implementation (IM), Testing and Evaluation (TE), Coordination (CO) and the actual team construction and role assignment can require/cover less than those six roles. Teams: We form number of teams which form a set and, from each team, we assign one functional role to each participant. CTF Score (CTF Result): The CTF objectively measures the participant scores for each functional role to indicate the skill and expertise level. The score matrix is of size , where the element on the -th row and -th column corresponds to the -th participant (the -th element in ) and the -th role (the -th element in ), i.e., the -th participant’s score for the -th role category. Capacity of Participant/Team: A participant’s capacity is the sum of its scores across all role categories. For example, if a participant can assume all the roles and has the bandwidth to do so (in our implementation for our study, we assume that executing a role is significant and limit this to one role as described in Section 5.2), then the capacity directly affects the participant’s contribution to the team. A team’s capacity is the sum of the CTF scores for all role categories for all team members (a subset of the entire participants).

CTF for Team/Role Assignment: We take the score matrix and assemble the team and then assign the roles within each team, as described in Figure 2

. The team assembly results in a vector

of length and whose element is an element in indicating to which team the participant belongs. The role assignment takes the team assembly result and the score matrix to assign roles to each team members, resulting in a vector of length and whose element is an element in . including the elements of team information is the team assembly solution, while including the elements of role information is the role assignment solution. (In Section 6, gets updated to , both of which have the same format for the role assignment.)

Functional Role Pre-Assigned Based on CTF Academic


Relevant coursework

Part- and full-time Internships

Knowledge of current literature; Literature search; Research Integration; System analysis
Software and
Hardware Skills

Experience in programming languages, software tools, or hardware (e.g., networking, router or server, embedded platforms)
Communication Skills

Leadership experience; Teamwork skills; Relationship management
Areas of
Assigned Functional Role
Table 1. Team Knowledge Sharing Assignment Work Sheet for Team Discussion-Based Role Assignment (as presented to the students but compressed and re-formatted to fit this manuscript)

5.2. CTF for Team/Role Assignment Implementation and Rules

We implemented the CTF-informed team assembly and role assignment for our study. While we explain our implementation and envision it being directly applicable to many team scenarios, other implementations can adapt the framework in Section 5 in different ways (e.g., different metric and algorithms) to better accommodate their distinct team requirements and context.

Solution Feasibility      The following are the set of assumptions and rules we used for our framework and thus helped define the solution feasibility. 1) Roles are essential. Once the set or the subset of roles is established for each team (setting the expectations and requirements for a team), all roles need to be filled. Otherwise, the team does not meet the expectations, is incomplete, and not fully functional. 2) Executing the roles is substantial. One person can fulfill and execute one role. Having one person execute multiple roles is excessive and infeasible. 3) Roles require expertise. Roles require positive CTF score in the role. The participants not demonstrating any capability in the role and having zero point is not appropriate for the role. 4) Participants can help each other. Multiple people can work on the same role in a team.

These assumptions yield the feasibility requirements for the team assembly and role assignment, i.e., viable solutions satisfying the assumptions exist. For example, the feasibility requires that the number of teams be smaller than the number of total participants divided by the number of roles so that there are at least number of participants in each team to take on all the roles, i.e., where is the floor of or the greatest number smaller than . Also, the number of people having the capability/expertise for any role needs to exceed the number of teams because, otherwise, there is at least one team which is incomplete and does not have the appropriate participants for the required roles.

Solution Selection

     In team assembly, we prioritized the fairness between the teams in their expertise and capabilities, i.e., reduce the variance across the teams, and reduced the expertise/capability discrepancy across the team members within the team. After assembling the team members, we assigned the roles to the team members to maximize the

team score aggregating the CTF scores for the member/role combinations while assuming zero scores for the roles not assigned.

Team Assembly      Each team took turn in selecting the participant with the highest score in the remaining/non-selected participants. The turn taking reversed in direction in each round of selection, e.g., the team selecting the last in the previous round selects the first in the next round. We have considered an alternative scenario of assembling teams so that it minimizes the between-team variance in capacities, but such an approach increases the between-member variance in each team. The algorithm for our approach can also find a solution in polynomial time in O(). We further analyze our team assembly approach in Section 7.1.

Initial Role Assignment      Given each team resulting from the Team Assembly, we applied the Hungarian algorithm to assign the roles to the team members. Hungarian algorithm (Kuhn, 1955)

is a popular combinatorial optimization algorithm for assignment and finds the optimal solution in polynomial time. Assuming that we keep only the score of a participant for the role that he/she gets assigned (e.g., the scores for the other roles for the participant effectively becomes zero), Hungarian algorithm in our implementation maximizes the aggregate score/contributions from the participants. Since the Hungarian Algorithm minimizes the cost, we pre-processed

to generate another matrix, which is a sub-matrix of (selecting only the participants/rows within the team according to ), negating the scores for all elements (i.e., if each element is and is the maximum score element in the newly generated matrix, we replace it with ) and appending extra zero-element columns to make it a square matrix (if the number of team members exceed the number of roles).

6. Team Discussion for Role Assignment

After assembling teams and assigning roles to each team member based on the CTF-based algorithm, we implemented a team-based discussion as part of the two-pronged approach to team knowledge sharing. Students as a team were instructed to submit in writing their responses to the following prompts: (1) reflect on their role pre-assigned based on the CTF and how accurately it represented their expertise, (2) share their backgrounds and expertise beyond what they have demonstrated in the CTF exercise by filling out the worksheet in Table 1, (3) consider if their initially assigned roles change based on the team discussion, and (4) reflect on how both the CTF and team discussion helped them recognize and utilize team members’ knowledge for the project. In Section 7.3 and Section 7.4, we report the results of a qualitative analysis of student teams’ submission of this assignment as well as a quantitative comparison of course learning outcomes between the students who experienced the CTF exercise and those who did not.

7. Results and Analyses

7.1. CTF Results

In this section, we describe the results for CTF session, team assembly, and initial role assignment for our beta-testing involving 4 participants (simpler) and the actual CTF session involving 10 participants (larger number and matrices). While we show the variable values for the simpler beta-testing session, for the later CTF session results, we omit the variable assignments and values for the space limitation (e.g., is a 1010 matrix), but rather focus on the description and insights of the results and analyses. While the beta-testing CTF session was solely for testing and fine-tuning the CTF itself, as described in Section 4, the actual CTF session with the participants in the Virtual Teams course resulted in team assembly and role assignments for all participants.

CTF Results for Beta-Testing (4 Participants, 4 Roles, and 1 Team)      From the beta-testing session including four participants (), we formed one team of four team members (all participants are in one team, i.e., ) where the team requires the roles of Investigation (IN), Design (DE), Implementation (IM), and Coordination (CO), i.e., we used , = { IN, DE, IM, CO }. The selection of the roles is application and context dependent, for example, the Analysis (AN) and Testing and Evaluation (TE) roles can be outsourced or be added in a later stage for expanding the team. As a result of the CTF game, . The team assembly resulted in all the four participants being in the same team (team “1” in this case), i.e., . The initial role assignment to be , i.e., the first participant takes on the Design role; the second participant takes on the Investigation role; the third participant takes on the Coordination role; and the fourth participant the Implementation role. The team score aggregating the scores for these participant/role combinations while assuming zero scores for the roles not assigned (in contrast to the “team capacity”) is: 257+103+238+61=659.

The required roles for the team affect the team assembly and the initial role assignment. For example, if we have greater flexibility in the roles needed for the team and can choose = { DE, AN, IM, CO } replacing IN with AN, then we actually can have greater team performance in the aggregate score. The CTF scores become: . The team assembly resulted in still the same team (team “1” in this case), i.e., . The initial role assignment resulted in . Compared with the earlier case with the aggregate team scores increases from before (659) to 257+290+61+55=663 using . The third participant also had higher role-aggregate score than the second participant, which would have been different if there were multiple teams.

CTF Results for Our Main Study (10 Participants, 5 Roles, and 2 Teams)      From the CTF session at the start of the Virtual Teams course involving ten participants (), we formed two teams of five team members each (i.e., ) where the team required the roles of Investigation (IN), Design (DE), Analysis (AN), Implementation (IM), and Coordination (CO), i.e., we used , = { IN, DE, AN, IM, CO }. The selection of the roles is application and context dependent, for example, the Testing and Evaluation (TE) roles can be outsourced or be added later when expanding the team. As a result of the CTF game, the scores for each participant for each role is given and constructed (omitted for space reason). Two teams with feasible/valid solutions for team assembly and initial role assignment were constructed where, for each team, every team member took a distinct role. The team score aggregating the scores for these participant/role combinations was 179+350+115+81+264=989 for Team 1 and 253+325+95+116+8=797 for Team 2. This team assembly and the initial role assignment were used for the rest of the Virtual Teams Course team exercises, including the further role assignment for updating the roles.

(a) CTF results: team capacity after CTF-based team assembly
(b) CTF results: team score after CTF-based role assignment
(c) Evaluation results comparing 2020 (no CTF) vs. 2021 (with CTF-based team assembly and initial role assignment). Q5 and Q6 are specific to the CTF components newly added in 2021.
Figure 3. Results and Analyses

7.2. CTF Algorithm Comparison Analysis

We analyzed the CTF results in team assembly and initial role assignments. While the role assignment had a clear goal and definition for optimization to maximize the team score and to maximize the aggregate scores/contributions from the team members, the team assembly incorporated the trade-off between increasing the team capacity and decreasing the spread/discrepancies across the team members within a team in Section 5.2. Therefore, we compared our approach with the following alternatives: Averaged/Balanced Case) tries all possible team-assembly cases and computes the average across those cases, while Max Capacity for Team 1 maximizes and prioritizes the capacity of Team 1 while the rest of the lower-capacity participants become Team 2. The Averaged/Balanced Case yielded the same capacities for both Team 1 and Team 2 and corresponded to the bound for making the team as fair and balanced as possible; however, this average case was infeasible as it includes capacities and scores in fraction. Figure 2(a) shows the team capacity (the sum of all of the participant capacities) after the team assembly. It shows that our approach was close to the Averaged/Balanced Case where Team 1 only outperformed by and Team 2 only lagged by . In contrast, the most imbalanced case for Max Capacity for Team 1 had a discrepancy of +31.645% for Team 1 and -31.645% for Team 2.

We also verified that, once the teams are assembled, the polynomial-time-efficient Hungarian algorithm maximizes the team score by comparing it with all other possible cases for role assignments (exhaustive search on the role assignments). Figure 2(b) shows the team scores (only counting and summing the participant scores for his/her role) after the Hungarian Algorithm was applied for role assignments. While the Averaged/Balanced Case had the team score difference of between Team 1 and Team 2, our approach had a difference of between Team 1 and Team 2 team scores and Max Capacity for Team 1 has a difference of between the two teams’ scores.

7.3. Qualitative Analysis of Team Discussions

In their qualitative reflections on the CTF exercise and the team discussion for the course assignment, both teams indicated that the initial role assignment based on the CTF performance was an accurate representation of their team members’ expertise. One team made no adjustment to their initial role assignment, while the other team made only one role swap between two members. Students’ reflections suggest that the team discussion facilitated expertise recognition and role assignment in two ways. Firstly, the team discussion helped them confirm and reinforce each member’s expertise demonstrated in the CTF exercise. For instance, one team stated that ”we discussed our backgrounds in-depth and learned that our backgrounds lent themselves to the functions we had been assigned by the algorithm.” The other team also stated that ”the CTF provided a starting point to compare our roles against our skills. The team discussion was an effective mechanism for hashing out any other details.” Secondly, the team discussion helped teams fine-tune their understanding about the members’ relative expertise in different role categories and offered an opportunity for members to express their preferences. The team who switched roles between two members stated that ”our discussion helped us understand where each person would fit best when also considering personal preference along with skill set. We also discussed where we felt the team had holes or less experience and where we had overlapping experience. [Member A]’s experience in classwork with software projects is a better fit for Implementation. [Member B]’s work experience as an information security analyst is better suited for Analysis.” These student reports suggest that the algorithmic and communicative processes are complementary and in combination enhance the accuracy of expertise recognition and role assignment.

7.4. Quantitative Analysis of The Impact of CTF on Team Collaboration

To gauge the impact of the CTF exercise on students’ learning in team collaboration skills, we compared student responses to an end-of-the-course questionnaire between the course with and without the CTF component. All other elements of the two versions of the course (e.g., reading, individual assignments, lectures, discussion forums) were exactly the same, and the incorporation of the CTF exercise was the only differentiating factor. In the questionnaire, students were asked to indicate their level of agreement in a scale of 1-5, 1 being ”strongly disagree” and 5 being ”strongly agree”, to four statements: Q1 stated ”My understanding of team processes has improved after completing this course.”, Q2 ”After taking this course, I feel more confident about my own team work skills.”, Q3 ”After taking this course, I recognize the value of team efforts in the field of cybersecurity more significantly.”, Q4 ”The course contents (e.g., readings, lecture, assignments) were applicable to my work.” Two additional questions specific to the CTF exercise were asked to the students who experienced the CTF exercise: Q5 stated ”The results of the CTF exercise helped me learn about my team members’ backgrounds and expertise.”, Q6 ”The results of the CTF exercise helped my team members assign functional roles to each other effectively in the team assignment.”

The comparison222One-way ANOVA was performed to test the statistical significance when comparing means between the two groups. However, due to the small sample size (N=5 in each group), the statistical procedure is deemed inappropriate, and therefore, its results are not fully reported in the paper. The results of one-way ANOVA are available upon request. of the means between the two groups shows a positive impact of the CTF exercise in all four questions Q1-Q4, as shown in Figure 2(c). Students who experienced the CTF exercise showed higher ratings in their understanding of team processes (mean with CTF=5.0, mean without CTF=4.2), their confidence in team collaboration skills (mean with CTF=4.2, mean without CTF=4.0), the appreciation for team efforts (mean with CTF=4.6, mean without CTF=4.2), and the applicability of course materials (mean with CTF=4.2, mean without CTF=4.0) than those who did not experience the CTF exercise. Students’ average ratings for the CTF exercise (Q5 mean=4.2; Q6 mean=3.8) also indicate a positive impact of the CTF exercise on expertise recognition and role assignment.

8. Conclusion

Team collaboration skills are one of the most desired competencies in the contemporary workforce including the field of cybersecurity. While the benefits of teamwork lie in the diversity of skill sets that individual members can bring to collective tasks, teams often fail to recognize individual members’ strengths and expertise or to fully utilize them. In an effort to enhance expertise recognition whereby team members can assign roles to each other in line with their areas of expertise and share each other’s expertise more effectively, we examined in our work if the two-pronged approach—a CTF-based algorithm and a team discussion—positively affects expertise recognition in teams. Our results show that our computer-human process is effective for team assembly and role assignment and made a positive impact on the participants’ learning of team collaboration skills and the course materials.


This material is based upon work supported by the National Science Foundation under Grant No. 1922410.


  • Y. Bai, S. Chang, K. Lew, and S. Wuthier (2021) Enhancing cybersecurity education and workforce through colorado-washington security scholar program. CCSC Northwestern Regional Conference (), pp. . Cited by: footnote 1.
  • M. Beltran, M. Calvo, and S. Gonzalez (2018) Experiences Using Capture The Flag Competitions to Introduce Gamification in Undergraduate Computer Security Labs. In 2018 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA, pp. 574–579 (en). External Links: ISBN 978-1-72811-360-9, Link, Document Cited by: §2.2.
  • D. P. Brendon and A. B. Hollingshead (2004) Transactive memory systems in organizations: matching tasks, expertise, and people. Organization Science 15 (6), pp. 633–644. Cited by: §2.1.
  • I. Cabrera, J. Villablon, and J. Chavez. (2017) Blending communities and team-based learning in a programming course. IEEE Transactions on Education 60 (4), pp. 288–295. Cited by: §1.
  • R. Cherinka (2019) Using Cyber Competitions to Build a Cyber Security Talent Pipeline and Skilled Workforce. In Intelligent Computing, K. Arai, S. Kapoor, and R. Bhatia (Eds.), Vol. 857, pp. 280–289 (en). Note: Series Title: Advances in Intelligent Systems and Computing External Links: ISBN 978-3-030-01176-5 978-3-030-01177-2, Link, Document Cited by: §2.2.
  • A. B. Hollingshead, N. Gupta, K. Yoon, and D. P. Brandon (2011) Transactive memory: past, present, and future. In Theories of team cognition: Cross-disciplinary perspectives, M. Letsky (Ed.), pp. 421–455. External Links: Document Cited by: §1, §2.1.
  • S. Karagiannis, E. Maragkos-Belmpas, and E. Magkos (2020) An Analysis and Evaluation of Open Source Capture the Flag Platforms as Cybersecurity e-Learning Tools. In Information Security Education. Information Security in Action, L. Drevin, S. Von Solms, and M. Theocharidou (Eds.), Vol. 579, pp. 61–77 (en). Note: Series Title: IFIP Advances in Information and Communication Technology External Links: ISBN 978-3-030-59290-5 978-3-030-59291-2, Link, Document Cited by: §4.
  • M. Katsantonis, P. Fouliras, and I. Mavridis (2017) Conceptual analysis of cyber security education based on live competitions. In 2017 IEEE Global Engineering Education Conference (EDUCON), Athens, Greece, pp. 771–779 (en). External Links: ISBN 978-1-5090-5467-1, Link, Document Cited by: §2.2.
  • M. Knüpfer, T. Bierwirth, L. Stiemert, M. Schopp, S. Seeber, D. Pöhn, and P. Hillmann (2020) Cyber Taxi: A Taxonomy of Interactive Cyber Training and Education Systems. In Model-driven Simulation and Training Environments for Cybersecurity, Vol. 12512, pp. 3–21 (en). Note: Series Title: Lecture Notes in Computer Science External Links: ISBN 978-3-030-62432-3 978-3-030-62433-0, Link, Document Cited by: §2.2.
  • H. W. Kuhn (1955) The hungarian method for the assignment problem. Naval Research Logistics Quarterly 2 (1-2), pp. 83–97. External Links: Document, Link, Cited by: §5.2.
  • H. W. Prabawa, E. Junaeti, and Y. Permana (2017) Using Capture the Flag in Classroom: Game-based Implementation in Network Security Learning. pp. 6 (en). Cited by: §2.2.
  • Y. Ren and L. Argote (2011) Transactive memory systems 1985–2010: an integrative framework of key dimensions, antecedents, and consequences.. The Academy of Management Annals 5 (1), pp. 189–229. Cited by: §2.1.
  • D. M. Wegner (1987) Transactive memory: a contemporary analysis of the group mind. In Theories of group behavior, G. R. Goethals (Ed.), pp. 185–205. Cited by: §2.1.
  • K. Yoon and A. B. Hollingshead (2010) Cultural stereotyping, convergent expectations, and performance in cross-cultural collaborations. Social Psychological and Personality Science 1 (2), pp. 160–167. External Links: Document Cited by: §2.1.
  • K. Yoon, K. Knight, and D. Martin (2019) Deceiving team members about competence: its motives and consequences. Western Journal of Communication 83 (2), pp. 210–231. External Links: Document Cited by: §2.1.
  • K. Yoon and Y. Zhu (In Press) Social media affordances and transactive memory systems in virtual teams. Management Communication Quarterly. External Links: Document Cited by: §2.1.
  • K. Yoon and S. Chang (2021) Teaching team collaboration in cybersecurity: a case study from the transactive memory systems perspective. In 2021 IEEE Global Engineering Education Conference (EDUCON), Vienna, Austria, pp. 841–845 (en). External Links: Document Cited by: §1.
  • K. Zhang, S. Wuthier, K. Yoon, and S. Chang (2022) Designing and using capture the flag for coordination and interaction in engineering education. In 2022 IEEE Global Engineering Education Conference (EDUCON), Vol. , pp. 1555–1560. External Links: Document Cited by: §4.