TecCoBot: Technology-aided support for self-regulated learning

11/23/2021
by   Norbert Pengel, et al.
0

In addition to formal learning at universities, like in lecture halls and seminar rooms, students are regularly confronted with self-study activities. Instead of being left to their own devices, students might benefit from a proper design of such activities, including pedagogical interventions. Such designs can increase the degree of activity and the contribution of self-study activities to the achievement of learning outcomes. Especially in times of a global pandemic, self-study activities are increasingly executed at home, where students already use technology-enhanced materials, processes, and digital platforms. Thus we pick up these building blocks and introduce TecCoBot within this paper. TecCoBot is not only a chatbot, supporting students in reading texts by offering writing assignments and providing automated feedback on these, but also implements a design for self-study activities, typically only offered to a few students as face-to-face mentoring.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

09/21/2018

Effects of Automated Interventions in Programming Assignments: Evidence from a Field Experiment

A typical problem in MOOCs is the missing opportunity for course conduct...
04/20/2020

Influence of COVID-19 confinement in students performance in higher education

This study explores the effects of COVID-19 confinement in the students ...
10/12/2019

RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities

This paper presents a platform called RiPPLE (Recommendation in Personal...
03/05/2021

An automated approach to mitigate transcription errors in braille texts for the Portuguese language

The quota system in Brazil made it possible to include blind students in...
05/26/2021

Blockchain-Based Approach to Foster Student Engagement on Campus

On-campus activities like positions of responsibility in campus amenitie...
05/10/2010

Recognizability of Individual Creative Style Within and Across Domains: Preliminary Studies

It is hypothesized that creativity arises from the self-mending capacity...
11/26/2018

Challenges in the Automatic Analysis of Students' Diagnostic Reasoning

Diagnostic reasoning is a key component of many professions. To improve ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Self-study activities are an integral part of today’s workload in higher education [1]. However, studies on the workload of students show that self-study activities plays a subordinate role and that too little time is spent on independent learning [2]. One approach that is widely recognised as effective in positively influencing students is mentoring [3]. However, individual face-to-face mentoring requires a lot of time and effort. Especially in higher education, where a large number of students is often supervised by only a small number of lecturers, it is impossible to provide holistic mentoring to all students.

As part of the German research project tech4comp, which searches for design concepts to scale mentoring activities by utilising technology, we developed a chatbot that is able to provide informative feedback for a large number of students fast, as part of self-study activities. This is even more important, as students currently cannot be at a university due the corona pandemic 2020 and thus cannot make use of face-to-face mentoring.

Various models describe the knowledge and research process in (educational) design research. Based on a review of existing models of design research, a generic model was developed [7], which contains three core phases in a flexible and iterative structure: 1) analysis/exploration, 2) design/construction, and 3) evaluation/reflection, which we currently use in our work.

In this paper we address the results of the first two phases, while our focus is on a prototype for a technology-aided support system that provides students with automated feedback in form of knowledge graphs on submitted writing assignments via a chatbot.

2 Requirements and Related Work

2.0.1 Evaluating student needs

In accordance with the multi-method approach, we use both qualitative and quantitative methods. We used the “Teaching Analysis Poll (TAP)” as a qualitative method for exploring students needs in terms of learning. Actually it is an instrument for course evaluation due to open-ended questions. This method has a low degree of structuring and thus the greatest possible exploratory character of result generation. In 2019 296 of 628 enrolled students were surveyed in an educational science seminar in teacher education at Leipzig University. The test persons were asked about the parts of teaching-learning setting and activities of teaching staff of universities that either promoted or hindered learning. The evaluation of the results was also carried out with a qualitative method of empirical social research, the qualitative content analysis in direct coding on the data material. A coding guideline prepared for TAP111guideline:https://https://epub.uni-regensburg.de/35604/ served as a structuring aid for this. The structuring analysis was carried out in several rounds in which the data material was compared with the individual categories. These TAPs took place in all seminar groups in the above mentioned period and offered qualitative intermediate feedback on the courses, which was already condensed to statements that were accepted by a majority in a Think-Pair-Share procedure. Altogether, a clear picture of the learning situation as perceived by the students could be derived. In addition to the large number of texts to be read in the seminars, lack of focus on the content and its structure, which is difficult to recognise, were mentioned as obstacles to learning. Following this poll, a quantitative survey with a closed questionnaire is planned. The aim of this survey is again to determine students’ need for support and to test hypotheses derived from the qualitative data (in this respect it is a sequential research design). Accompanying this, however, the theoretical research that also takes place in this phase is important for structuring the initial situation of research process [6].

2.0.2 Mentoring

It is understood as dyadic relationship between mentor and mentee and can be an effective pedagogical measure to support learners in their learning process [4]. A holistic view of mentoring includes cognitive, motivational and emotional dimensions of this interaction, while ensuring learner autonomy and self-determination.

If mentoring is to be designed to promote and motivate learning, the findings of learning research are important, especially on self-regulated learning and the related metacognitive regulation. The greatest effects of mentoring can be expected when optimal conditions for effective learning processes are in place, including frequent and high-quality feedback, practice and consolidation of what has been learned. Currently, we focus on mentoring to support domain-specific learning processes, especially knowledge.

In order to systematize the concept of mentoring, which is presented in the research literature as being not very homogeneous, we refer to its functional criteria [8], [9]. These include (among other things) support for the transfer of scientific expertise, which aims to promote the students’ knowledge relevant to the chosen subject area. The development of the prototype will focus on knowledge of the students.

2.0.3 Generating feedback

For mentoring to be accepted by students, the quality of feedback is important [10]. Findings from empirical educational research clearly show that feedback can have a positive influence on learning and development processes. In addition to personal factors of the sender and recipient (e.g. expertise, self-efficacy, attribution patterns) and situational factors (cultural conditions, binding character), most important is to design the transmission of feedback in a way that promotes development of the student [11, 13, 12]. A distinction can be made between informative and controlling feedback [14]. The latter impairs intrinsic motivation by building up pressure and thus, according to the theory of self-determination [14], a condition for sustainable learning. According to a synthesis of meta-analyses of feedback in schools [15], informative feedback is also considered more effective than praise or punishment. Important for the effectiveness of feedback is that it offers learners clues or encouragement and relates directly to learning goals.

Accordingly, we provide writing assignments and automated feedback for accompanied self-study activities. The communication of these tasks and feedback are transmitted with a chatbot and is characterized by an understandable everyday language without academic language in order to increase the possibility of reflection and also to stimulate cognitive, metacognitive and motivational strategies. Through this intervention communicated by chatbot, students should understand content and integrate it into their existing knowledge structure. By activating metacognitive strategies, it should be possible for them to better monitor their own learning process.

Figure 1: Workflow of the mentoring prototype for text feedback

3 Approach and Implementation

3.0.1 Feedback with T-MITOCAR

We are using the software T-MITOCAR to provide individual feedback on writing assignments. T-MITOCAR constructs re-representations of knowledge from prose text through a computer-linguistic analysis, without the need to incorporate an external knowledge base [16, 17]. The resulting knowledge graphs look similar to mind maps and may be modeled from texts passages, like book chapters, or whole texts, like written by students. In our case, the knowledge graph of a students text is compared with the graph of reading assignment text (reference text) to generate concrete feedback. For instance, this feedback can inform students about which key concepts are part of their text model and also show how these are interlinked. It can also inform about which concepts overlap with the model of the reference text, which are different, and about how the concepts are linked in the assignment text. It should also be noted that similarity to the assignment text depends on content and on the writing task, so similarity is not always a goal parameter [18, 19, 20, 21]. All this information can aid reflection and text revision and, importantly, in combination with active writing it can foster an in-depth study of the topic in self-regulated learning [22].

3.0.2 Feedback provided by Chatbot

Chatbots are conversational interfaces, allowing humans to interact with software using natural language. Therefore chatbots are known to be relatively easy to use and intuitive with a low entry barrier and good accessibility [24], which makes them suitable for university context with users on different levels of digital literacy. Classic mentoring typically takes place within conversations between a mentor and mentee. The conversational nature of chatbot interactions thus seems to be an easy choice to digitally emulate mentoring processes. In contrast to most static graphical interfaces a chatbot is inherently adaptive to some degree since it reacts to the users input in the course of conversation, depending on the chatbot’s underlying knowledge base. To a large extent the efficacy of mentoring lies in its adaptiveness towards the mentee and his or her needs and competences [25]. For this a chatbot can fulfill an important requirement for technology-aided mentoring although we still need to determine the elements the chatbot needs to adapt to through the course of conversation. Other elements known to be key factors for good mentoring are the trust and respect within a mentoring relationship [26]. Reportedly chatbots are potentially trusted and felt safe to share information with by their users depending on implementation and design details of the chatbot, like for example its human-likeness, humor or professional appearance [23]. This as well as other specifics of the conversational design, like ethical factors, transparency, the degree of human-likeness or opportunities for playfulness within the chatbot conversation, still need to be subject to our future consideration.

Incorporating the results of the analysis and exploration phase, first ideas for solutions were generated and implemented as part of a prototype222https://gitlab.com/Tech4Comp This prototype consists of three parts: the chatbot, a UI componentinserted into the LMS and a service, creating the knowledge graphs and the comparisons of these graphs. A first test took place in April 2020. In summer 2020 we will make the chatbot available to students for the first time. In the following we describe one prototype as result of the design and construction phase.

3.0.3 Implemented Prototype

A first prototype was developed, tackling the requirements and needs described above. It is presented to end users, e.g. students, as a special view in a learning management system (LMS), providing a personalized chat interface with a chatbot. This chatbot was implemented and trained using the open source natural language processing framework RASA

333https://rasa.com.

The chatbot’s main purpose is to offer writing assignments to students, gather the results and provide feedback. Therefore students are able to upload their written texts within the chat. Such individual texts are proxied through a las2peer network (see next paragraph) to a service generating the feedback. The first step this service executes is a cleaning process for the issued texts, as the input for further processing steps needs to be in a specific format. Subsequently the service involves T-MITOCAR to do two things: 1) request a knowledge graph that visualizes the concepts and concept connections of the students’ text, and 2) request a graph comparison of the just created graph with a reference knowledge graph. T-MITOCAR generated this reference graph from another written text, i.e. from a textbook, that contains domain knowledge about the same topic the students’ text is about. To hand back proper feedback to students, a feedback template is filled with specific comparison results, which is eventually converted to a feedback PDF document. Depending on the mentoring scenario, either the students knowledge graph, the reference knowledge graph or the textual comparison of the two knowledge graphs is transferred through the las2peer network back to the chatbot, which finally provides the feedback document via the chat interface to the individual student. This whole process is depicted in figure 1.

The former described frontend is a LMS-Plugin providing a single thread of a RocketChat444https://rocket.chat/ instance to end users, implemented as a web-component. Besides the actual chat the plugin also handles authentication, authorization, and data protection in accordance to the European GDPR rules. Regarding authentication and authorization an OpenId Connect (OIDC) login barrier is used, which makes sure that student credentials stay safe. Using OIDC furthermore allows students to use one account to login to the LMS, the chat client, and the las2peer network. The latter one is a decentralized open source environment for transferring and storing user data without inheriting a central authority [28]. It connects several nodes in a peer-to-peer fashion, protecting stored data and communication by using asymmetric encryption.

4 Summary and Future work

We have sketched the prototype TecCoBot that is able to present automated feedback on writing assignments provided by the software T-MITOCAR. TecCoBot scales the potential of a special face-to-face mentoring scenario to a large number of students, by enhancing already existing processes, materials and platforms. This technology-aided support enables providing students a part of mentoring on demand. Through the use of a software we are able to provide all students individual feedback in time, even in courses with a large number of students and few lecturers. Our approach allows students to receive feedback where otherwise they either would have to do without or significant additional human resources would have to be employed. In this way we make the support for learning from texts in self-study activities available to as many students as possible and whenever they want. The technologies we have used allow to consider the requirements for privacy, security and informational self-determination. In the near future we will conduct an application-oriented evaluation regarding the implemented chatbot, as well as reflect about the described processes, which will probably lead to new and adapted research questions. Furthermore we will extend the current chatbot prototype with more parts of mentoring. We also want to collect specific data of students to make technology-aided mentoring more effective and generating knowledge about how it works. Finally we want to identify design concepts to make parts of face-to-face mentoring scalable.

References

  • [1] European Union: ECTS Users’ Guide. Luxembourg: Publications Office of the European Union (2015).
  • [2] Metzger, C.: ZEITLast: Lehrzeit und Lernzeit. Studierbarkeit von BA-/BScStudiengängen als Adaption von Lehrorganisation und Zeitmanagement unter Berücksichtigung von Fächerkultur und neuen Technologien. In: S. Mandel, M. Rutishauer and E. Seiler Schiedt (Hrsg.). Digitale Medien für Lehre und Forschung. Waxmann, Münster (2010). (S. 287-302).
  • [3] Eby, L.T., Dolan, E.L.: Mentoring in postsecondary education and organizational settings. In: P.J. Hartung, M.L. Savickas, W.B. Walsh (Eds.). Handbook of Career Intervention, Volume 2. Psychological Association, Washington, DC, USA (2015). (S. 383-395).
  • [4] Ziegler, A.: Mentoring - Konzeptuelle Grundlagen und Wirksamkeitsanalyse. In: H. Stoeger, A. Ziegler, D. Schimke und H. Stöger (Hrsg.): Mentoring - Theoretische Hintergründe, empirische Befunde und praktische Anwendungen. 1. Auflage. Pabst Science Publishers, Lengerich (2009), S. 7-30.
  • [5] Koebis, L.: Digital-gestütztes Mentoring an Hochschulen. Working Paper tech4comp (2020).
  • [6] Reinmann, G.: Welchen Stellenwert hat die Entwicklung im Kontext von Design Research? Wie wird Entwicklung zu einem wissenschaftlichen Akt. design-based research, 63–78 (2014).
  • [7] McKenney, S., Reeves, T.C.: Conducting educational design research. Routledge, Taylor and Francis Group, London (2019).
  • [8] Nora, A., Crisp, G.: Mentoring students: Conceptualizing and validating the multi-dimensions of a support system. Journal of College Student Retention: Research, Theory & Practice, (9), 337–356 (2007).
  • [9] Gershenfeld, S.: A Review of Undergraduate Mentoring Programs. Review of Educational Research, 84(3), 365–391 (2012).
  • [10] Höher, F.: Übersicht über den aktuellen Forschungsstand zu Mentoring. In: F. Höher (Hrsg.). Vernetztes Lernen im Mentoring. Springer, Wiesbaden (2014).
  • [11] Kluger, A., DeNisi, A.: he Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory. Psychological Bulletin. 119. (1996).
  • [12] Hattie, J.: Lernen sichtbar machen. Schneider-Verlag, Hohengehren (2013).
  • [13] Hattie, J., Timperley, H.: The power of feedback. Review of educational research 77, 81–112 (2007).
  • [14] Deci, E. L., & Ryan, R. M.: Intrinsic motivation and self-determination in human behavior. New York: Plenum Press. (1985).
  • [15] Hattie, J.: Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand (1999).
  • [16] Pirnay-Dummer, P.: Knowledge Elicitation. In J. M. Spector (Ed.), The SAGE Encyclopedia of Educational Technology (pp. 438-442). SAGE, Thousand Oaks, CA, (2015).
  • [17] Pirnay-Dummer, P.: Linguistic analysis tools. In C. A. MacArthur, S. Graham & J. Fitzgerald (Eds.), Handbook of Writing Research (pp. 427-442). Guilford, Guilford Publications (2015).
  • [18] Aimeuer, E.: Application and Assessment of Cognitive-Dissonance Theory in the Learning Process. Journal of Universal Computer Science 4(3), 180–187 (1998).
  • [19] Helbig, H.: Knowledge representation and the semantics of natural language. Springer, Berlin, New York (2006).
  • [20] Jonassen, D.: On the Role of Concepts in Learning and Instructional Design. Educational Technology Research and Development 54(2), 177–196 (2006).
  • [21] van den Broek, P.: Using Texts in Science Education: Cognitive Processes and Knowledge Representation. Science 328(5977), 453–456 (2010).
  • [22] Tynjaelae, P., Mason, L., Lonka, K. (2001). Writing as a Learning Tool. Integrating Theory and Practice. Dordrecht: Springer (Studies in Writing, 7).
  • [23]

    Følstad, A., Nordheim, C. B., & Bjørkli, C. A.: What makes users trust a chatbot for customer service? An exploratory interview study. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11193 LNCS, 194–208. (2018).

  • [24] Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., & McTear, M. (2019). Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? Proceedings of the 31st European Conference on Cognitive Ergonomics: “‘Design for Cognition,’” (September), 207–214. (2019)
  • [25] Vandewaetere, Mieke, and Geraldine Clarebout. "Advanced technologies for personalized learning, instruction, and performance." Handbook of research on educational communications and technology. Springer, New York, NY, 2014. 425-437. (2014).
  • [26] Alemdag, Ecenaz; Erdem, Mukaddes: Designing an e-mentoring program for novice teachers in Turkey and investigating online interactions and program outcomes. In: Mentoring & Tutoring: Partnership in Learning 24 (2), S. 123–150. (2017).
  • [27] Følstad, A., Nordheim, C. B., & Bjørkli, C. A.: What makes users trust a chatbot for customer service? An exploratory interview study. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11193 LNCS, 194–208 (2018).
  • [28] Klamma, Ralf and Renzel, Dominik and de Lange, Peter and Janßen, Holger: las2peer - A Primer, August 2016, DOI: 10.13140/RG.2.2.31456.48645