Rapid changes in the society such as high volatility in labor market demands (Vom Lehn, 2020), the COVID situation around the world, and the growing need for personalization in learning environments (Shearer et al., 2020; Alamri et al., 2020) have led to an increased attention to online informal learning (Lockee, 2021; Zhang et al., 2020). Nevertheless, educational systems that are capable to handle wide range of learners’ context and requirements in these informal learning environments are still in their infancy (Berman, 2020).
Although there have been many attempts to create personalized educational systems that meet learners’ individual needs (Rojas-López and García-Peñalvo, 2022; Tavakoli et al., 2020c), scalability is still one of the most significant problems in this domain. This scalability problem leads to a situation that educational systems focus on specific key content domains (which can be manageable through a regular updating process) (Mousavinasab et al., 2021; Tavakoli et al., 2020b), and/or become less sensitive about the quality of educational content (management of personal learning pathways, content quality, etc.) they offer (Zhang et al., 2020). We need a scalable dynamic curriculum development approach in order to be able to offer personalised education to learners. Such an approach should consider learners’ needs (e.g. their needs towards labour market), capture relevant knowledge areas, include high-quality educational content for each area, and also be maintained with minimum efforts.
In this paper, we propose a curriculum development system to tackle the above mentioned challenges of online, personalized, informal learning environments using Artificial Intelligence (AI) and Crowdsourcing approaches. We defined and implemented four components with respective services including the crowd management service in our platform to empower curriculum developers when creating and maintaining a specific curriculum.
To validate our proposed system, we integrated it into our personal learning environment platform project, eDoer111www.edoer.eu
, which aims to provide labour market driven personalized learning pathways and content. Subsequently, we calculated the accuracy of our proposed recommendation system by comparing it with pre-defined, data science related learning content (the domain we selected for evaluating our system) by three experts. Moreover, we interviewed eight education experts to provide us feedback on our objectives and the usability of our system.
2. State of the Art
Recent literature shows that there have been many attempts to overcome problems of curriculum development in online, informal education environments. We categorized these attempts based on the methodology they used in the following sections. At the end, we explain the lessons we learned to conclude our literature review.
2.1. Artificial Intelligence Based Curriculum Development
AI has been aiding curriculum development predominantly with using Machine Learning(Somasundaram et al., 2020; Tavakoli et al., 2020a) and Text Mining(Pattanshetti et al., 2018; Molavi et al., 2020) methods. (Somasundaram et al., 2020) proposed an educational program model (i.e. prerequisite, content, expected outcome) based on labour market demand using AI back-propagation concept in order to help learners to up-skill themselves towards their current or desired job. Although they claim that the result of their curriculum model is promising, they focused on the single area of Internet of Things (IoT). (Pattanshetti et al., 2018)
created knowledge graphs ofOpen Educational Resources (OERs) by applying Natural Language Processing (NLP) techniques to help authors to deliver their content to the proper audience. However, they only focused on content level and not on higher level learning goals. Moreover, some researches (Tavakoli et al., 2020a; Pattanshetti et al., 2021) proposed approaches that perform automatic quality assessment for educational resources in order to help content providers to filter out low-quality content from their resources list. (Molavi et al., 2020) introduces a novel method that uses Latent Dirichlet Allocation (LDA) (Jelodar et al., 2019) algorithm to extract topics covered by specific educational resources, in order to build learning pathways. Their application only focused on building curriculum for educating text mining to learners.
2.2. Crowdsourcing Based Curriculum Development
As user participation in online learning platforms increases(Jiang et al., 2018), "the wisdom of the crowd" has a great potential for both teachers and learners. "Crowdsourcing for education" has been used for content-creation(Cross et al., 2014) and also for sharing practical and theoretical knowledge(Pandey et al., 2017) in large scale. (Weld et al., 2012) concludes that including crowd’s opinion in the process of education is useful when it comes to building scalable and personalized curriculum. (Farasat et al., 2017) integrates crowdsourcing into pedagogical paradigm as "Crowdlearning". They suggest that including students into the creation of the curriculum not only increases the amount of content produced, but also improves the depth and performance of learning. However, crowdsourcing has its own flaws as well. Participation time and its effectiveness for complex tasks is one of the most noted issues(Stewart et al., 2010; Yang and Qi, 2021).
2.3. Lessons we learned
While crowdsourcing is generally useful for content-creation for curriculum development, and also contributes to a system’s scalability, the time consumed by a participant can cause motivation problems and prevent their effective participation(Stewart et al., 2010). On the other hand, AI can help us to automate some tedious tasks such as quality assessment and learning topic extraction from educational resources. Therefore, we propose a novel curriculum development method, which utilizes the benefits of both AI and crowdsourcing approaches to generate dynamic and scalable personalized curricula.
The main research objectives of this paper are:
How can we empower educational service providers by using crowdsourcing to create, validate and maintain curricula?
Does supporting curriculum designers and learning content authors with AI by recommending learning topics and high-quality educational resources for specific knowledge areas facilitate the process of developing up-to-date curriculum?
3. Curriculum Development Framework
We defined four main components for our framework: 1. high-level leaning goals which consist of skills, 2. skills which consist of learning topics, 3. learning topics which consist of educational packages, and 4. educational packages which include one or more educational resources. To help the contributors managing their content, each of these components is enhanced with the following services: 1. Add service for defining a component for the first time, 2. Suggestion service for collecting users’ suggestions on a component and also automatically accept or reject them based on crowd opinions, 3. Recommendation service for providing insights for contributors based on existing open data (e.g. job vacancies, educational resources, and standard taxonomies) in order to help them adding and updating content. Additionally, we designed a crowd management component to monitor and analyze contributors’ activities and opinions when maintaining learning content.
At first we explain the details of designing and implementing our system, and subsequently show a first use case on an existing personalized educational platform.
3.1. Managing High-level Learning Goals
For adding high-level learning goals (consisting multiple skills), we collected titles and optional descriptions together with the following key contextual features to help learners (Tavakoli et al., 2020c): industry, company, city, and country. However, authors can decide not to contextualise learning objectives by keeping each of these fields as General. Afterwards, our system recommends a list of goal related skills based on the title of the high-level goal, in order to capture necessary skills to master for the given learning goal. For these recommendations, we rely on the ESCO 222European Skills/Competences, qualifications, and Occupations: https://ec.europa.eu/esco/portal/home data-set, which is a European multilingual classification system of skills, competences and occupations. ESCO is continuously updated by subject matter experts of the European Commission and includes 13,485 skills, linked to 2942 occupations. We match the title of the high-level goal with existing occupations in ESCO using the Bleu score (Papineni et al., 2002) concept. We decided to use Bleu score as it allows us to capture the closest term (occupation in this case) in the ESCO data-set no matter in what language the title is. Finally, we recommend a list of skills linked to the closest occupation to the author. Authors can either select skills from the list, or add new skills manually. After finalizing the skill list, the author can sort skills based on the order they need to be shown to learners. Figure 1 shows an screenshot about adding a high-level goal to our system.
After adding a high-level goal, users can view the page333Figure: https://raw.githubusercontent.com/ali-faraji90/edoer/main/LAK22/Goal.png of the newly added high-level goal, which includes:
The list of skills associated with the goal.
Opinion of other users (crowd) regarding the importance of a particular skill in relation to this goal. At each skill, users can see the importance of the particular skill for that particular high-level goal, based on up and down-votes (see 3.5.1) of the crowd.
The suggestion list. After defining a high-level goal for the first time, editing can be done by addressing the crowd opinion. Therefore, all users can suggest: 1. adding a skill, 2. deleting a skill, 3. reordering skills for each high-level goal. To provide insights for users and to keep each goal updated, our system recommends other, potentially related skills using the aforementioned algorithm. After adding a suggestion, other users can provide their opinion about a particular suggestion by up and down-voting. Our system captures these actions and suggests a decision whether to reject or accept the suggestion (see 3.5)
3.2. Managing Skills
3.2.1. Defining Skills
In order to define a skill, users receive string auto-completions as they are typing the title of a skill. This function is based on existing skills in the ESCO standards. Auto-completion not only facilitates skill title definition, but also helps users to refrain from adding different titles for a single skill. After filling out the title and description fields, our system uses Youtube playlists in order to provide insight for contributors regarding existing learning topics on any new skill. The system automatically searches the skill title, collects related videos, and extracts the most important key-words by applying TF-IDF (Ramos and others, 2003) algorithm on the the collected video titles444Figure: https://raw.githubusercontent.com/ali-faraji90/edoer/main/LAK22/Add-skill.png. The author can either select from this AI generated recommendation list or type in learning topics manually. Again, after finalizing the topic list, the author can sort the topics into an order of importance for learners. As depicted on Figure 2 after defining the skill, a skill page will be created including: 1. the list of learning topics, 2. crowd opinion about the learning topic’s importance (see 3.5.1), 3. the suggestion list including system recommended learning topics (see 3.2.2), adding, deleting, and reordering suggestions, which are monitoring our suggestion reviewing process (see 3.5.3), and 4. high-level learning goals associated with this skill.
3.2.2. Editing Skills
After adding a skill, similarly to high-level goals, the crowd has the ability to edit its content. In order to contribute to the skill updating process, we provide some learning topic recommendations that other users can easily add as a suggestion (see Figure 2). These topic recommendations are generated as follows:
Collecting learning topics. We search each new learning topic title on Youtube555using the Pafy library: https://pypi.org/project/pafy/, and collect at least 50 educational video transcripts labelled with that particular topic title. This exercise results in a two-column data-set including transcripts of educational resources in the first column, and 2. their associated learning topic in the second column.
Text pre-processing. On each collected transcript, we apply standard text pre-processing steps including converting to lower case, removing special characters, stemming, and
adding n-grams (bi/tri-grams).
Applying labelled topic modeling. We use LLDA (Labelled Latent Dirichlet Allocation) to extract topics from transcripts. LLDA is a supervised version of LDA (Jelodar et al., 2019)
, which is a generative probabilistic topic model aiming at extracting the most important keywords for each topic labels. We collect the ten most important keywords, their probabilities for each topic and store them in therecommendation list
Collecting content for skills. The recommendation list we generated through topic modelling of learning topics is usually not exhaustive enough, as some skill related topics may not be represented well on the list. Therefore, we also perform search the title of the given skill on Youtube playlists and collect at least 200 videos for each skill.
Applying LDA model on skill related learning resources. As for these resources we do not have any topic labels, we use an LDA model, an unsupervised machine learning method, which considers each transcript as a distribution of different topics, each topic as a distribution of different words, and aims to extract existing topics together with their distribution of words. To find the number of learning topics related to a skill, which is the input of the LDA model (as parameter ), we calculate Coherence (Röder et al., 2015) for different number of topics (between 2 to 50), and set with the topic amount that results in the highest coherence value. Again, for each newly extracted topic, we extract the ten most important words (and their probabilities) and add them to the previous words in our recommendation list.
Removing the existing topics. In order to prevent recommending learning topics that are already associated with a skill, we remove those, already associated, topics from our recommendation list.
Sorting the recommendation list. To capture the most relevant topics quickly, we sort the words on the recommendation list according to their probability values in our topic models, and start recommending from the first item on the list.
3.3. Managing Learning Topics
Users can add learning topics by defining their title and description first. Afterwards, a topic page is created and contributors can start adding educational materials to a given topic. To facilitate adding an educational package to each topic, we collect a list of educational materials from Youtube and Wikipedia666Using Wikipedia library: https://pypi.org/project/wikipedia/ based on the title of the topic. Afterwards, each collected educational material passes through the following automatic quality control process:
Metadata based quality control. (Tavakoli et al., 2021) showed a close relationship between the metadata quality and the content quality of educational resources. Accordingly, they built an openly available quality prediction model that detects the quality of educational materials based on their metadata. We adopted this approach to filter out low quality educational materials. Furthermore, based on this quality prediction model, we also calculated a quality score, which is shown to users for each recommended educational resource.
Content based quality control. To predict the relevancy of a particular educational resource to a learning topic, and, at the same time, to filter out irrelevant educational resources, we use the probability values of each word in the LLDA model discussed in the previous section (see 3.2.2).
As recommended educational resources go through this automatic quality control process, users can directly associate those resources to learning topics. However, authors can still provide suggestions, for instance decoupling an educational resource from a topic, when they think it is not an appropriate (not relevant or it has low quality) content777Figure: https://raw.githubusercontent.com/ali-faraji90/edoer/main/LAK22/Topic.png. Moreover, users can also have an overview of skills that require the knowledge of that particular learning topic.
3.4. Managing Educational Packages
Users can also define educational packages including one or more educational resources. To do this, firstly, users need to name a title and an optional description for an educational package. Afterwards, they can add one or more educational resources either by importing from a URL
, or 2. uploading an educational resource. Subsequently, contributors can fine tune and set properties (i.e. title, description, format type, estimated time needed to complete the resource, source of the content, includes an example/theory or not, level of details, and if it is a recording of a class-based instruction or not) for each of the resources in the package. It should be mentioned that in order to facilitate the property setting process, we implemented a property extractor component which retrieves information fromYoutube educational videos and Wikipedia. Finally, our system recommends related learning topics to contributors (using the LLDA model for the topics (see 3.2.2)) in order to help them set the learning topics that are covered by the new educational package.
3.5. Crowd’s Opinion Management
In this part, we will explain our voting system, how contributors receive points, and the process of suggestions’ reviewing.
3.5.1. Voting System
Users can up and down-vote skills and topics based on their perceived importance to their containing component. Therefore, with this feature, contributors are able to show how important 1. a skill for a high-level goal, 2. a learning topic for a skill, and 3. an educational package for a learning topic. Also, users can up/down-vote new suggestions (i.e. adding, deleting, and reordering suggestions) and give their achieved points in the context of the suggestion (see 3.5.2) to help a suggestion be accepted or rejected. By using this mechanism of voting, the system puts more weight on reliable users in the suggestion reviewing process (see 3.5.3).
3.5.2. Achieving Points
Contributors can collect points on each skill and also each learning topic. They can collect a point if their defined skills and topics receive an up-vote or if they are added as a learning goal to others’ profile. Moreover, if the educational material they added receives up-votes from learners, they also receive a point for the target topic(s) of that educational material.
3.5.3. Reviewing Process of Suggestions
Each and every suggestion needs to receive a minimum number of points to be approved in the system (currently ten points, but it can be changed in our configuration files) within a predefined period (currently a week, however it can also be changed in our configuration files), otherwise the suggestion will be automatically rejected. When a suggestion receives the minimum required points for approval, the system calculates the rate of positive received points to the all received points. If the rate is greater than 75% (again customizable in our configuration), the suggestion is automatically accepted, otherwise rejected.
3.6. Use-case: Personalized, Goal-Driven Learning Recommendations
Our personalized learning system, eDoer888www.edoer.eu, is a platform in which learners 1. set their learning goals, 2. receive skill lists related to their target goals, 3. select skills from the list, what they want to master, 4. generate their learning dashboard, which includes all selected skills and associated learning topics, and 5. receive personalized educational resources based on their preferences and behavior history on the platform. Figure 3 shows learners’ dashboard in our personalized learning content recommendation system. We used this educational platform as the first use case for our AI-based crowdsourcing system, to investigate on one hand how this technology aids contributors and authors to define and maintain educational content, and on the other hand, how does it empower learners by providing up-to-date personalized curricula.
To evaluate the system, we decided to measure the accuracy of our recommendations during the authoring process in the domain of Data Science. Moreover, we designed a semi-structured subject matter expert interview protocol, to evaluate the authoring process with users. In this section, we will showcase the results of these efforts.
4.1. Recommendation Accuracy
To evaluate our recommendations, we asked three experts with at least five years of academic and ten years of industrial experience in the area of Data Science to manually (without using our platform) specify the following high-level goals: Data Scientist, Data Analyst, and Business Analyst together with related skills and learning topics. Subsequently, they used our system to define these jobs with our skill recommendations (see 3.1). The list of skills produced by the experts were then compared with the list of recommendations of our system. The results showed that our system, on average, had an F1-score of 89% when it comes to recommending relevant skills for high level goals. When it comes to learning topic recommendation for data science related skills (see 3.2) our evaluation resulted in the following F1-scores: Python programming: 83%, Machine learning: 76%, Statistics: 79%, Data visualization: 75%, which meant 79% as weighted average.
Finally, when recommending specific learning content for learning topics (see 3.3), experts examined their validity. Only if the recommended content was marked as high-quality and relevant to the topic, we considered it as a valid recommendation. This evaluation revealed that our educational content recommendation method provided high-quality relevant materials in 93% of time999As different curricula are continuously developed in our system, to receive an up-to-date data-set of the existing components and recommendations, contact the authors..
4.2. Subject Matter Evaluation
In order to evaluate the proposed authoring process, we designed and executed interviews with 8 senior members (i.e. managers, professors, associate professors, and researchers) from different organizations: Participant_1 from Ericsson company in Sweden, Participant_2 University of Amsterdam, Participant_3 KU Leuven University, Participant_4 from TIB - Leibniz Information Centre for Science an Technology, Participant_5 from University of Bonn, Participant_6 from Netherlands AI Coalition, Participant_7 from the American Psychological Association, Participant_8 from Career and Life Planning (CALP). The structure of the interviews was as follows: 1. Introducing the system and its logic (~20 minutes), 2. using the authoring system and its features for about 20 minutes, and 3. going through a semi-structured interview with the assistance of a questionnaire101010The questionnaire is available here: https://forms.gle/A17KHhWGoUH9WFuc8 (~20 minutes).
Participant_1 and participant_2 mentioned that providing an environment in which learners can be informed about when and how to build their careers is the most important part of the system. Also, Participant_3 considered providing insights for authors regarding the relevant skills, learning topics, and educational materials as a key feature of our system. Participant_4, Participants_5 and Participant_7 emphasized that combining AI with crowdsourcing can improve the quality of each recommendation component, and therefore this is the most promising function of the system. However, they suggested to provide as much clarity as possible for users on how our AI and crowd-management components collaborate with each other. Participant_6, besides pointing at the usefulness of our system when it comes to matching jobs and their required knowledge dynamically, mentioned that integrating the ability for testing a knowledge should be part of our future steps. Participant_8 believed goals page and curriculum are the most important elements of our system and thought enriching these parts should have a priority in our future work.
Ultimately, 100% of the interviewees agreed that "creating dynamic personalized curricula for learners" as our objective is extremely important and timely. Also, only 12.5% (1 out of 8) was unsatisfied with the usability of our prototype system, which shows that most of the participants consider our prototype usable already.
5. Conclusion and Future Work
Throughout this paper, we showcased a novel learning content authoring system which helps authors to build personalized curricula for learners. By providing intelligent recommendations, this system aids contributors to define 1. high-level learning goals consisting of skills, 2. skills built by learning topics, and 3. learning topics with related educational materials. We believe that such a system not only helps authors to define and maintain their educational content, but also empowers learners through setting their own learning objectives, receiving personalized recommendations, and being up-to-date on desirable knowledge. Evaluating our recommendations in the context of the data science showed that our system can provide 89% F1-accuracy in matching high-level goals and their skills, 79% F1-accuracy matching skills and their learning topics, and 93% precision in recommending high-quality, relevant educational materials. Moreover, we validated the main objective and usability of our system by interviewing eight subject matter experts in the area of education which showed that they were satisfied with the objective and usability of the our proposed system.
- Using personalized learning as an instructional approach to motivate learners in online higher education: learner self-determination and intrinsic motivation. Journal of Research on Technology in Education 52 (3), pp. 322–352. Cited by: §1.
- A critical examination of informal learning spaces. Higher Education Research & Development 39 (1), pp. 127–140. Cited by: §1.
- VidWiki: enabling the crowd to improve the legibility of online educational videos. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 1167–1175. Cited by: §2.2.
- Crowdlearning: towards collaborative problem-posing at scale. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale, pp. 221–224. Cited by: §2.2.
- Latent dirichlet allocation (lda) and topic modeling: models, applications, a survey. Multimedia Tools and Applications 78 (11), pp. 15169–15211. Cited by: §2.1, item 3.
- A review on crowdsourcing for education: state of the art of literature and practice.. In PACIS, pp. 180. Cited by: §2.2.
- Online education in the post-covid era. Nature Electronics 4 (1), pp. 5–6. Cited by: §1.
- Extracting topics from open educational resources. In European Conference on Technology Enhanced Learning, pp. 455–460. Cited by: §2.1.
- Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments 29 (1), pp. 142–163. Cited by: §1.
- Gut instinct: creating scientific theories with online learners. In Proceedings of the 2017 CHI conference on human factors in computing systems, pp. 6825–6836. Cited by: §2.2.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318. Cited by: §3.1.
- The open corpus challenge in elearning. Knowledge Management & E-Learning: An International Journal 10 (1), pp. 67–85. Cited by: §2.1.
- Proposed elearning framework using open corpus web resources. In 2021 International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT), pp. 1–5. Cited by: §2.1.
- Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, Vol. 242, pp. 29–48. Cited by: §3.2.1.
- Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining, pp. 399–408. Cited by: item 5.
- Personalized education for a programming course in higher education. In Research Anthology on Computational Thinking, Programming, and Robotics in the Classroom, pp. 344–367. Cited by: §1.
- What students want: a vision of a future online learning experience grounded in distance education theory. American Journal of Distance Education 34 (1), pp. 36–52. Cited by: §1.
- Curriculum design using artificial intelligence (ai) back propagation method. Procedia Computer Science 172, pp. 134–138. Cited by: §2.1.
- Crowdsourcing participation inequality: a scout model for the enterprise domain. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 30–33. Cited by: §2.2, §2.3.
- Quality prediction of open educational resources - a metadata-based approach. Cited by: §2.1.
- Metadata analysis of open educational resources. In LAK21: 11th International Learning Analytics and Knowledge Conference, pp. 626–631. Cited by: item 1.
- OER recommendations to support career development. IEEE Frontiers in Education (FIE). Cited by: §1.
- Labour market information driven, personalized, oer recommendation system for lifelong learners. Cited by: §1, §3.1.
- Labor market polarization, the decline of routine work, and technological change: a quantitative analysis. Journal of Monetary Economics 110, pp. 62–80. Cited by: §1.
- Personalized online education—a crowdsourcing challenge. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, Cited by: §2.2.
- The nonlinear impact of task rewards and duration on solvers’ participation behavior: a study on online crowdsourcing platform. Journal of Theoretical and Applied Electronic Commerce Research 16 (4), pp. 709–726. Cited by: §2.2.
- Understanding the implementation of personalized learning: a research synthesis. Educational Research Review, pp. 100339. Cited by: §1, §1.