Usefulness of Instructor Annotations on Flipped Learning Preparation Video System

09/25/2019 ∙ by Shintaro Uchiyama, et al. ∙ TUT 0

Flipped learning is a method that flips in/out class activities to make lectures learner-centered. In flipped learning, comments from learners on preparation material are useful information for instructors to consider before deciding in-class topics. Thus, we arrive at the notion that receiving comments from instructors will be effective for learners watching the video. By including annotations from instructors, we propose to improve the quality of content for learners and thus enhance learners' motivation and study satisfaction. To achieve this, we introduced "Steering Mark," a tool that enables learners to easily grasp the overall structure of a video, to the video learning system. We examined the effectiveness and influence of Steering Mark through an experiment with 34 undergraduate learners. As a result, Steering Mark was found to be useful in improving the quality of video content for learners.



There are no comments yet.


page 3

page 5

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Recently, flipped learning has become a popular learning method [1] [2] [3]. Traditionally, instructors teach learners in the classroom, and learners review and apply this knowledge to homework outside the classroom. Flipped learning flips these in/out classroom activities. Learners learn by watching videos or using other material outside the classroom, and then they review and apply this knowledge in the classroom. This method improves learning interest because it allows learners to take in information at their own pace during the out-of-classroom preparation time, as well as enables the learner and instructor to spend more time on communicating and discussing topics in the classroom. Therefore, flipped learning turns traditional lectures into learner-centered education.

In several applications of flipped learning [4], video is preferred over other preparation materials (such as podcasts and books), and its usage has been studied. Okumoto et al. proposed the “Response Collector” (RC) for preparation video watching [5]. The RC collects learners’ responses to support classroom activities. It allows learners to mark content as “Interesting”, “Important”, “Difficult”, or add a “Question” on a lecture video. The “Question” option provides especially important information that helps instructors to decide in-class topics because it shows in detail what learners have difficulty understanding.

Flipped learning systems are dependent on learners preparing independently before class. Okumoto et al. suggest that some learners are not inclined to watch the video; there is a need to support these learners in flipped learning practices. Herreid and Schiller [6] said that learners may resist flipped learning because learners are required to work before class, framing this as a problem with these systems. In flipped learning, if learners do not prepare before class, they will not be able to perform well in the classroom. Clearly, additional functionality is required to ensure that video material is effective.

To make learners prepare before class, it is important to encourage proactive learning [1]. A popular method is assigning homework [7]; however, we believe that this may give learners a sense of compulsion and deprive them of self-motivation, thus hindering learning.

We propose a way to improve learners’ motivation for preparation learning by incorporating annotations from instructors into video material. Preparation videos are often accessed on the Internet or using DVDs [4]. However, using common methods, it is difficult for learners to understand the overall structure and contents of the learning material at a glance; hence, they watch the videos without understanding the overall structure. We assume that this is one of the factors that decreases their motivation to learn in preparation for classes - Yilmaz [8] has shown that learners’ interest in engaging with the learning material corresponds to their motivation and study satisfaction.

This paper proposes and evaluates additional mechanisms that let the instructor add marks to the flipped learning video material. Our mechanism is realized on top of the RC, and these marks can be regarded as the first response to the video from the instructor. These marks from the instructor show learners the overall structure and focus of the video material, and our experiments show that they improve learners’ impressions of the videos. Therefore, the proposed mechanism will encourage more learners to watch the videos. Our contributions are as follows.

  1. We argue the problem of missing learner-instructor interaction during preparation learning in flipped learning systems (in Section II).

  2. We propose a simple mechanism for inserting instructor marks on videos (in Section III).

  3. We discussed that this may form a kind of interaction between instructors and learners during preparation learning (in Section II).

  4. We show, through experiments, that the mechanism improves impressions of the videos (in Section IV).

Ii Approach

(a) RC without Steering Mark
(b) RC with Steering Mark
Fig. 1: By adding annotations to content, virtual interaction is formed. Introducing Steering Marks to RC, we can achieve interactions in both directions.

The first step in applying flipped learning involves using a system that collects learners’ responses to apply them in the classroom. We noticed that the current RC lacks the idea of interaction between learners and instructors [9]. We can think preparation learning as a kind of distance education. We should let students and instructors interact more on preparation learning. Moreover, Kuo et al. [10] reported that learner-content and learner-instructor interactions affect learner satisfaction. Therefore, we expect that rather than enforcement methods such as homework assignments, improving the quality of content for learners and increasing interactions will enhance preparation learning satisfaction. This in turn would enhance learners’ motivation to watch videos again and continue to prepare before classes. Now, the problem lies in realizing interactions during preparation learning.

As in Fig. 1(a), RC without Steering Mark collects responses from learners and provides virtual interaction between learners and instructors. We call these responses interactions because they are synchronized with the video. We have chosen the same mechanism but in reverse, as shown in Fig. 1(b). First, the instructor watches the video material, considering the learners’ perspective, and they add marks and messages to the video. These will show the focus of the video as well as its overall structure. This will enable virtual communication from instructors to learners, thus achieving interactions in both directions, as is shown in Fig. 1(b).

Iii Proposed Method

We incorporated “Steering Mark” into RC to motivate learners to prepare for classes by using a two-way interaction system. As shown in the bottom of Fig. 2(a), Steering Mark on RC provides learners with timestamps in the video to show topic changes. Additionally, if a title is set to Steering Mark, it will be shown on mouseover as Fig. 2(b) shows. We expected the primary advantage of Steering Mark to be an improvement in impressions of the video content because learners could grasp the structure from the start, and they could use Steering Mark to skip already known topics.

Video annotation is a well-known method for educational technology. However, we believe that Steering Mark has difference based on flipped learning structure. Preparation videos in flipped learning are just focused on teaching basic knowledge of lecture, unlike distance education. Therefore, in other words, we must think the use of the knowledge in in-class activity, not just let learners watch the videos. Thus, our methods’ difference is that an instructor adds Steering Marks into the video while thinking about that learners’ use of knowledge in the video.

We developed Steering Mark to work similarly to when learners add their own marks on the related system [5]. An instructor can add a Steering Mark into a video as they like; click on the button to the right of Fig. 2(a) while playing the video, and a dialog that sets the title to the Steering Mark appears and the video pauses. We assumed that this procedure would not increase instructors’ labor time because they could check the video and add Steering Marks simultaneously.

(a) Overview. Steering Mark is indicated by a bicycle icon at the bottom.
(b) On mouseover, the title will be shown.
Fig. 2: RC with Steering Mark: Our proposed method “Steering Mark” on RC. We expect that the mark lets learners grasp the overall structure of the video at a glance.

Iv User Study

We set research questions to evaluate whether Steering Mark on RC works effectively to fulfill its purpose.

Is the Steering Mark useful as a function that adds value to watching the video on the system?

What kind of influence does the Steering Mark have on the collection of responses from learners? In order to verify these research questions, we performed experiments using a system incorporated with the Steering Mark.

No. Question
Answer form
1 Did you watch the entire n-th video contents provided by the Response Collector?
Binary (watched the whole, watched only a part or not watched)
1-1 Did you check the contents of the whole n-th video provided by the Response Collector while performing this exercise?
Binary (yes, no)
1-2 Please choose the one closest to the reason you did not watch the whole video
Four alternatives (Already known/It was difficult to watch/System problem/Other)
2 Was it difficult to watch the video provided by the Response Collector when you first watched it?
Four-step evaluation
3 Was it easy to grasp all the content?
Four-step evaluation
4 Did you add marks that were available?
Four-step evaluation
5 Did the use of Steering Mark (with slide title) change?
Four-step evaluation
6 Did the Steering Mark (with slide title) change your impression of the video content?
Four-step evaluation
7 Was it troublesome to watch the video provided by the Response Collector when you watched it during the exercises?
Four-step evaluation
8 How much did you refer to the content of the video provided by the Response Collector in performing the exercise?
Four-step evaluation
9 Were you able to quickly find the topic you want to see in the video provided by the Response Collector?
Four-step evaluation
10 Did you use the marks that you inserted?
Four-step evaluation
11 Were the inserted marks useful?
Four-step evaluation
12 Did you use Steering Mark (with slide title)?
Four-step evaluation
13 Was Steering Mark (with slide title) useful?
Four-step evaluation
14 Which did you use more - marks which you added or Steering Mark (with slide title)?
Four-step evaluation
15 This video is distributed using YouTube. Did you feel that you wanted to watch in your familiar environment instead of the Response Collector?
Binary (yes, no)
15-1 Please enter the reason that you prefer to watch in your familiar environment.
Free text
16 Additional Comments
Free text
TABLE I: Questionnaire of Impression Evaluation
Time Content Video length Settings
First class Introduction of Git 22m43s None
Second class Git with GitHub 15m27s Steering Mark was added to the slide break
Third class Prototyping and external design 11m57s Steering Mark with slide title was added to the slide break
Fourth class Testing 12m32s None
TABLE II: Content and system setting in each class time
Class time 1 2 3 4
Number of people 51 50 51 35
TABLE III: Number of people who answered to the questionnaire

Iv-a Experiment Procedure

We conducted an experiment with 51 learners who took one of the classes for “Software Exercise” at Toyohashi University of Technology, Japan. We encouraged learners to use the system as a reference in performing exercises after preparation learning. Table II shows content and system settings in each class timeslot. These classes are all in Japanese.

In this experiment, we set up preparation learning time using our system in the class so as to evaluate the change in impression caused by Steering Mark. At the end of each class, we gave learners an impression evaluation questionnaire as shown in Table I. For statistical analysis, we conducted tests using a previous class as a control group. We also used the user behavior log of the mark addition history and the playing history from the system.

In the results, we used Wilcoxon’s signed rank test for the four-step evaluated questions and McNemar’s test for the binary questions. We used the paired-t test for log analyzing. But we normalized the logs because each class used different video lengths, as shown in Table 


Iv-B Result

Table III shows the number of people who answered the questionnaire. As Table III shows, some learners did not answer the questionnaire, therefore, in the subsequent statistical tests we used 34 subjects who answered all the questions.

Two questions, Q6 (Fig. 3) and Q9 (Fig. 4), that ask about the relevance of Steering Mark to the video content and two questions that ask about the usage of Steering Mark versus a self-mark have significant differences in their responses. We tested logs that counted the number of times a self-mark was added, self-mark was clicked and Steering Mark was clicked, and then we obtained the significant differences between the self-mark click log (Fig. 10) and Steering Mark click (Fig. 5) log.

Iv-B1 Influence of Steering Mark to the video content

Fig. 3: Aggregation of Q6: “Did the Steering Mark (with slide title) change your impression of the video content?” In this result, there is a significant difference in the third vs. fourth. This shows that positive impressions of the video content decreased in the fourth because Steering Mark was not active. Note.
Fig. 4: Aggregation of Q9: “Were you able to quickly find the topic you want to see in the video provided by the Response Collector?” In this result, there is a significant difference in the third vs. fourth. This shows that subjects were not able to find the topic that they wanted to watch. Additionally, 75% or more subjects responded with “good” in both the second and third. Note.
Fig. 5: Aggregation of Steering Mark click log per 5 min: In this result, there is a significant difference in the second vs. third. This shows that subjects clicked Steering Mark in the second more than third. Note.
Fig. 6: Aggregation of Q5: “Did the use of Steering Mark (with slide title) change?” In this result, there is no significant difference.

We used questionnaire aggregation and log analysis to find out whether Steering Mark is useful as a function that adds value to videos to encourage watching on the system.

The questionnaire results of both Q6 - “Did the Steering Mark (with slide title) change the impression of the video content?” (Fig. 3) - and Q9 - “Were you able to quickly find the topic you wanted to see in the video provided by the Response Collector?” (Fig. 4) - show significant differences () in the third class vs. the fourth class. It was shown that the subjects’ impression of the video content was negatively impacted when moving from an environment with Steering Mark to an environment without it. Moreover, 75% or more subjects responded with “good” in both the second and third classes answers to Q6 and Q9. Additionally, the Steering Mark’s click log (Fig. 5) shows a significant difference (). In this result, the mean difference is -2.86.

It was shown that the impression of the video content was improved using Steering Mark. On the other hand, it was not possible to verify whether it had any influence on the content when slide titles were added to the Steering Mark.

Iv-B2 Influence of Steering Mark on response collection from learners

We used questionnaire aggregation and log analysis to find out whether Steering Mark influenced response collection from learners.

A questionnaire result of Q4; “Did you add marks that were available?” (Fig. 7) shows a significant difference () in the first class vs. the second class. However, the log of self-marks added (Fig. 8) does not show a significant difference in any set. Alternatively, responses to Q10, “Did you use the marks which you added?”, (Fig. 9) suggests () that subjects clicked on self-marks more in the fourth class than the third class. The self-mark click log (Fig. 10) suggests () that subjects click on self-marks less in the third class than the second class. On the other hand, the log does not show or suggest significant differences in the third class vs. the fourth class, but it tends to increase ().

It was suggested that with the influence of Steering Mark, the use of self-mark tends to decrease for learners’ response collection.

Fig. 7: Aggregation of Q4: “Did you add marks that were available?” In this result, there is a significant difference in the first vs. second. This shows that subjects no longer put self-marks in the second. Note.
Fig. 8: Aggregation of self-mark added log per 5 min: In this result, there is no significant difference.
Fig. 9: Aggregation of Q10: “Did you use the marks which you added?” In this result, there is a marginal difference in the third vs. fourth. This suggests that subjects use self-marks as alternative to Steering Mark in the fourth. Note.
Fig. 10: Aggregation of self-mark click log per 5 min: In this result, there is a marginal difference in the second vs. third. This suggests that subjects no longer click on self-marks in the third. Note.

V Discussion

V-a RQ(A):Usefulness of Steering Mark

Our results show that Steering Mark is indeed useful as a function that adds value to video content to promote watching on the system, as shown in Section IV-B1. Additionally, we found that Steering Mark makes it easy for learners to access topics that they want to watch. These results are supported by several answers such as “It is easy to review when the Steering Mark is available” and “It is easy to access the topic that I wish to watch when the Steering Mark is available” from the additional comments in the questionnaire. It also shows that they may recognize the usefulness of Steering Mark.

It was unclear whether adding a slide title to Steering Mark influenced changes in learners’ impressions. The number of Steering Mark clicks decreased significantly as Fig. 5 shows; however, we can say that there was no difference in subjects’ recognition of its use, as it is possible that they could find topics that they wanted to watch without clicking on the Steering Mark. In addition, we received the following response: “The slide title of the Steering Mark was very good”. Therefore, when we add a slide title to the Steering Mark, it changes user behavior.

V-B RQ(B):The influence of Steering Mark

As a result of examining what kind of influence Steering Mark has on the collection of learner responses, which is the purpose of RC, it did not become statistically clear if Steering Mark is added to the video contents, as shown in Section IV-B2.

We measured a significant negative difference between the first and second classes for Q4 (Fig. 7). However, we did not get a significant difference in the self-mark added analysis; instead, the count slightly decreased. For this reason, subjects might have felt as if their self-marking behavior decreased because there is a 7-min difference in the video length between the first class and the second class. In addition, in the click log analysis, an increase in self-mark clicks was not indicated in the third class vs. the fourth class; however an increase was observed in the third class vs. the fourth class for Q10 (Fig. 9). Furthermore, considering that a significant negative difference is obtained in the third class vs. the fourth class in Q6 (Fig. 3), it is possible that the impression of the video contents worsened because subjects tended to use self-marking as an alternative to Steering Mark, and the RC without Steering Mark was unfriendly.

Vi Conclusion

We introduced Steering Mark into the RC. Steering Mark facilitates annotations from instructors, adding value to the video content with the aim of having learners prepare in advance through the flipped learning system. We examined the effectiveness and the influence of Steering Mark, in terms of the number of responses, through a contrast experiment conducted in each class in a university. In this experiment, we analyzed the impression evaluation questionnaire that was given to learners after the end of class and watching behavior logs to examine the effectiveness and influence composing RC without Steering Mark and RC with Steering Mark.

As a result, Steering Mark was found to be a useful function to improve video content. This suggests that the Steering Mark will help learners grasp the overall structure of the video at a glance, without influencing critical response collection from learners.


A part of this research was supported by JSPS KAKENHI Grant Number 17KO0484.


  • [1] J. Bergmann and A. Sams, Flip your classroom: Reach Every Student in Every Class Every Day.   ISTE, 2012.
  • [2] J. Bishop and M. A. Verleger, “The Flipped Classroom: A Survey of the Research,” in 2013 ASEE Annual Conference & Exposition, 2013.
  • [3] N. T. T. Thai, B. De Wever, and M. Valcke, “The impact of a flipped classroom design on learning performance in higher education: Looking for the best “blend” of lectures and guiding questions with feedback,” Computers and Education, vol. 107, pp. 113–126, 2017.
  • [4] L. Abeysekera and P. Dawson, “Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research,” Higher Education Research and Development, vol. 34, no. 1, pp. 1–14, 2015.
  • [5] H. Okumoto, M. Yoshida, K. Umemura, and Y. Ichikawa, “Response Collector: A Video Learning System for Flipped Classrooms,” in 2018 5th International Conference on Advanced Informatics: Concept Theory and Applications, 2018.
  • [6] C. F. Herreid and N. A. Schiller, “Case studies and the flipped classroom,” Journal of College Science Teaching, vol. 42, no. 5, pp. 62–66, 2012.
  • [7] S. J. DeLozier and M. G. Rhodes, “Flipped Classrooms: a Review of Key Ideas and Recommendations for Practice,” Educational Psychology Review, vol. 29, no. 1, pp. 141–151, 2017.
  • [8] R. Yilmaz, “Exploring the role of e-learning readiness on student satisfaction and motivation in flipped classroom,” Computers in Human Behavior, vol. 70, pp. 251–260, 2017.
  • [9] T. Anderson, “Modes of interaction in distance education: Recent developments and research questions,” in Handbook of Distance Education, 2003, pp. 129–144.
  • [10] Y. C. Kuo, A. E. Walker, K. E. Schroder, and B. R. Belland, “Interaction, Internet self-efficacy, and self-regulated learning as predictors of student satisfaction in online education courses,” Internet and Higher Education, vol. 20, pp. 35–50, 2014.