Older Adults and Crowdsourcing: Android TV App for Evaluating TEDx Subtitle Quality

09/29/2018 ∙ by Kinga Skorupska, et al. ∙ PJATK 0

In this paper we describe the insights from an exploratory qualitative pilot study testing the feasibility of a solution that would encourage older adults to participate in online crowdsourcing tasks in a non-computer scenario. Therefore, we developed an Android TV application using Amara API to retrieve subtitles for TEDx talks which allows the participants to detect and categorize errors to support the quality of the translation and transcription processes. It relies on the older adults' innate skills as long-time native language users and the motivating factors of this socially and personally beneficial task. The study allowed us to verify the underlying concept of using Smart TVs as interfaces for crowdsourcing, as well as possible barriers, including the interface, configuration issues, topics and the process itself. We have also assessed the older adults' interaction and engagement with this TV-enabled online crowdsourcing task and we are convinced that the design of our setup addresses some key barriers to crowdsourcing by older adults. It also validates avenues for further research in this area focused on such considerations as autonomy and freedom of choice, familiarity, physical and cognitive comfort as well as building confidence and the edutainment value.



There are no comments yet.


page 6

page 7

page 8

page 9

page 10

page 20

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

According to the latest population data published by Eurostat the share of older adults, defined as people aged 65+, is increasing in every member and candidate state. In European Union member states (EU-28) this share has risen by 2.4% between 2006 and 2016. Moreover, the long term 2015 EUROPOP projection covering the time up to 2080 shows that this trend will continue, and people aged 65+ are expected to comprise 29.1% of the total EU-28 population by 2080, while latest data shows they already made up 19.2% of the population in 2016 (pop, 2017). The same is increasingly true of Western societies all over the world (UNr, 2015). For example, according to the U.S. Census Bureau by the year 2050, the population aged 65+ in the United States will almost double, reaching over 20% of the society (Ortman et al., 2014). While the number of older adults is already significant, their potential remains largely untapped because of a shortage of adequate research-informed activities and programs allowing them to contribute to the society. Moreover, numerous studies confirm that developing sustainable solutions for older adults is still challenging. In particular, Knowles and Hanson underscore that there is still room for further research and establishing a more holistic approach to the problem, despite extensive progress in that field over the last 20 years (Knowles and Hanson, 2018b, a). Access to a lot of available activities relies heavily on ICT skills and familiarity with computers and technology, as well as the accessibility of the solutions. Moreover, older adults are often not provided with adequate motivation to take part in these activities, e.g. crowdsourcing (Brewer et al., 2016), as they are not challenging, fun or do not hold an explicit connection to real life.

It is our intention to address some of these concerns by designing and testing the feasibility of an engaging crowdsourcing solution for older adults that would encourage them to participate in crowdsourcing tasks. Therefore, we decided to assess older adults’ interaction and engagement with an online crowdsourcing task relying on their innate language skills, at home in a non-computer scenario. For this purpose we launched a qualitative pilot study to build and test a language-focused solution and verify the underlying concept and barriers, including the interface, configuration issues, interaction, topics and the process itself.

In the course of the study, we developed and tested an Android TV application for crowdsourcing TED and TEDx subtitle errors that enables the participants to detect and categorize them. In this study we operate at the intersection of interests of four groups of potential stakeholders, depicted in Table 1.

Group Their stake
Older adults as the target group for our solution and the crowdsourcers motivated by the edutainment aspect of the task and its positive impact
Translation communities as direct beneficiaries of the insights from the errors detected and problems signaled by the crowdsourcers
General public as the ones benefiting from the improved quality of subtitles
Developers as the ones involved in addressing the barriers to crowdsourcing and benefiting from insights from our development process and tests
Table 1. Four groups of stakeholders

Our assumptions, based on our previous work with older adults in a similar field (Kopeć et al., 2017), were that they enjoy using Android tablets (Kopeć et al., 2017) and therefore can learn to proficiently use Smart TVs, which have the benefits of relying on a familiar setting (TV set) and interfaces (e.g. Teletext), a simple remote, as well as sport a larger screen size. Furthermore, older adults are experienced users of their native language and can ascertain if utterances seem natural. It was our expectation that they would see improving the quality of subtitles for TED and TEDx talks as an interesting and motivating task due to its educational value.

Therefore, our study aims to explore a few research aspects. Our prime objective was to determine the potential of Smart TVs as crowdsourcing platforms for older adults. This necessitated to first answer the question of whether older adults interact with Smart TVs in a more familiar fashion than with computers. Secondly, we wanted to ascertain if they consider themselves competent enough in their native language to detect errors in subtitles. And lastly, we examined if such a task is engaging for them, and what are the emergent barriers to their involvement, and possible ways to overcome them. We hope that researching these questions will help us draw conclusions on how to better include older adults in ICT-dependent crowdsourcing tasks, and in online activities with the use of some new, but familiar interfaces and to find promising avenues for further study. At the same time, we want to propose and verify the design and setup of online crowdsourcing translation process support based on commodity goods, in this case the Smart TV platform, and to mitigate the barriers that may exist for this type of activity, also related to vital challenges of crowd work systems raised at CSCW in recent years (Kittur et al., 2013; Culbertson et al., 2017; Zhu et al., 2014; Yu et al., 2014).

The paper is organized as follows: first we present the related works on topics, such as older adults’ empowerment, crowdsourcing and volunteering, subtitling TED and TEDx, designing for older adults and Android TV in particular; second, we describe our methods, including the design of the DreamTV crowdsourcing tool and the testing protocol; third the results section follows, providing a summary of our collected qualitative data; finally, we interpret our results in the discussion section, which is followed by our conclusions and future research plans.

2. Related work

2.1. Challenges of empowering older adults

There exist increasingly more studies at the intersection of HCI and aging, but as Vines et al. concluded in their discourse analysis, many focus on stereotypes related to health, socialization and technology (Vines et al., 2015) instead of exploring the aging process and looking for opportunities. One positive example is Carrol et al.’s study of older adults as ”organizational firekeepers” (Carroll et al., 2012) in roles as keepers of history, co-designers and members of intergenerational teams contributing valuable complementary skills. So, while it is true that some older adults do not have faith and confidence in their skills, especially if they are ICT related (Sandhu et al., 2013), such barriers can be partially mitigated to enable their valuable contributions. One way to do this is by introducing a positive social context and support (Orzeszek et al., 2017). While in our previous research this context was strongly based on direct social interaction between generations (Kopeć et al., 2017, 2018a), in our current study it is built on the clear social benefit of the opportunity to crowdsource edutainment content (Rapeepisarn et al., 2006): having a lasting positive effect on a large number of viewers using subtitles e.g. for learning languages and educating the general population. This translates into a strong positive motivator, as one of the most common needs expressed by older adults is the need to feel useful, help others and contribute to the common good in a social setting (Brewer et al., 2016). Additionally, it also uses one of the advantages that older adults hold over the younger generation: their experience, explored for example by Balcerzak et al. (Balcerzak et al., 2017, 2018). Just as their vast knowledge of the cultural and historical context had a positive impact on their performance in a historical location based game (Kopeć et al., 2017), so can their long time experience with their native language act as an empowering factor, allowing them to feel confident, and competent enough to climb the ladder of ICT proficiency (a benefit they are aware of (Aula, 2004)); eventually even making them ready to join the ICT solution development process, as in the SPIRAL method (Kopeć et al., 2018b). On top of the question of feeling competent, there are specific physiological changes in the brain of older adults, which comprise of deterioration of working memory, and consequently, the ability to acquire new information (Wolfson et al., 2014). But, this effect can be mitigated by creating step-by-step instructions for doing an ICT task in a workshop format (Kopeć et al., 2017), and specific instruction design elements targeting these problems (Wolfson et al., 2014). At the same time, there is evidence that crystallized intelligence, which consists of general knowledge derived from experience, not only does not decline (Wang and Kaufman, 1993), but may benefit from aging (McArdle et al., 2000). This may prove to be an asset in tasks related to the native language, e.g. subtitling, as crystallized intelligence is primarily measured through general world knowledge and language tasks (Baghaei and Tabatabaee, 2015).

2.2. Volunteering and crowdsourcing

Encouraging older adults’ engagement in volunteering and crowdsourcing activities in general (Morrow-Howell, 2010) benefits the society as a whole. Such activities have positive effects on older adults’ well-being (Morrow-Howell et al., 2003), as well as their mental and physical health (Lum and Lightfoot, 2005). Thus, they can be regarded as a protective factor for their psychological well-being (Greenfield and Marks, 2004; Hao, 2008). In particular, they can also pose certain health advantages, as staying active and learning new things can delay the onset of age-related issues, e.g. mild cognitive decline and some related aspects of dementia (Kötteritzsch et al., 2014). These benefits were confirmed in a study of Lum and Lightfoot, who showed that volunteering slows the negative effects of aging and helps to combat depression (Lum and Lightfoot, 2005). They may extend to online crowdsourcing tasks, as the mental benefits may be connected to the results of the study by Yang and Cheng-Yu (Yang and Lai, 2010), who found that Wikipedia editors are internally driven by self-concept motivation: their need to maintain a consistent and positive image of themselves, which they thus satisfy. This issue was also explored in previous studies on Wikipedia editing by older adults (Nielek et al., 2017b). As such, engaging older adults in interesting tasks which have a high social value may be beneficial on all fronts.

Older adults have proven to be more aware of their needs and abilities than the younger generation (Djoub, 2013); therefore, they are more selective in choosing their activities. Moreover, they differ from the younger generation in their online behavior and decision-making (von Helversen et al., 2018), which, alongside their generally lower ICT skills, may explain how little interest they expressed in the Mechanical Turk platform populated by tedious and repetitive crowdsourcing tasks (Brewer et al., 2016).

Volunteering and user engagement are often considered in terms of using gamification techniques, i.e. the use of game design elements in non-game contexts (Deterding et al., 2011). Some researchers claim that gamification is a more psychological than technological issue (Zichermann and Cunningham, 2011). Many methods and gamification tools are based on sound psychological foundations, like the well-established self-determination theory (Ryan and Deci, 2000), which was verified in the context of older adults (Vallerand and O’Connor, 1989). Although there are some promising reports on using gamification elements in older adults crowdsourcing tasks (Itoko et al., 2014), there are also more recent reports that show difficulties in this field (Brewer et al., 2016).

2.3. Subtitling TED and TEDx talks

TED111When TED first launched it was an invite-only conference combining Technology, Entertainment and Design (TED) which eventually became an annual event in California. In 2001 the conference was acquired by Chris Anderson, who in 2006 shared the recorded talks online, paving the way for the current popularity of TED as an edutainment platform. More information about TED is available online at: https://www.ted.com/about/our-organization/history-of-ted is a nonprofit organization that organizes conferences on the topics of technology, entertainment and design. Its mission is to share ”ideas worth spreading” with the wider audience through the talks recorded during its conferences. To popularize the format across the world, TED launched the TEDx222The TEDx programme allows volunteers to apply for a TEDx license to organize a TED-like conference in their local area. More information about TEDx is available online at: https://www.ted.com/about/programs-initiatives/tedx-program initiative of volunteer organized TED-like conferences around the globe. Since 2009, when this initiative was launched, the number of published TEDx talks in over 100 different languages has reached 100,000. These talks totaled a billion views in large thanks to the TED Open Translation Program, now rebranded as TED Translators333The TED Translators programme allows volunteers to create subtitles for TED, TEDx and TED-Ed talks on Amara. More information about this programme is available online at: https://www.ted.com/about/programs-initiatives/ted-translators. Within the TED Translators initiative volunteers contribute subtitles to TED, TEDx and TED-Ed videos in 100+ languages, making them accessible to everyone.

The TED Translators initiative is a true example of a community of practice which Leave and Wenger (Lave and Wenger, 1991) describe as a group driven by common interest that collectively improves and gains knowledge. In 2014, Cámara, in a survey-based study (Cámara de la Fuente, 2014), identified the need to contribute to the TED mission of ”spreading ideas” as a key volunteering motivation of the TED Translators. As such, improving the quality of these subtitles is a promising example of a potentially engaging task, which is beneficial both socially (granting access to all) and individually (learning about interesting topics). As the subtitles are sourced in a large part from non-professional volunteers, who make mistakes that can be detected by experienced native speakers of the language, this makes the task doubly important and rewarding.

Additionally, within the TED Translators project, there exist various challenges, largely dependent on the language community. For example, not enough volunteers to satisfy the ever-growing demand for translations; as of April 17, 2018 there are 56,617 TEDx videos added to the TED team in Amara444Amara is a subtitling platform where people can volunteer to create subtitles for videos with the use of a streamlined online editor. More on Amara is available online at: https://amara.org (Jansen et al., 2014), an in-browser subtitling platform used by TED for its TED Translators project. Moreover, as TED Translators employ a 3-step quality assurance process, there are not enough experienced reviewers (step 2) and Language Coordinators (step 3) who can correct, and approve the subtitles for the talks so that they can be published.

Thus, not only is there potential to introduce a fourth Quality Assurance step, or to facilitate steps 2 and 3, but also to employ Machine Transcription and Translation to grant the international audience access to the content of the videos that were not yet translated. The inclusion of Machine Translation (MT) in the human translation process, especially where quantity is important, is a clear path ahead as it can shorten the process by up to 40% (Etchegoyhen et al., 2014)

. This trend is also evident in the development avenues of market-leading Computer Assisted Translation (CAT) tools, such as MemoQ or Trados. There is clearly room for involving crowdsourcing and citizen science in the context of Machine Learning and Natural Language Processing of the subtitles e.g. through supervised learning processes. Increasing both the quantity and the quality of subtitles is important, because they provide access to a wide range of ideas, also scientific, to the hard of hearing, non-native speakers and the general public in an easy to digest, edutainment format. So, making them more accessible can be a positive motivator for the volunteers.

2.4. Designing for older adults and Android TV

Although older adults are a very non-uniform group in terms of their ICT-skills and needs, there are some general guidelines, also relevant in Smart TV solution design. In ”Designing Displays for Older Adults”, Pak noted the older adults’ need for context, consistency and low working memory burden, and their low tolerance for design clutter (Pak and McLaughlin, 2010). A similar focus on the need to simplify is visible in the findings of the study by Silva et al. (Silva et al., 2013) concerning a TV dance application, where users likely experienced an information overload due too much information being visible on the screen when trying to repeat dance moves. More generally, Pan et al. (Pan et al., 2015) proposed a framework which consists of ”symbolic familiarity, cultural familiarity, and actionable familiarity”, all of which have positive effects on the adoption of technology by older adults. Also, as shown by our research involving an Android location-based game, (Kopeć et al., 2017) older adults are quite comfortable with tablets. According to Tsai et al. (Tsai et al., 2015), this is due to the larger screen size than phones. Thus, as large screen size and design familiarity, consistency and simplicity are key considerations, for our exploratory study we decided to make use of the increasingly popular Smart TVs. Moreover, such TV sets can be controlled by a remote, which may mitigate older adults’ possible problems with ”mapping actions to devices” signaled by Fisk et al. (Fisk et al., 2009). So through focusing on Smart TVs, we not only deliver a large screen size, but also a familiar and simple experience of using a remote control and interacting with text on screen, not unlike Teletext. Some preliminary studies on using Smart TVs for older adults in the context of Living Labs had been initiated (Alaoui and Lewkowicz, 2015), however, this is still a largely unexplored area of research with few general studies on interfaces for TV apps for older adults, such as the one by Nunes et al., who among other things, recommended minimizing navigation steps, clearly marking selections, using the middle of the screen, simple language and allowing for enough time to read (Nunes et al., 2012). This encouraged us, based on our findings concerning older adults and Living Labs (Kopeć et al., 2017), to explore the new technological, but familiar, opportunities that Smart TVs present in the home environment, especially in the context of crowdsourcing. According to numerous analyses, there is still tremendous dominance of using TV sets over computer and mobile equipment, especially among older adults (defined as 65+). For example, Nielsen research on the US market shows that older adults spend over 50 hours a week in front of the TV, while fewer than 5 hours in front of the PC, including using video on PC (less than an hour a week).555More information on TV viewing habits by age can be found online at: https://www.recode.net/2016/6/27/12041028/tv-hours-per-week-nielsen Although there are numerous works on various aspects of designing services and interfaces for older adults (Dickinson et al., 2007), in particular for interactive television use-case scenario (Carmichael, 1999), the development of new trends of Smart TV, VOD (Video On Demand) and OTT (Over the Top) internet-based streaming services and applications, which all largely allow users to freely choose content to watch, led us to thoroughly rethink the service we planned in this crowdsourcing use case with older adults.

3. Methods

Based on our research and experience, we developed a pilot of an exploratory study concept for using crowdsourcing without the computer in a TV-enabled setup, as well as the tool to support the subtitle translation and review processes: the DreamTV application. The findings from our previous work in this field in the Living Lab context (Kopeć et al., 2017; Nielek et al., 2017a; Orzeszek et al., 2017) informed the design of our testing protocol, which presents subtitling as a task useful for a wide group of viewers (motivation), and introduces the users to its basic concepts, as well as verifies their understanding with an on-paper exercise (empowerment). We believe that the design of our study addresses key barriers to information technology and crowdsourcing by older adults, as presented in Table 2.

Figure 1. The Xiaomi MiBox, a TV Set-top box (STB) with a simple remote can turn any TV with an HDMI socket into a Smart TV with the Android TV OS. (Source: Wikipedia, CC BY-SA 3.0 license)
Barriers Solutions
Uncomfortable and costly setup (Kopeć et al., 2017; Sandhu et al., 2013) TV sets at home (Alaoui and Lewkowicz, 2015; Kopeć et al., 2017)
Unfamiliar interfaces (Orzeszek et al., 2017; Aula, 2004; Czaja et al., 2006; Czaja and Lee, 2009) Text and remote control (Fisk et al., 2009)
Repetitive tasks (Brewer et al., 2016) Educational wide-domain TEDx videos (the, 2017)
Unclear social benefit (Brewer et al., 2016) Improving subtitles for all (Cámara de la Fuente, 2014)
Unclear personal benefit (Brewer et al., 2016; Vines et al., 2015) Learning, positive self-image (Yang and Lai, 2010; Nielek et al., 2017b)
Unsocial nature of the task (Vines et al., 2011) Part of the community (Carroll et al., 2012; Nielek et al., 2017a)
Table 2. Overview of signaled barriers to crowdsourcing by older adults with proposed solutions inspired by literature.

Here, it is important to note that subtitles have only started to become more common in Poland in the 1990s, mostly in cinemas and later online. The preferred TV format is using a voice-over translation, which is also a popular practice in Cambodia, Mongolia, Vietnam and some other East European countries.666Voice-over translation is common in part as it is cheaper than dubbing, as it usually involves only one voice actor, and there is no need to mix the original soundtrack which just plays quiet in the background, but also because of force of habit. Unlike dubbing, which employs a cast of actors to recreate the soundtrack in the target language, voice-over translation usually employs the technique of using a single voice actor talking over a quieter original film soundtrack. This fact may form an additional barrier for the older adults’ engagement, as our tests were conducted with Polish participants using Polish language subtitles.

3.1. DreamTV application

Therefore, based on our analysis of possible solutions to the aforementioned barriers and taking into account the needs of various translation communities, for the purpose of our exploratory study we developed the DreamTV application, which allows users to detect errors in subtitles. Its main features are described below.

3.1.1. Video selection

The application allows the users to choose the videos to play (Fig. 2). It remembers the previous choices and the progress of the user on each video, making it possible to resume the tasks. It keeps track of the users’ contributions and generates statistics. The identified errors are written in a database, which is accessible online.

Figure 2. The main screen of DreamTV allows the users to select new videos, or continue watching their previous ones.

3.1.2. Playing the video mode

The application works as a regular player, so between choosing error categories the users can enjoy watching videos with subtitles (Fig. 3), as an edutainment activity (Rapeepisarn et al., 2006).

Figure 3. The video player screen with subtitles

3.1.3. Error detection and category choice dialog

Once the users spot a mistake in the subtitles appearing in the video mode they can click the middle round button on the remote (Fig. 1) to pause the video. The application then overlays the error detection and category selection dialog over the video; on the right the subtitle is visible in the context of the subtitles surrounding it (Fig. 4). The user can then select if this is the subtitle they meant to pause at, and then they can choose and save the appropriate error category for the selected subtitle, at the same time restarting the video playback from a little before it was paused.

3.1.4. Error categories

The error categories whose feasibility we decided to explore are based on key categories of errors in subtitles as extrapolated from OTPedia777The wiki site of TED Translators features subtitling guides to style, workflow and most common considerations for subtitles, including errors and formatting issues. The wiki can be found online at: https://translations.ted.com/Portal:Main (TED Translators’ wiki with resources used as training and reference materials for translators). They are selected with the assumption of being easy to detect and interpret by viewers without much prior training.

  • Timing: if the subtitle starts too early, or too late, or finishes too early in relation to the audio track, or if it disappears too quickly and it is impossible to read it - an important consideration when audio is present according to Armstrong (Armstrong, 2013).

  • Grammar: if there is a grammatical mistake: either with the tense, form, ending, punctuation or spelling.

  • Meaning: if there is difficulty understanding the meaning of the utterance or a suspected difference in meaning, especially in translation.

  • Style: if the wrong register is used in relation to the topic of the talk, or if the wording of a phrase is awkward, a potential calque, or could be corrected.

These four main error categories are also more intuitive, unlike existing models of quality assessment of subtitles by professionals (Romero-Fresco and Pöchhacker, 2017), and they focus on information which could be useful for reviewers to pinpoint the errors later. This is unlike existing studies of subtitle quality assessment that focus on error detection without the intent to make later corrections of the same texts easier, as they largely relate to live TV subtitling, e.g. by Ofcom888Ofcom is the communications regulator of the UK, operating under Communications Act 2003 to ensure great quality of communications services in the UK. More information on Ofcom can be found online at: https://www.ofcom.org.uk/. It is worth noting, that the Android TV remote used in the study was equipped with a microphone that could be used to provide additional comments or hints for translators through an option available in the basic interface (called advanced) visible in Fig. 4. This interface also allows the users to choose either one main, or a few error types in each subtitle, and display the reading speed required for it.

To carry out our tests, however, we have developed a simplified interface (called simple), which allows the users to mark only one error category. This was done to estimate if the participants would request more, or less complexity in the error detection dialog, and to keep the task middle-weight as a fusion of edutainment (postulated error detection only, without categorizations) and full correction crowdsourcing (existing basic interface allowing the users to record short messages, justifying their choices and offering corrections).

Figure 4. The advanced dialog allows users not only to select error categories, but also to record voice comments and see the subtitle reading speed.

3.1.5. Settings

Apart from the choice between the simple and advanced interface mentioned above, it is possible to choose to work on videos based on the audio and subtitle language combinations, where choosing the same language for both would limit the videos to transcription tasks only. To aid in further study iterations we introduced the ”testing mode” to limit the TEDx video selection to our predefined videos.

Figure 5. The settings allow the user to switch from the simple to the advanced mode, as well as limit the choice to specific videos in the testing mode.

3.2. The journey of application development

The application development process was based on early usability tests, co-design and consultations with the TED Translators community members. It was also informed by insights concerning empowerment and engagement from our aforementioned Living Lab’s activities centered around older adults. Based on these we scaled down our initial idea of editing faulty subtitles to error detection, which is more aligned with the need for simplicity, the edutainment aspect, and more suitable for control via the remote. Since then, the development of the application has focused on the unique way of interaction with the app using the simplest remote, as well as creating a nonintrusive interface to promote the fun aspect of the task. In line with Android TV guidelines, we designed a pop-up dialog system that appears on screen only when the users press the middle PLAY/PAUSE button, as this way they get the full screen experience without any clutter elements that could distract them. Some of the other early major changes included:

  • Testing multiple combinations of the remote buttons.

  • UX color coding, sizing (readability).

  • UX flow and default behaviors (settings, auto-selection).

  • Interface labeling and action defining for clarity.

  • Changes to initial error categories (made the same for translation and transcription).

  • Overlay view for subtitle context visibility.

  • Subtitle skipping forward/backwards functionality (timing adjustment for the needs of older adults and consistency).

While the application was received favorably in our early tests, we decided to field test it using a structured protocol with older adults in the context that it was meant to be used (Living Lab): their own TV sets at home.

3.3. Study design, participants and testing protocol

As it was mentioned above, we decided to employ the distributed Living Lab approach. In particular, we invited older adults to participate in the pilot study supervised by the researchers. The research protocol, involving individual testing at home (Fig. 6), consists of the following main elements: an introduction to the study, the DigComp survey testing familiarity with computers and variety of ICT tasks, a semi-structured introductory interview that tests openness to new experiences as well as familiarity and preferences regarding subtitles, the explanation of the main elements of the project, that is the study and its benefits, an introduction to subtitles, and an on-paper exercise training error category detection skills, a demonstration and a hands-on test and the free interaction with the application and our five pre-selected and redacted test videos. The whole protocol takes about two hours to complete, and the reported preliminary tests were conducted in the first months of 2018.

Figure 6. The at-home setup with a TV set facilitates the edutainment value of the task.

3.3.1. Introduction

As the first step, we thanked the recruited participants for wanting to take part in this research and informed them that they can stop at any time and for any reason. We read the declaration of consent to them and asked them if they would like to sign it, and only then continued the research session.

3.3.2. DigComp based survey

The survey is a tool for measuring indicators of digital competence, developed by IPTS, and funded by the European Commission. It uses questions related to the Digital Skills Indicator(dig, 2014) and broad ICT competence areas related to information (e.g. reading online news), communication (e.g. sending e-mails or making video calls), content creation (e.g. creating content, programming), safety (e.g. using anti-virus or a firewall) and problem solving (e.g. installing new devices, using Internet banking), based on the Digital Competence Framework (Ferrari, 2013).

3.3.3. Introductory interview

The semi-structured in depth introductory interview aimed to gauge two main issues. First, familiarity and attitude towards subtitles with questions related to cinema and viewing habits. Second, openness to new experiences, including questions about habits related to new experiences, especially educational, and key interests and topic preferences.

3.3.4. On-paper exercises

The paper exercises are a key component of our step-by-step introduction method for user empowerment as they allow the older adults to become familiar with key notions in subtitling. This includes an explanation of what subtitles are for, and who can benefit from them, as well as some technical information such as reading speed, maximum line length and best places to introduce line breaks, and finally the specific task of subtitle error categorization. The exercises consist of a selection of text snippets from our chosen test videos, each containing either one, or two errors. Towards the end of the task, the results are discussed with the participant to ensure that they understand the key principles of subtitling, as well as the error categories in the application. The test also allowed us to estimate the confidence of older adults when detecting language errors.

3.3.5. Using the DreamTV application with test videos

For our study we have selected five videos to represent different types of challenges. For this reason, these videos were controlled for the following characteristics:

General features:

  • a) Topic.

  • b) Video length.

  • c) Source language (spoken).

Ease of comprehension:

  • d) Jargon saturation.

  • e) Required reading speed expressed in characters per second (ch/s).

  • f) Presence of the speaker on-screen.

Types of errors:

  • g) Error source (machine or human).

  • h) Error saturation.

  • i) Error category.

No Video Characteristics
1 Rebuilding Dreams One Bedroom at a time by Joanna McCoy at TEDxKraków a) Activism b) 10 min. c) Polish d) moderate modern e) medium (around 10–18 ch/s) f) on-screen g) human natural and simulated h) high-moderate (around 1 error /40 sec of video) i) mostly grammar with some timing
2 General Einstein Theory of Relativity by Krzysztof Meissner at TEDxMarszałkowska a) Physics b) 11 min. c) Polish d) moderate science e) medium (around 11–18 ch/s) f) on-screen g) human natural and simulated h) moderate (around 1 error /1 minute of video) i) predominantly grammar
3 Will the Ocean ever run out of Fish by Ayana Elizabeth-Johnson and Jeniffer Jacquet a) Environment b) 10 min. c) English d) high e) very fast (around 16–21 ch/s) f) off-screen narration over a cartoon g) human natural h) very low (around 1 error /2 minutes of video) i) mixed
4 The hidden ways stairs shape your life by David Rockwell a) Culture b) 3 min. c) English d) low e) fast (around 13–21 ch/s) f) on-screen and off-screen g) Machine Translation with no corrections other than line alignment h) very high (around 1 error /10 seconds of video) i) mostly style and grammar
5 Inventing is the easy part. Marketing takes work by Daniel Schnitzer at TEDxPittsburgh a) Travel and technology b) 5 min. 30 s. c) English d) low e) medium (between 11–18 ch/s) f) on-screen g) MT with human corrections for obvious mistakes h) high (around 1 error /30 sec. of video) i) mostly grammar, style and meaning
Table 3. Characteristics of the videos used in the study. (Explanations: (a) Topic. b) Video length. c) Source language (spoken). d) Jargon saturation. e) Required reading speed expressed in characters per second (ch/s). f) Presence of the speaker on-screen. g) Error source (machine or human). h) Error saturation. i) Error category.)

The videos selected for the preliminary study, visible in Table 3, allowed us to observe a variety of factors at play, in order to determine the most interesting areas of further inquiry. The machine translations were generated using the original human-made transcripts, which were then imported to SubtitleEdit999SubtitleEdit is a free open-source subtitle editor with a wealth of useful features which available online at: http://www.nikse.dk/subtitleedit/. This software has the option to use the Google Translate MT engine to generate subtitle translations; the generated lines then had their alignment fixed to reflect the transcript. The human natural errors are a combination of organic errors from the community, as well as errors introduced by researchers based on the most common translation/transcription errors lists on OTPedia101010The TED Translators wiki features common error lists for many of the project’s languages. To see the list for Polish refer to the materials available online at: https://translations.ted.com/Polish to introduce varying error saturation levels. The tests were done with Polish language subtitles, as they were conducted with older adults in Poland.

3.3.6. Exit IDIs

In the exit interviews, we asked, among other things, about the feelings towards the general task of finding mistakes in subtitles, and in particular about their attitudes towards different videos we have tested. We also asked for any suggestions for increasing the attractiveness of this task in general, as well as concerning the topics of the videos, the reading speed, the preparatory exercises and the controls (remote) and setup. Additionally, we asked the participants if they would like to continue this task on their own, if they enjoyed the language part of the task and if they would like to try more/different types of language-related tasks in the future.

4. Results

Below we present the characteristics of the participants with the study summary, followed by some more detailed results.

4.1. Study group and research highlights

We invited seven older adults to our preliminary exploratory study: three female participants and four male participants. The study group was thoroughly selected from our Living Lab in Warsaw to cover several conditions: different age, occupation (one active person before retirement, two retired but still professionally active, two retired for several years and two retired for more than fifteen years). All participants live in Warsaw, the capital city of Poland, and they are native Polish speakers with limited or very limited English skills. There was a 20 year age span: the youngest participant was 60 years old and the oldest one was 79, with a mean age of 70,85 (SD=6,87).

Based on introductory interviews and DigComp surveys we can describe this group as active users, above basic ICT skills. Everybody, except one, has smartphones, and the majority have Smart TVs. They are rather intensive and frequent Internet users, more than once a week.

Older adults in our study, even ones who report using the Internet only occasionally, and owning no smartphone, are very comfortable using Smart TVs at home. They do not see them as complex, but rather ”interesting” (P4) and ”useful” (P3), and ”quite easy after some practicing” (P1, P2), some picked up on how to navigate them with a remote control within minutes (P1, P2, P3, P5), and all of them (P1–7) were able to learn how to perform the task in the DreamTV application in just one session. In case of P6 and P7, the learning curve was lower because their remote was similar to the one used in the study.

While all of the participants report that they rarely watch movies with subtitles, and prefer the voice over (P1–7) or dubbed content (P6), they see subtitles as ”useful” (P5, P6, P7), as ”cheaper and faster to make than voice over” (P2), and the task of improving them as ”pleasurable and fun” (P3), ”quite pleasant” (P1, P2) and ”manageable” (P5). Most appreciate how ”interesting” the videos are (P1, P3, P4, P6, P7), but one participant wished the topics were more ”practical and useful for me” and said it was ”not their thing” (P5). While the older adults were either good or very good at finding stylistic mistakes related to their natural language experience, especially in MT texts, they had trouble with technicalities such as timing, and small grammatical mistakes, like punctuation. Additionally, all of them complained about the reading speed required to understand the subtitles and sometimes paused the video to read the previous lines (P3, P4, P5, P6). While very few of the lines were above the 21 ch/s, which according to TED Translators’ guidelines is the maximum reading speed, even the common industry standard of 17ch/s maximum was an issue.

4.2. Detailed results

4.2.1. On-paper exercises

The participants were very sure of their answers when taking the tests, and most of them were able to explain their reasoning for choosing one error category over another, or marking a few at once. However, there appeared to be a dilemma relating to ambiguity and an overlap in the categories perceived by the participants, which is typical, as one type of error can be easily related to others. For this reason, we provided the option to mark multiple errors in the basic interface of the application.

The biggest issue was with the subtitle stylistic conventions, such as correct subtitle division (e.g. not leaving linking words at the end of the line and breaking up syntactic wholes), or the written format of sound information.

4.2.2. Test videos

The participants were mostly very confident when marking mistakes, and in the videos which were more saturated with grammatical and stylistic mistakes (videos no. 2, 5), they detected many of them. The stylistic aspects connected to language conventions were easy to pick up for older adults, especially in video no. 5. However, in video no. 4, which had machine translated subtitles, the stylistic problems were so saturated, that the meaning of large parts was unclear (P6, P7) or the whole video was unclear to some study participants (P1, P2, P4), even when reading the dialog list (Fig. 4). Moreover, video no. 4, which was Machine Translated, did not have any compression done to the subtitles, which pushed the reading speed up. The dialog list visible on the subtitle error choice screen (Fig. 4) proved to be very useful for two reasons. The main reason is that it was common that all of the study participants (P1–7) paused the video too late, and they had to use the dialog list to go back to the subtitle with the error, even reading it out loud (P3, P1, P6). Additionally, it helped comprehension as when the study participants felt they were getting lost, because of high reading speed required, or the jargon, they would pause the video and read the subtitles to make sure they understood the text (P3, P4). One surprise was that none of the study participants detected any timing errors in the videos. When asked about this, one participant replied that ”they had to read” (P5).

While some study participants were indecisive about choosing the main error category (P4, P5) others (P3) were so eager to continue that they kept forgetting to mark it, naturally wanting to mark an error only. This correlated to how engaged they were in the topic of the video. The more interesting they found the video, the fewer mistakes they detected (P1–P4, P6). Some participants requested to introduce an in-app tutorial, which would allow them to mark errors ”without the worry that they will make a mistake” (P1, P4). Some study participants found the reading aspect to be difficult because of impaired vision. However, they just sat closer to the TV set to mitigate the effect and continue the task (P1, P3), but for most this was not an issue.

4.2.3. Exit IDIs

The participants felt they were capable of detecting errors in Polish subtitles for the videos (P1–7), and one of them even said: ”There should be more subtitle testers like me, but not young people because they have little experience” (P3). Most of them claimed that they would be very happy to be able to do the task for a longer time by themselves with new videos like these (P1–4), but all of them would welcome the incentive of choosing topics interesting for them (P1–P7), as well as controlling for time (”The movies should be shorter, then I could watch anything! Just give me ten 5 minute films and I can do that for an hour” P3). Also, some of them (P2) are willing to do this activity in intergenerational context, i.e. with their grandchildren, who live abroad, when they come home to visit them during the holidays. Moreover, the videos in the study were deemed as quite interesting, and most of the study participants said that they ”learned a lot” (P1–4). However, the participants had some issues with the specialized jargon used by the speakers (e.g. ”it is not explained what is this photon” or ”Spiderman, this is not Polish” by P2 and ”kryptonite, must be a mistake” by P5, P6) and with the subtitles disappearing ”too fast” (P1–7). While they enjoy finding errors in subtitles, they would not be interested to create their own, as it is ”too much typing and time” (P1, P2, P4), apart from one who said that they ”would have to try” (P3). They also said they enjoyed the language aspect, and they would be willing to try other language-related tasks in the future (P1–3, P6, P7).

4.3. Notes on errors marked

None of the synchronization errors were found, but also none were incorrectly marked. Stylistic errors were marked as such if the problem was with language style, while largely ignored when it was connected to subtitle conventions and formatting. The most common errors found were thus stylistic and grammatical errors related to spelling and general language correctness. Capitalization and punctuation errors, belonging to the grammar category, were difficult to spot for the older adults, we expect because some of the errors required them to remember the punctuation from the previous lines. On top of this, many study participants interpreted the ”meaning” category, as something to mark when they encountered an unfamiliar term, or a divergence between spoken phrase and its transcription for the native language, which made for a few false positives to mark more specialized words and subtitles with higher compression rates.

5. Discussion

Based on the exploratory research and pilot evaluation of our setup of online crowdsourcing translation process support via Smart TV platform we identified five promising areas of consideration when designing similar systems. These include (1) autonomy and freedom of choice, (2) physical comfort and (3) cognitive comfort to aid comprehension as well as (4) building confidence and (5) providing the edutainment value. While these five considerations form a starting point as preliminary guidelines for further inquiries, it is important to remember the limitations of this exploratory study, as it was conducted with older adults in Poland, among the more active and ICT-aware well-educated individuals belonging to the economic lower and middle-class111111The best status class to describe this group would be Intelligentsia, which is the class of educated, but not necessarily well-off people in Poland.. The following discussion is aimed to inform the design of systems which could better include such older adults in ICT-dependent crowdsourcing tasks. For each of the signaled general considerations there are preliminary proposals of best practices to mitigate or overcome the observed barriers and accommodate preferences which are discussed below.

Promising considerations for designing crowdsourcing systems for older adults:

5.1. Autonomy and freedom of choice

The question of the ability to choose the topic, duration and engagement level with the content at any time appeared to be of crucial importance for the engagement of our group of older adults.

5.1.1. Freedom to choose the topic

The older adults wanted to freely choose the content to watch, based on their preferences. This is of key importance, as in our use case their primary motivation comes from the alignment with their interests providing the edutainment value. Thus, one crucial insight is that it is necessary to introduce older adults’ profiling, perhaps in the form of a few questions during the initial setup of the app to allow the participants to be served videos based on their interests, as well as topic categorizations within the app. Moreover, for longer studies it is important to introduce the function to search by the topic of the video, as there are clearly some topics favored by older adults and because in this case their interests act as a motivational element. This could be supported by a user profiling system coupled with an adaptive mechanism, that monitors and follows user preferences and their changes over time.

5.1.2. Freedom to control task duration

The older adults wished to be able to choose how long they want the videos to be, and how much time they would like to devote to this task overall, which is another potential adaptive service.

5.1.3. Freedom to contribute as little or as much as one wishes

The older adults were pleased about the learning potential of the videos, but they were also eager to finish watching the videos early if they were not as interested in the topic (P1, P5). This suggests that the task is just as engaging as the video, which corresponds to the edutainment concept. However, the more engrossed older adults were the fewer errors they seemed to detect, which suggests prompts or some other technique could be introduced to remind the viewers of the task. This, combined with their unwillingness to create subtitles themselves, as it is too much time and effort, shows that this type of crowdsourcing should rely on quantity of contributors, rather than their accuracy because participants are unlikely to go back to the videos to watch them again and verify their own accuracy when detecting errors.

5.2. Physical comfort and familiarity

It indeed seems that our group of older adults interacted with Smart TVs in a more familiar, and definitely more comfortable fashion than with computers due to their long-time familiarity with the setup and controls, as well as the physical comfort granted by their leisure home environment.

5.2.1. Physically comfortable setup (TV)

Regarding the Smart TV interface, people who watch TV at home already have a very comfortable setup for spending time in front of the TV Set, unlike their computer setup at home, which, as we observed, is mostly designed for short sessions. This corresponds with numerous market research done e.g. by Nielsen. Despite the task appearing in the same setup as one of their favorite leisure time activities, that is watching TV, our group of older adults viewed using the DreamTV application as more demanding, which raises the questions, how much time would they be willing to devote to it, and if the solution is sustainable in the long run. These ought to be further researched.

5.2.2. Ease of navigation with clearly labeled controls (remote)

The Android TV interface proved to be very easy to navigate, as the older adults participating in our study are quite used to navigating the menus and teletext by clicking arrow buttons on the remote. We observed that this was a largely familiar and comfortable experience for them, despite the remote being much different from theirs for P1–P5. One suggestion we received was to mark the remote control with buttons of different colors, as now ”they look the same” (P3, P4) or separate the direction keys (P6, P7). The colors could then correspond to the colors of actions which can be taken on the screen to assure cohesion of design between the physical controller (the remote) and the application design on screen.

5.2.3. Ability to take breaks

The older adults valued the task also for the ability to take breaks, pause, make tea (P3, P5) and readjust without losing sight of the context and feeling pressured into hurrying. This suggests that there ought to be no timeouts or time limits imposed in the design of other crowdsourcing tasks.

5.2.4. Adjustable text size and colors

There ought to be the possibility to adjust the size of the letters and introduce high contrast selection to accommodate vision impairments. The text size should be adjustable based on the possible needs of older adults, as not all of them have their TVs at a comfortable distance to read the fonts, and have varying degrees of light in the living room. While some of them compensated by moving the chairs closer to the screens, for long time use it would be better if they could stay on the couch where they are comfortable. We expect this to be a lesser issue in societies where subtitle use rates are higher.

5.3. Cognitive comfort to aid comprehension

5.3.1. Adjustable reading speed

The required reading speed ought to be in the low range (14 ch/s at most) or possibly the speed of the video playback could be adjustable, or even adaptive. What also follows, is that in order to allow older adults to be more effective at finding mistakes in Machine Translated subtitles the software has to use language compression algorithms, to cap the required reading speed at 14 ch/s for the ease of comprehension. This also holds true for human organic errors, as some human-translated videos, especially by untrained volunteers, use close to literal translation with no compression121212More information on problems and techniques of subtitle compression can be found on TED Translators’ wiki in the following location:https://translations.ted.com on the ”How to Compress Subtitles” wiki page., which tends to make subtitles longer.

5.3.2. Ability to pause, go back and view the context for reference

It should be possible to view the video transcript and to go back to any previous point, as older adults have slower reflexes and often clicked pause too late. They also welcomed the ability to pause and check the context of the dialog list131313More on this feature was explained in section 3.1.3, while the dialog list providing context is visible in Fig. 4. for reference, in case they felt they may have missed something.

5.4. Building confidence

Based on our participatory observation and exit interviews, the older adults in our group consider themselves competent enough in their native language to detect errors, and with a well structured step-by-step tutorial, also errors in subtitles. With a more extensive tutorial on types of errors common in subtitles, combined with a quick refresher of grammar rules, especially focused on punctuation, the older adults will be a good group for this type of crowdsourcing.

5.4.1. Making use of older adults’ language skills

Without extensive prior training and with some limited empowerment provided by the researchers, the older adults can use their natural language experience to find mistakes, especially with grammar and style, which works well with MT errors, but excludes technical problems such as formatting and synchronization. While making use of older adults’ expert native user proficiency in language enabled crowdsourcing is very empowering, the question of the evolution of language also deserves attention. Some older adults were not familiar with technical or pop culture references (P2, P5 and P6) and they assumed these may have been mistakes. To mitigate this barrier we propose that a reference dictionary could be available to the participants via the app, in which they could verify their linguistic hunches and at the same time learn more modern terms for the contexts they encounter.

5.4.2. Providing a theoretical and practical tutorial

While grammar and style seem to be the best category for this type of task, the timing issues were not detected by anybody, neither in transcription nor in translation tasks. It is possible the difficulty with finding timing (sync) issues was due to not including this category in the exercises before the task, as they were made on-paper and were limited to the choice between ”grammar”, ”style” and ”meaning”. Moreover, the technical aspects of the subtitling process, despite appearing in the protocol on-paper exercise, such as subtitle stylistic conventions, were largely overlooked, and would require more training, or an introduction of the subtitle-specific tutorial to the application. In general, it seems that tasks where the subtitles are a translation (videos no. 3, 4, 5) are a better target for language correction than transcription tasks (videos no. 1, 2) where older adults tended to try to focus if the words written are exactly as said when finding mistakes. (especially P5). We conclude that it would be beneficial to introduce a tutorial within the app that would prime the older adults to focus on the most common error types.

5.5. Providing the edutainment value

The content we used for this activity, that is the TED and TEDx talks is quite engaging and broad in terms of domain. One challenge that exists then, is to frame other crowdsourcing tasks in such way as to provide edutainment value for the participants, which would allow them to consider this task to be closer to their leisure time than to work.

5.5.1. Using wide-domain educational content

As we observed the older adults from our group find improving the quality of subtitles for TED and TEDx talks to be an interesting and motivating task because of its educational value, and what has to be stressed, personally for them with the topics of their choice. Therefore it seemed that easily providing older adults with the choice of topic they are interested in, and at the same time ensuring that they are learning is a necessity when designing edutainment-like crowdsourcing. The DreamTV application facilitates this, as TED and TEDx talks are characterized by a very large domain, and multiple videos have a wider focus than just technology, entertainment and design.

Additional note

As more ICT-proficient generations enter retirement age in Western societies, such studies may soon prove to be more generalizable, both because of rising levels of ICT literacy, and the introduction of affordable solutions such as the Android TV STB, like Xiaomi MiBox, which can turn any TV set into a Smart TV. As such, it is a promising setup for an increasingly larger group of older adults, who are more ICT-savvy and have access to cheap technology able to turn their TV sets into Smart TVs.

6. Conclusions and further work

6.1. Overview

The Smart TV setup and the interface are promising in the field of research with older adults, especially due to the familiarity, large screen size, general accessibility and comfortable at-home setup. Crowdsourcing subtitle errors is also perceived as an interesting task, which can be very engrossing and motivating depending on how it is framed. This study allowed us to evaluate the feasibility and direction of further inquiries in the area of TV-enabled crowdsourcing tasks, relying on native language proficiency of older adults. The key areas of interest when designing such systems based on our exploratory study seem to be (1) autonomy and freedom of choice, (2) physical comfort and familiarity, (3) cognitive comfort to aid comprehension as well as (4) building confidence and (5) introducing the edutainment value. Therefore, we consider our study an important voice, and a contribution into the discussion of vital research areas of human aspects of collaborative systems development and crowd work, raised in particular at CSCW in the recent years by numerous researchers, e.g. from Carnegie Mellon, Northwestern and Stanford Universities (Kittur et al., 2013; Culbertson et al., 2017; Zhu et al., 2014; Yu et al., 2014). In particular we are convinced that the Smart TV interface is promising for developing sustainable crowdsourcing solutions, and platforms that could truly involve older adults in crowd tasks by lowering ICT barriers and motivating them in various ways, from providing engaging content and edutainment, to ensuring social inclusion and enabling contribution to the society at large.

6.2. A follow-up long term study

However, further work is necessary to verify our presented preliminary considerations in a follow-up long-term study. This is feasible, as we have promising findings that some participants would like to continue, and some (P2 and P3) in fact have already subscribed, with one participant (P2) suggesting to do the task with grandchildren, thus implying that the intergenerational dimension could be an additional motivator. Below we present a brief overview of five main areas of future research.

6.2.1. Sustainability of the solution and verification of preliminary findings

Given our observations and feedback, in the long-term study we would like to verify the five above-mentioned design considerations as well as the sustainability of this solution in terms of: continued interest, likelihood of stopping playback to mark an error, detecting false error positives, as well as changing motivation depending on the context of the task (e.g. intergenerational). As observed in our preliminary study, older adults wish to engage in crowdsourcing if it resembles leisure activities they can do for fun. In general, we believe that many older adults will be able to sustain interest in the task, once it has a wider range of content available that may be better aligned with their interests.

6.2.2. Error detection rates

Another interesting research path is the examination of the inverse relationship between the study participants’ interest in a specific video, and the number of errors they detect. In order to test this, we intend to allow older adults to add an option to rate each video they have watched. To further examine the differences between detection rates the long term study would feature videos with both errors naturally occurring after human translation, as well as originating from machine translation-aided tasks. As our results indicate in both of these scenarios, the quality of translations can benefit from the involvement of older adults as proofreaders marking style and comprehension problems as well as grammar issues. Such data could be later used to facilitate Machine Learning processes in the Machine Translation field, but also help improve the quality of human translations.

6.2.3. Comprehension

Yet another aspect of the further study could focus on ways of addressing the comprehension difficulties we encountered, caused by the need to focus on many aspects of the subtitles and making choices regarding the error categories, in particular, by removing error categories altogether, and just enabling an error to be reported. There exists one related ground of research: crowdsourcing the errors that are perceived as errors by the older adults, and comparing them to mistakes which are generally regarded by the translation communities and edutainment scene as a whole. Older adults often have signaled problems with the speed and comprehension of the subtitles themselves. Freely gathered data on perceived errors could better inform ways of making these forms more accessible to older adults.

6.2.4. Ease of setup

At the same time, although the Smart TV interface and our application proved to be enjoyable and easy to interact with, it is important to verify if older adults would be able to set up the system on their own, and if not, how to make it easier for them. This alone would pave way for further inquiries into the feasibility of using Smart TV services for older adults.

6.2.5. Motivation

As is often the case with crowdsourcing, this task also depends on the scale of contributions. Even if some older adults, according to their psychological profile and motivation, may choose to just watch the videos for fun, the others may use the opportunity to detect errors, and with the effect of scale working the true errors would be detected more reliably. We would also like to test the motivational component involving statistics of the errors found, and possibly gamifying them. Once the DreamTV application has been more extensively tested and developed to include the suggestions, and to resolve the difficulties occurring, we would like to allow as many users as possible free access to the application, and to collect longitudinal quantitative data on the errors reported. At the same time, we would like to test the personal (increasing the positive self-image) and social (engaging in a community) potential of this type of task we postulated when designing the DreamTV crowdsourcing solution. We could test it by encouraging older adults to engage with the TED Translators community, members of which could request their help in proofreading their own translations with the help of the application.

6.3. Closing remarks

Although some of our findings can be generalized to inform future design further work needs to be conducted with larger and more diverse groups alongside follow-up long-term studies. For example, in terms of wealth, education, language abilities and proficiency with technology, as well as with younger adults or speakers of other languages (e.g. migrants). This will help to explore which of our preliminary findings may be universal, and which ones rather depend on factors specific to different groups.

Figure 7. As the application allows users to select their favorite videos and save their progress it has potential for sustaining engagement, which is also helpful for conducting long term studies.

7. Acknowledgments

We would like to thank the TED Translators community for their insights and work on making interesting ideas accessible to more people through their subtitles, as well as the older adults who participated in this study. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 690962.


  • (1)
  • dig (2014) 2014. Measuring Digital Skills across the EU: EU wide indicators of Digital Competence. https://ec.europa.eu/digital-single-market/en/news/measuring-digital-skills-across-eu-eu-wide-indicators-digital-competence. (2014).
  • UNr (2015) 2015. World Population Aging 2015. Technical Report. United Nations Department of Economic and Social Affairs. 9–39 pages.
  • pop (2017) 2017. Population structure and ageing. http://ec.europa.eu/eurostat/statistics-explained/index.php/Population_structure_and_ageing. (2017).
  • the (2017) 2017. Turning to Education for Fun. (Dec 2017). Retrieved August 20, 2018 from https://www.nytimes.com/2015/03/20/education/turning-to-education-for-fun.html
  • Alaoui and Lewkowicz (2015) M. Alaoui and M. Lewkowicz. 2015. Practical issues related to the implication of elderlies in the design process – The case of a Living Lab approach for designing and evaluating social TV services. IRBM 36, 5 (2015), 259–265. https://doi.org/10.1016/j.irbm.2015.06.002
  • Armstrong (2013) Mike Armstrong. 2013. The Development of a Methodology to Evaluate the Perceived Quality of Live TV Subtitles. Technical Report. British Broadcasting Company.
  • Aula (2004) Anne Aula. 2004. Learning to use computers at a later age. In HCI and the Older Population. University of Glasgow, Leeds, UK, 3–5.
  • Baghaei and Tabatabaee (2015) Purya Baghaei and Mona Tabatabaee. 2015. The C-Test: An integrative measure of crystallized intelligence. Journal of Intelligence 3, 2 (2015), 46–58.
  • Balcerzak et al. (2017) Bartłomiej Balcerzak, Wiesław Kopeć, Radosław Nielek, Sebastian Kruk, Kamil Warpechowski, Mateusz Wasik, and Marek Wegrzyn. 2017. Press F1 for Help: Participatory Design for Dealing with On-line and Real Life Security of Older Adults. In Proceedings of the 12th International Scientific and Technical Conference on Computer Sciences and Information Technologies, CSIT 2017, Vol. 1. IEEE, 240–243. https://doi.org/10.1109/STC-CSIT.2017.8098778
  • Balcerzak et al. (2018) Bartlomiej Balcerzak, Wieslaw Kopec, Radoslaw Nielek, Kamil Warpechowski, and Agnieszka Czajka. 2018. From Close the Door to Do not Click and Back. Security by Design for Older Adults. In Advances in Intelligent Systems and Computing II. Springer, 40–53. https://doi.org/10.1007/978-3-319-70581-1_3
  • Brewer et al. (2016) Robin Brewer, Meredith Ringel Morris, and Anne Marie Piper. 2016. Why would anybody do this?: Understanding Older Adults’ Motivations and Challenges in Crowd Work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246–2257.
  • Cámara de la Fuente (2014) L Cámara de la Fuente. 2014. Multilingual Crowdsourcing Motivation on Global Social Media, Case Study: TED OTP. Sendebar 25 (2014), 197–218.
  • Carmichael (1999) Alex Carmichael. 1999. Style guide for the design of interactive television services for elderly viewers. (1999).
  • Carroll et al. (2012) John Carroll, Gregorio Convertino, Umer Farooq, and Mary Beth Rosson. 2012. The firekeepers: Aging considered as a resource. Universal Access in the Information Society 11, 1 (03 2012), 7–15. https://doi.org/10.1007/s10209-011-0229-9
  • Culbertson et al. (2017) Gabriel Culbertson, Solace Shen, Erik Andersen, and Malte Jung. 2017. Have your cake and eat it too: Foreign language learning with a crowdsourced video captioning system. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 286–296.
  • Czaja et al. (2006) Sara J Czaja, Neil Charness, Arthur D Fisk, Christopher Hertzog, Sankaran N Nair, Wendy A Rogers, and Joseph Sharit. 2006. Factors predicting the use of technology: Findings from the center for research and education on aging and technology enhancement (CREATE). Psychology and aging 21, 2 (2006), 333.
  • Czaja and Lee (2009) Sara J Czaja and Chin Chin Lee. 2009. Information technology and older adults. Human-computer interaction: Designing for diverse users and domains (2009), 18–30.
  • Deterding et al. (2011) Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From game design elements to gamefulness: defining gamification. In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments. ACM, 9–15.
  • Dickinson et al. (2007) Anna Dickinson, John Arnott, and Suzanne Prior. 2007. Methods for human-computer interaction research with older people. Behaviour & Information Technology 26, 4 (2007), 343–352.
  • Djoub (2013) Zineb Djoub. 2013. ICT education and motivating elderly people. In Ariadna; cultura, educación y tecnología, Vol. 1. 88–92. https://doi.org/10.6035/Ariadna.2013.1.15
  • Etchegoyhen et al. (2014) Thierry Etchegoyhen, Lindsay Bywood, Mark Fishel, Panayota Georgakopoulou, Jie Jiang, Gerard Van Loenhout, Arantza Del Pozo, Mirjam Sepesy Maucec, Anja Turner, and Martin Volk. 2014. Machine Translation for Subtitling: A Large-Scale Evaluation.. In LREC. 46–53.
  • Ferrari (2013) Anusca Ferrari. 2013. DIGCOMP: A framework for developing and understanding digital competence in Europe. Technical Report.
  • Fisk et al. (2009) A.D. Fisk, S.J. Czaja, W.A. Rogers, N. Charness, and J. Sharit. 2009. Designing for Older Adults: Principles and Creative Human Factors Approaches, Second Edition. CRC Press.
  • Greenfield and Marks (2004) Emily A Greenfield and Nadine F Marks. 2004. Formal volunteering as a protective factor for older adults’ psychological well-being. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences 59, 5 (2004), S258–S264.
  • Hao (2008) Yanni Hao. 2008. Productive activities and psychological well-being among older adults. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences 63, 2 (2008), S64–S72.
  • Itoko et al. (2014) Toshinari Itoko, Shoma Arita, Masatomo Kobayashi, and Hironobu Takagi. 2014. Involving senior workers in crowdsourced proofreading. In International Conference on Universal Access in Human-Computer Interaction. Springer, 106–117.
  • Jansen et al. (2014) Dean Jansen, Aleli Alcala, and Francisco Guzman. 2014. Amara: A sustainable, global solution for accessibility, powered by communities of volunteers. In International Conference on Universal Access in Human-Computer Interaction. Springer, 401–411.
  • Kittur et al. (2013) Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301–1318.
  • Knowles and Hanson (2018a) Bran Knowles and Vicki L. Hanson. 2018a. Older Adults’ Deployment of ’Distrust’. ACM Trans. Comput.-Hum. Interact. 25, 4, Article 21 (Aug. 2018), 25 pages. https://doi.org/10.1145/3196490
  • Knowles and Hanson (2018b) Bran Knowles and Vicki L. Hanson. 2018b. The Wisdom of Older Technology (Non)Users. Commun. ACM 61, 3 (Feb. 2018), 72–77. https://doi.org/10.1145/3179995
  • Kopeć et al. (2017) Wiesław Kopeć, Katarzyna Abramczuk, Bartłomiej Balcerzak, Marta Juźwin, Katarzyna Gniadzik, Grzegorz Kowalik, and Radosław Nielek. 2017. A Location-Based Game for Two Generations: Teaching Mobile Technology to the Elderly with the Support of Young Volunteers. In eHealth 360. Springer, 84–91. https://doi.org/10.1007/978-3-319-49655-9_12
  • Kopeć et al. (2018a) Wiesław Kopeć, Bartłomiej Balcerzak, Radosław Nielek, Grzegorz Kowalik, Adam Wierzbicki, and Fabio Casati. 2018a. Older Adults and Hackathons: A Qualitative Study. Empirical Software Engineering 23, 4 (2018), 1895–1930. https://doi.org/10.1007/s10664-017-9565-6
  • Kopeć et al. (2018b) Wiesław Kopeć, Radosław Nielek, and Adam Wierzbicki. 2018b. Guidelines Towards Better Participation of Older Adults in Software Development Processes using a new SPIRAL Method and Participatory Approach. In Proceedings of the CHASE’18: International Workshop on Cooperative and Human Aspects of Software (ICSE ’18). ACM, New York, NY, USA. https://doi.org/10.1145/3195836.3195840
  • Kopeć et al. (2017) Wiesław Kopeć, Kinga Skorupska, Anna Jaskulska, Katarzyna Abramczuk, Radoslaw Nielek, and Adam Wierzbicki. 2017. LivingLab PJAIT: Towards Better Urban Participation of Seniors. In Proceedings of the International Conference on Web Intelligence (WI ’17). ACM, New York, NY, USA, 1085–1092. https://doi.org/10.1145/3106426.3109040
  • Kötteritzsch et al. (2014) Anna Kötteritzsch, Michael Koch, and Fritjof Lemân. 2014. Adaptive training for older adults based on dynamic diagnosis of mild cognitive impairments and dementia. In International Workshop on Ambient Assisted Living. Springer, 364–368.
  • Lave and Wenger (1991) J. Lave and E. Wenger. 1991. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press.
  • Lum and Lightfoot (2005) Terry Y Lum and Elizabeth Lightfoot. 2005. The effects of volunteering on the physical and mental health of older people. Research on aging 27, 1 (2005), 31–55.
  • McArdle et al. (2000) John J McArdle, Fumiaki Hamagami, William Meredith, and Katherine P Bradway. 2000. Modeling the dynamic hypotheses of Gf–Gc theory using longitudinal life-span data. Learning and Individual Differences 12, 1 (2000), 53–79.
  • Morrow-Howell (2010) Nancy Morrow-Howell. 2010. Volunteering in later life: Research frontiers. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences 65, 4 (2010), 461–469.
  • Morrow-Howell et al. (2003) Nancy Morrow-Howell, Jim Hinterlong, Philip A Rozario, and Fengyan Tang. 2003. Effects of volunteering on the well-being of older adults. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences 58, 3 (2003), S137–S145.
  • Nielek et al. (2017a) Radoslaw Nielek, Miroslaw Ciastek, and Wiesław Kopeć. 2017a. Emotions Make Cities Live: Towards Mapping Emotions of Older Adults on Urban Space. In Proceedings of the International Conference on Web Intelligence (WI ’17). ACM, New York, NY, USA, 1076–1079. https://doi.org/10.1145/3106426.3109041
  • Nielek et al. (2017b) Radoslaw Nielek, Marta Lutostańska, Wiesław Kopeć, and Adam Wierzbicki. 2017b. Turned 70?: It is Time to Start Editing Wikipedia. In Proceedings of the International Conference on Web Intelligence (WI ’17). ACM, New York, NY, USA, 899–906. https://doi.org/10.1145/3106426.3106539
  • Nunes et al. (2012) Francisco Nunes, Maureen Kerwin, and Paula Alexandra Silva. 2012. Design Recommendations for TV User Interfaces for Older Adults: Findings from the eCAALYX Project. In Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’12). ACM, New York, NY, USA, 41–48. https://doi.org/10.1145/2384916.2384924
  • Ortman et al. (2014) Jennifer M Ortman, Victoria A Velkoff, Howard Hogan, et al. 2014. An aging nation: the older population in the United States. Washington, DC: US Census Bureau (2014), 25–1140.
  • Orzeszek et al. (2017) Dorota Orzeszek, Wieslaw Kopec, Marcin Wichrowski, Radoslaw Nielek, Bartlomiej Balcerzak, Grzegorz Kowalik, and Malwina Puchalska-Kaminska. 2017. Beyond Participatory Design: Towards a Model for Teaching Seniors Application Design. In Proceedings of the ER Forum 2017 co-located with the 36th International Conference on Conceptual Modelling (ER 2017), Valencia, Spain, - November 6-9, 2017. (CEUR Workshop Proceedings), Vol. 1979. 29–43.
  • Pak and McLaughlin (2010) R. Pak and A. McLaughlin. 2010. Designing Displays for Older Adults. CRC Press.
  • Pan et al. (2015) Zhengxiang Pan, Chunyan Miao, Han Yu, Cyril Leung, and Jing Jih Chin. 2015. The effects of familiarity design on the adoption of wellness games by the elderly. In 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Vol. 2. IEEE, 387–390.
  • Rapeepisarn et al. (2006) Kowit Rapeepisarn, Kok Wai Wong, Chun Che Fung, and Arnold Depickere. 2006. Similarities and differences between learn through play and edutainment. In Proceedings of the 3rd Australasian conference on Interactive entertainment. Murdoch University, 28–32.
  • Romero-Fresco and Pöchhacker (2017) Pablo Romero-Fresco and Franz Pöchhacker. 2017. Quality assessment in interlingual live subtitling: The NTR Model. 16 (01 2017), 149–167.
  • Ryan and Deci (2000) Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist 55, 1 (2000), 68.
  • Sandhu et al. (2013) Jatinder Sandhu, Leela Damodaran, and Leonie Ramondt. 2013. ICT Skills Acquisition by Older People: Motivations for learning and barriers to progression. International Journal of Education and Ageing 3, 1 (2013), 25–42.
  • Silva et al. (2013) Paula Alexandra Silva, Francisco Nunes, Ana Vasconcelos, Maureen Kerwin, Ricardo Moutinho, and Pedro Teixeira. 2013. Using the Smartphone Accelerometer to Monitor Fall Risk while Playing a Game: The Design and Usability Evaluation of Dance! Don’t Fall. 8027 (07 2013), 754–763.
  • Tsai et al. (2015) Hsin-yi Sandy Tsai, Ruth Shillair, Shelia R Cotten, Vicki Winstead, and Elizabeth Yost. 2015. Getting grandma online: Are tablets the answer for increasing digital inclusion for Older Adults in the US? Educational gerontology 41, 10 (2015), 695–709.
  • Vallerand and O’Connor (1989) Robert J Vallerand and Brian P O’Connor. 1989. Motivation in the elderly: A theoretical framework and some promising findings. Canadian Psychology/Psychologie canadienne 30, 3 (1989), 538.
  • Vines et al. (2011) John Vines, Mark Blythe, Paul Dunphy, and Andrew Monk. 2011. Eighty something: banking for the older old. In Proceedings of the 25th BCS Conference on Human-Computer Interaction. British Computer Society, 64–73.
  • Vines et al. (2015) John Vines, Gary Pritchard, Peter Wright, Patrick Olivier, and Katie Brittain. 2015. An Age-Old Problem: Examining the Discourses of Ageing in HCI and Strategies for Future Research. ACM Trans. Comput.-Hum. Interact. 22, 1, Article 2 (Feb. 2015), 27 pages. https://doi.org/10.1145/2696867
  • von Helversen et al. (2018) Bettina von Helversen, Katarzyna Abramczuk, Wiesław Kopeć, and Radoslaw Nielek. 2018. Influence of Consumer Reviews on Online Purchasing Decisions in Older and Younger Adults. Decision Support Systems 113 (2018), 1–10. https://doi.org/10.1016/j.dss.2018.05.006
  • Wang and Kaufman (1993) Jing-Jen Wang and Alan S Kaufman. 1993. Changes in fluid and crystallized intelligence across the 20-to 90-year age range on the K-BIT. Journal of Psychoeducational Assessment 11, 1 (1993), 29–37.
  • Wolfson et al. (2014) Natalie E Wolfson, Thomas M Cavanagh, and Kurt Kraiger. 2014. Older adults and technology-based instruction: Optimizing learning outcomes and transfer. Academy of Management Learning & Education 13, 1 (2014), 26–44.
  • Yang and Lai (2010) Heng-Li Yang and Cheng-Yu Lai. 2010. Motivations of Wikipedia content contributors. Computers in Human Behavior 26, 6 (2010), 1377–1383. https://doi.org/10.1016/j.chb.2010.04.011
  • Yu et al. (2014) Lixiu Yu, Paul André, Aniket Kittur, and Robert Kraut. 2014. A comparison of social, learning, and financial strategies on crowd engagement and output quality. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 967–978.
  • Zhu et al. (2014) Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1445–1455.
  • Zichermann and Cunningham (2011) Gabe Zichermann and Christopher Cunningham. 2011. Gamification by design: Implementing game mechanics in web and mobile apps (1st ed.). ” O’Reilly Media, Inc.”.