PeyeDF: an Eye-Tracking Application for Reading and Self-Indexing Research

04/27/2019 ∙ by Marco Filetti, et al. ∙ Helsingin yliopisto aalto Microsoft 0

PeyeDF is a Portable Document Format (PDF) reader with eye tracking support, available as free and open source software. It is especially useful to researchers investigating reading and learning phenomena, as it integrates PDF reading-related behavioural data with gaze-related data. It is suitable for short and long-term research and supports multiple eye tracking systems. We utilised it to conduct an experiment which demonstrated that features obtained from both gaze and reading data collected in the past can predict reading comprehension which takes place in the future. PeyeDF also provides an integrated means for data collection and indexing using the DiMe personal data storage system. It is designed to collect data in the background without interfering with the reading experience, behaving like a modern lightweight PDF reader. Moreover, it supports annotations, tagging and collaborative work. A modular design allows the application to be easily modified in order to support additional eye tracking protocols and run controlled experiments. We discuss the implementation of the software and report on the results of the experiment which we conducted with it.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 7

page 8

page 9

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

Eye tracking is a valuable, non-invasive means of investigation. It has seen increased adoption rates in the past decade, thanks to gradual but steady technological and cost-related improvements (Jacob and Karn, 2003). Many fields of research have been taking advantage of its capabilities, ranging from design (Djamasbi et al., 2010), collaboration (Duchowski, 2002), learning (Gütl et al., 2005), cognition (Blair et al., 2009; Lallé et al., 2016; Wilson and Russell, 2003) and usability (Jacob and Karn, 2003).

To date, the experiment design, experiment procedure and utilization of the eye tracking devices have been a laborious task for users and researchers, in particular for a long run experiment. In many circumstances, most of the users fail having a tangible connection with the software user interface and the researchers have to spend huge amount of time on implementing the experiments using eye tracking software. In the context of providing an appealing user experience and easier experiment design procedure, in particular for long run experiments, we present PeyeDF, an open-source eye-tracking enabled reading tool. PeyeDF is suitable for the conduction of eye-tracking-based reading and learning experiments conducted over long periods of time. For users, PeyeDF is easy to use, lightweight, stable software that provides support for storing easily-performed manual document annotations. For experimenters, it provides out-of-the box support for multiple eye tracking platforms.

1.1. Related work

Deciphering the state of one’s mind by performing inferences on eye movements is a research topic that has been undertaken for various purposes. The span of this research area is wide-range and includes observer task inference (Borji and Itti, 2014), tagging image pleasantness (R.-Tavakoli et al., 2015), inferring traits (Hoppe et al., 2018), predicting one’s age (Zhang and Meur, 2018), information retrieval (R.-Tavakoli et al., 2016), etc. Reviewing all the related work is beyond the intent of this article, for brevity, we refer the readers to (Duchowski, 2002). A brief account of the most relevant work is, however, presented.

Investigating the relationship between eye movements and learning can provide great insight into cognitive functions. To investigate one’s cognitive performance during learning, reading has been a central focus in conjunction with the eye tracking. For example, it has been demonstrated that the duration and number of fixations increase along with the difficulty of the text being read (Rayner et al., 2006a). Such fixation-level measurements is utilised to assess how sentence construction affects comprehension difficulty (Traxler et al., 2002)

or to estimate the reader’s language expertise

(Kunze et al., 2013b). More examples are given in a review by (Richardson et al., 2007).

Within the concept of learning via “Lifelogging”, storing our data in personal databases (Gurrin et al., 2014) and retrieving previously manually annotated information is demonstrated to reinforce memory (O’Hara et al., 1998). Annotations might also be performed automatically with attentive documents (Buscher et al., 2012). Both are valid strategies for the location of important passages that exploit manually or implicitly (automatically) annotated information. One drawback of automated annotation approaches is that the reinforcement of learning induced by the act of manually annotating passages is lost. In this context, the possibility of manually annotating passages is a feature that must be provided by all learning platforms (Adler et al., 1998). The PeyeDF tries to alleviate this shortcoming by allowing both manual and implicit annotation schemes.

Electronic devices have introduced new research questions for learning, such as the investigation of screen-based reading behaviour. There are opposing views in favour (Chen et al., 2014; Liu, 2005) and against such technologies (Ackerman and Goldsmith, 2011; Mangen et al., 2013) in learning and education. Many factors such as meta-cognitive abilities, culture, familiarity, habit, improved display technologies and age are argued to influence performance of users in digital reading (Taipale, 2014; Franze et al., 2014; Ball and Hourcade, 2011). While electronic reading is gradually replacing paper-based reading (Adler et al., 1998; Liu, 2005), the reading experience is key to success in reading. In other words, being comfortable with digital tools can improve learning (Harrison, 2000). We, thus, need to have well-designed software, akin to PeyeDF, for learning studies.

Designing software for the collection of user data during reading and learning can be cumbersome; a number of software systems have been developed to facilitate such tasks. These are summarised in Table 1.

Adele is a e-learning suite that allows synchronisation of eye tracking data with course material (Gütl et al., 2005). Text 2.0 (Biedert et al., 2010b) and Eyebook (Biedert et al., 2010a) are two frameworks for web browsers designed to provide interactive feedback to gaze data. These two systems, along with that presented by (Campbell and Maglio, 2001), can distinguish between skimmed versus read text. Eye an Pen (Alamargot et al., 2006) enables synchronisation of handwriting with gaze data. Ogama is a system for slideshow study designs (Vosskühler et al., 2008). iTrace is an environment that enables eye tracking research on software systems, rather than static text (Shaffer et al., 2015). Eye tracking software prototypes have also been developed for tablets and smartphones, although these are limited in accuracy and speed due to current hardware (Wood and Bulling, 2014; Kunze et al., 2013a).

Task Tracer is a Windows application for tracking user activity across various tasks (Dragunov et al., 2005). It provides access to an open database that stores computer usage data. “Stuff I’ve Seen” is a system for personal data storage and retrieval (Dumais et al., 2016). Presto is a document storage system that organises material based on its contents, in addition to the traditional folder-based approach (Dourish et al., 1999). Xlibris is a tablet-based proprietary system for document storage and annotation (Schilit et al., 1998). Note that some these applications do not support eye tracking and are not open-source. They are, thus, not among the best choices for researchers.

Among the open-source applications, Varifocal Reader is an application that allows navigation of large text documents at various level of detail (e.g. headers, sub-headers) (Koch et al., 2014). It is aimed at exploring individual large documents (books), while PeyeDF is aimed at storing information about collections of documents of any size. GazeParser is an open-source library for low-cost eye tracking and data analysis; it consists of a video-based eyetracker and libraries (Sogo, 2013a). It is not aimed at reading nor does it support long-term data storage and retrieval. Eyerec (Santini et al., 2016) and openEyes (Li et al., 2006) are two open-source hardware specification for eye tracking.

To summarize, the current eye tracking-based systems, in one hand, do not support personal data storage (PDS) and can not support lifelogging. On the other hand, the PDS based systems do not support any eye tracking and can not be used in eye tracking research. The PeyeDF software addresses these shortcommings by providing support for eye tracking and PDS. Moreover, it is an open source, lightweight PDF reader that can be customized for numerous research purposes.

Name Reference Purpose Eye Tracking PDS* Open-source
Adele (Gütl et al., 2005) Learning
Text 2.0 (Biedert et al., 2010b) Reading
Eyebook (Biedert et al., 2010a) Reading
Eye and Pen (Alamargot et al., 2006) Learning
Ogama (Vosskühler et al., 2008) Presentations
iTrace (Shaffer et al., 2015) Programming
Eyetab (Wood and Bulling, 2014) Tablets
My Reading Life (Kunze et al., 2013a) Phones
Varifocal Reader (Koch et al., 2014) Large documents
Task Tracer (Dragunov et al., 2005) Office work
Stuff I’ve Seen (Dumais et al., 2016) PDS
Xlibris (Schilit et al., 1998) Tablets
GazeParser (Sogo, 2013a) Software library
Eyerec (Santini et al., 2016) Hardware
Openeyes (Li et al., 2006) Hardware
PeyeDF Reading

*Supports or includes a Personal Data Storage system.

Table 1. Comparison of software related to PeyeDF

1.2. Contribution

We introduce PeyeDF. It is an open-source eye-tracking enabled reading tool. The PeyeDF is designed to support personal data storage systems (PDS) and eye tracking support in combination with a lightweight document viewer that enhances user experience during learning experiments. Some of the unique properties of the PeyeDF includes,

  • tracking viewport position,

  • tracking zoom factors,

  • support for multiple eye tracking devices,

  • implicit data annotation

  • extendable modules,

  • supporting DiMe, a personal data storage system,

  • lightweight and user friendly interface

The PeyeDF eases the life of both experimenters and users in long run reading and learning experiments. It has been tested over a long (3+ years) at Aalto University, Finland and to run the experiment described in section 5. It can be also used as a stand alone self-indexing system when an eye tracker is not available.

2. Implementation

PeyeDF is available only for macOS and it has been highly optimised for the platform. All computation related to eye tracking data and behavioural data collection takes place in the background, so that the main thread (which updates the User Interface) is left unburdened, resulting in a smooth user experience; this is enabled by the implementation described in section 2. It also supports spotlight (described in subsection 3.3) and URL Types (explained in subsection 4.3). It implements the Swift API guidelines111https://swift.org/documentation/api-design-guidelines/ and the Apple Human Interface guidelines222https://developer.apple.com/design/human-interface-guidelines/macos/overview/themes/.

2.1. Installation

PeyeDF is a standard macOS application; as such, it can be installed by downloading a disk image from https://github.com/HIIT/PeyeDF/releases and dragging PeyeDF to the Applications folder. However, in order to function it requires the installation of DiMe and, optionally, a connection to a supported eye tracker. These steps are described below. An overview of the full system is given in Figure 1.

Figure 1. System overview. PeyeDF requires two way communication with DiMe. It supports the listed protocols for eye tracking communication. As discussed in subsection 4.7, the EyeDataProvider protocol generalises eye tracking support. It is also possible to export / import eye tracking and reading data via JSON files.

2.2. Prerequisites

PeyeDF requires DiMe for data storage and retrieval. DiMe is an open-source Personal Data Storage (PDS) system that supports multiple applications and data types (Sjöberg et al., 2016). DiMe itself requires the following software to be installed in order to be compiled.

2.3. Setup

Once the required software is installed, DiMe should be compiled and run. PeyeDF must then be configured to utilise DiMe using a predefined user and password. The procedure to do so is detailed below and the related code can be pasted into a Terminal (note that individual commands are separated by ‘;’ and otherwise each statement should be written in a single line).

  1. Clone the open-source DiMe repository:
    git clone --recursive https://github.com/HIIT/dime-server

  2. Run DiMe:
    cd dime-server; make run

  3. DiMe is ready Once ‘fi.hiit.dime.Application: Started’ appears; first-time compilation may take a few minutes.

  4. Navigate to http://localhost:8080 and create a user. For a test run, use Test1 as user and 123456 for password as these are the PeyeDF defaults.

  5. PeyeDF can now be started and should be fully functional. If a different user rather than the suggested Test1 was created during the previous step, navigate to PeyeDF Preferences, select the ‘DiMe’ tab and enter the chosen username and password.

Dime creates a .dime directory under the user’s home. This directory contains the full database; deleting this directory will reset DiMe. If the directory is copied or moved to another machine an installation of DiMe on that machine will be able to read the database (assuming that username and password match). DiMe can also be installed on a (local) network server so that it can be used by multiple users simultaneously.

3. Usage

PeyeDF works and behaves like a lightweight PDF reader – similarly to the stock ‘Preview’ application on macOS. PDF files can be opened with PeyeDF by default, if desired. Eye tracking support is not required for the application to work, although it is the main motivation for using PeyeDF, along with its data storage and indexing capabilities. A screenshot of the applications is provided in Figure 2, which also shows how PeyeDF captures the currently read paragraph while using an eye tracker.

Figure 2. Screenshot and paragraph detection. This figure shows a screenshot of PeyeDF (an instance of DocumentWindowController) with overlays representing fixations and their related paragraphs. A fixation is assigned to a paragraph when its height can be contained within 3 of visual angle from the fixation. PeyeDF stores all fixation and paragraph information in sets of ReadingEvents, as described in subsection 4.1.

PeyeDF supports two eye tracking hardware systems: SMI888https://www.smivision.com and Pupil Labs999https://www.pupil-labs.com. Hardware support is enabled via data input from the LSL101010https://github.com/sccn/labstreaminglayer, zeroMQ111111http://zeromq.org and MIDAS121212https://github.com/bwrc/midas protocols. If desired, PeyeDF can be extended to support additional protocols (and hence hardware) as discussed in subsection 4.7. To activate eye tracking while using PeyeDF, it is sufficient to select the desired protocol in the Preferences > Experiment dialog. An error message will be displayed if a valid connection was not be established. All gaze data is fetched in real time; at this stage, PeyeDF does not support off-line gaze recorders such as GazeParser (Sogo, 2013b). If eye tracking is currently being used, a tick for ‘Eye Tracking’ appears in the ‘Connections’ menu.

In the ‘Connections’ menu, a tick also indicates whether PeyeDF is successfully connected to DiMe. In case a connectivity problem occurs during usage, one of the images displayed in Figure 3 is displayed over the reading area to attract the user’s attention.

Figure 3. Error indicators. These images are shown over the reading area when a connection error is detected, in order to capture the user’s attention. The ‘Not receiving eye tracking data’ image is also shown when the user’s eyes have not been detected for an extended period of time (e.g. in case the user left the workstation).

PeyeDF collects eye tracking and reading data in the background, without noticeably affecting the reading experience. This is enabled by asynchronous processing of behavioural and eye tracking data in background queues (as described in subsection 4.6).

PeyeDF provides support for rapid annotations. Annotation support is critical for reading applications as actively highlighting passages can increase their retention (Fowler and Barker, 1974). It has been previously suggested that annotations in the digital world may be less effective than paper-based annotation when they are too complex to use (Ben-Yehudah and Eshet-Alkalai, 2014). Providing a means for quickly annotating sections of relevant text is then beneficial. To enable quick annotations, the ‘quick mark’ button (visible on the toolbar) should be activated. Annotations are then performed with a double-click on a paragraph to mark an “important” paragraph in yellow. A triple click would highlight the area as “critical”, using a stronger colour (red). It is also possible to annotate text by selecting a range of text and right clicking.

Tagging is also supported: users can assign tags (corresponding to short strings) to any range of text. Tags are stored in DiMe, and can subsequently searched and retrieved by PeyeDF or other supporting applications.

3.1. Refinder

It has been suggested that lack of visual cues in digital applications might impede reading (Ackerman and Goldsmith, 2011; Mangen et al., 2013). PeyeDF implements a Refinder functionality, which utilises colours, tags and overviews in order to provide spatiotemporal markers to previously read and annotated text. The Refinder has been included in PeyeDF for two reasons. Firstly, it improves the reading experience by providing a simple means to revisit previously highlighted text; this may motivate experiment participants in utilising PeyeDF rather than a default reader. Secondly, the Refinder has been designed in order to be extensible: it can be integrated into third party applications (as described in subsection 4.3) or it can be modified to investigate novel refinding approaches (all Refinder functionality is collected in the ‘Refinder’ folder in our repository131313http://github.com/HIIT/PeyeDF/tree/master/PeyeDF/Refinder).

Figure 4. Refinder screenshot. A screenshot of an instance of RefinderWindowController is shown in this figure. There can only be one such window in an instance of PeyeDF. Colours represent the type of annotations created by users (yellow for “important” and red for “critical”). In this example, blue represents unannotated paragraphs which contained at least three fixations. The colour bars represent the overall proportion of the given document which is marked in the given colour. Note that in the overview, blocks are overlaid over text, while in the main window annotations are displayed as a bar on the left side of the related paragraph.

3.2. Collaboration

PeyeDF also supports collaborative reading and tagging. To enable it, two (or more) collaborators must be connected to the same local network. Detected collaborators can be displayed by activating the Connections > Show network readers menu. One can then initiate collaboration by pressing the ‘invite’ button. Collaboration also requires DiMe. This allows users to “track” one another, if desired, showing exactly which paragraph is being read by the collaborator using eye tracking.

Real time sharing of gaze information has been shown to increase collaboration quality during problem solving (Schneider and Pea, 2013; Stein and Brennan, 2004), collaborative visual search (Brennan et al., 2008) and a planning task (Cherubini et al., 2010). It has also been shown to be useful, for example, in remote non-collaborative scenarios, such as virtual lectures (Sharma et al., 2015) or virtual conferencing (Vertegaal, 1999). Support for collaborative work in reading software is a desirable feature according to (Adler et al., 1998).

Application elements unique to the collaboration feature are shown in Figure 5. This feature is provided to keep the application on par with modern office packages and potentially allows for research in collaborative environments, with and without eye tracking.

Figure 5. Collaboration features. On the left, the list of active connections is shown; in this case, there is only one active connection. This displays the filename and title of the document currently being read by the collaborator(s). This item will also display a ‘Track’ checkbox upon the start of a collaborative reading session; this allows one collaborator to follow the gaze and / or reading viewport of the other. On the right, the overview of the collaborative session. Colours identify sections of the document which have been read by the local user, the collaborator, or both. This overview is a small window displayed along with the main DocumentWindowController

3.3. Spotlight integration

A minor but useful feature is spotlight integration. When searching for text on a macOS system, opening a PDF with PeyeDF will open it while searching for the previously searched text. Searched-for text is stored in SummaryReadingEvents. Events are explained in the next section.

4. Data Structures and APIs

This section describes the format of PeyeDF data structures, APIs, events and their usage. Figure 6 summarises the data types supported by DiMe. This information is also documented in the dime-server repository141414https://github.com/HIIT/dime-server/tree/master/src/main/java/fi/hiit/dime/data and in the corresponding wiki pages151515https://github.com/HIIT/PeyeDF/wiki/Data-Format.

For a generic user, accessing the stored data of an experiment is likely the most critical functionality of the software. Data can be retrieved in two ways: (1) by right clicking an entry (session) in the Refinder and choosing ‘Extract JSON’, which allows to save all ReadingEvents related to the selected session in a JSON file, and (2) by utilising the DiMe API, to be described in subsection 4.2. While the first approach is relatively easy, the second requires better understandings of the internals of the software, and it also enables one to further extend the software.

Figure 6. Data class diagram. Shows the relationships and format of data classes that PeyeDF exports. DiMeData is top-level class from which all other inherit; this is used in PeyeDF to identify classes which can be stored in DiMe. Field names were designed to be self-explanatory. Event is the most relevant class for retrieving eye data (along with its subclasses); this is explained in subsection 4.1. If needed, more details are provided, in the dime-server repository data folder or the PeyeDF repository wiki.

4.1. Data Types: Events and Information Elements

There exists two main data types, which should be distinguished from each other, (1) Events and (2) InformationElements. Events correspond to snapshots of user activity, while InformationElements represent documents or other information retrievable with a URI. In PeyeDF, there is only one value for InformationElement, which is ScientificDocument, a link to the PDF files to be retrieved. An InformationElement contains also a ContentHash and an appId. The first, a SHA hash of the textual content of a PDF, is used to identify textual similarity between PDFs. appId is simply a ContentHash with ‘PeyeDF_’ prefixed to distinguish between PeyeDF and other applications.

The data from experiments will be mostly stored in the Event and its variants. We, thus, dedicate most of this section to the Events data type. The software consists of several event data structures, namely Event, ReadingEvent, and SummaryReadingEvent. The first and most important is Event that contains Event.targettedResource field which contains the associated element of a given event, i.e., a pointer to InformationElement data type corresponding to the current event.

During the software operation, when a user wants to access a document for reading, the reading event is triggered to handle the retrieval of the relevant document and starts recording the action of reading. ReadingEvent

contains data collected from the moment the user started the act of reading until the reading stopped. A

ReadingEvent may last a few seconds or minutes. We assume reading action is stopped as soon as the window is scrolled or moved. A more detailed explanation of how we estimate whether the user is reading is provided in subsection 4.6.

Each ReadingEvent has a sessionId in terms of a string UUID in PeyeDF. During a reading session, the same sessionId is allocated to all ReadingEvents of that specific reading session. A session starts when the user opens a PDF document and terminates when the PDF is closed. Normally, one session comprises many events.

A ReadingEvent has the following fields: pageNumbers, pageLabels, pageRects and plainTextContent to convey the visible information within a session. pageNumbers and pageLabels contains the same number of elements and represent which page(s) that were visible to user. Page number is intended to be a page index starting from 0, while page label is the page number written on the page extracted from the PDF. If page label can not be extracted from PDF source the same indexing as page index will be used. plainTextContent contains all text that was visible on in the viewport(s). pageRects is an array of Rects, which represent positions of paragraph of various nature. These specify the viewport position when the event was generated and other paragraph-level information. Viewports are illustrated in Figure 7.

To distinguish Rects of different nature, the two fields of readingClass and classSource are introduced. The classSource defines what is the source of a Rect, e.g., eye tracker, a click, a viewport, etc. readingClass contains the information about the importance of the event, e.g., an important pargraph vs. a regular one.

Figure 7. Viewports. This figure displays the position of the viewport under different page arrangements. Note that origin is always at bottom left. When the viewport covers more than one page, the pageLabels and pageNumbers lists of a ReadingEvent will contain more than one value. The crop box represents the visible section of a PDF page (if a page has not been previously cropped, the crop box will correspond to the full page size). Viewports are specified using Rects within each ReadingEvent. Rects representing viewports are identified using a readingClass of 10 and a classSource of 1.

During eye tracking, a ReadingEvent can include one or several PageEyeData fields to contain information for fixation information. The fixations’ data is specified within multiple in-order arrays (e.g. horizontal and vertical positions, pupil size, duration). Their coordinates are specified in page space, with pageIndex identifying the page on which these took place. There is a pageEyeData entry for each page for which gaze data were detected. When using eye tracking, PeyeDF detects the paragraph of interest for the given fixation, using the implementation described in subsection 4.5.

A SummaryReadingEvent is triggered once a document is closed, i.e., a reading session is terminated. In consequence, the PeyeDF computes a summary for the reading session and adds it to the database. This information will be further used for future retrievals. The summary information consists of all the searched keywords and search hits in a document, along with their statistical information that may be useful for further relevance information retrieval within documents.

4.2. DiMe API

DiMe is the central backbone of PeyeDF data storage. PeyeDF populates DiMe using HTTP Rest API calls (POST or DELETE) in JSON; these data can be subsequently retrieved by consumers using the GET calls described in this subsection. PeyeDF itself uses GET calls for displaying data within the Refiner. The full DiMe API and configuration instructions are described by (Sjöberg et al., 2016).

Using the DiMe API, one can query the data structures discussed above. The queries useful in the PeyeDF context are: data/event, data/events, data/informationelement, data/informationelements, eventsearch and search. The singular endpoints (data/event and data/informationelement) are used to retrieve a single event or element as JSON. These terminate with the unique integer id that one wants to retrieve (e.g. data/informationelement/1 will fetch the first element stored in DiMe). The plural data endpoints retrieve 0 or more JSON objects and can be filtered using GET parameters. These are discussed more in detail in the next two subsections.

4.2.1. data/events

This endpoint is used to fetch ReadingEvents. This is used when fetching reading data created by PeyeDF in the format described in section 4. Parameters can be used to filter by type, actor, sessionId, elemId or contentHash. Query parameters can be chained.

Possible queries:

  • type allows to filter by type. PeyeDF uses two types: #ReadingEvent and #SummaryReadingEvent. Note that the type has to be prefixed by http://www.hiit.fi/ontologies/dime/ (this is to distinguish between types created by different organisations).

  • actor filters by the “creator” of the event. In the case of PeyeDF, there is only one actor: PeyeDF.

  • sessionId as previously indicated, each reading session, per document (from window open to window close) is assigned one sessionId. This query parameter can be used to fetch all events generated during a single session. This is an alphanumeric string.

  • elemId filters only events related to a given information element Id (an integer). This is used to obtain all events related to an individual PDF file.

  • contentHash retrieves all events which were related to a specific file. Using hashes to identify files allows them to be found after renaming or moving.

Example

This query will fetch all events created by PeyeDF. The targettedResource field within each returned event is used to identify the document related to a given event. This allows one to parse the list while searching for specific metrics and relate the given event(s) to a document.

4.2.2. data/informationelements

This endpoint fetches Scientific Documents (which are the only type of information element used by PeyeDF). This endpoint is used to fetch document(s) with a specific contentHash, of a specific type or with a given tag.

  • contentHash is an alphanumeric string uniquely identifying the document or file.

  • type; PeyeDF uses only type: #ScientificDocument. As with event queries, PeyeDF prefixes type with http://www.hiit.fi/ontologies/dime/.

  • tag allows to retrieve only documents for which a user assigned the given tag. The tag can appear in any part of the document.

Example

http://localhost:8080/api/data/informationelements?type=http://www.hiit.fi/Fontologies/dime/#ScientificDocument&tag=done fetches all ScientificDocuments (PDF files) that the user tagged with ‘done’.

4.2.3. eventsearch

To perform textual searches across all events, use the eventsearch endpoint. The query requires a parameter for the given query; optionally, any of the event filter parameters described above can be used.

Example

http://localhost:8080/api/eventsearch?query=pipe&type=http://www.hiit.fi/ontologies/dime/#ReadingEvent looks for ReadingEvents for which the word “pipe” appeared in the viewport. This can be used to retrieves parts of a document which have been read (or likely to have been read) by the user. The found results can be manually filtered to retrieve only events which contained fixations, for example.

4.2.4. search

To find documents that contain given the text use the search endpoint.

Example

http://localhost:8080/api/eventsearch?query=pipe&type=http://www.hiit.fi/ontologies/dime/#ScientificDocument finds all PDFs that contain the word “pipe” in any part of their body and have been previously opened by the user.

4.3. URL types

PeyeDF also supports the native macOS ‘URL Types’ corresponding to interprocess communication protocol. That is, third party applications can ask PeyeDF to open documents and optionally focus on given areas. Is possible to send messages to PeyeDF using URLs starting with the peyedf:// protocol. A custom URL allows for relatively simple interaction between applications.

There are two modes that can be used:

  • reader opens a file for a new reading session. It can be used by a search engine to direct to a specific search result.

  • refinder opens a previously read file. Useful for personal indexing services which access own reading data.

These appear where the host would normally appear in a URL (e.g. peyedf://refinder/…)

4.3.1. Reader

The format of the request is:

After reader/ one can refer to a file using a full path. If the file is already known by DiMe, it is also possible to use a contentHash, a given sessionId or appId.

The search query is optional. If enclosed in double quotes (" or a %22 escape sequence) it is treated as an exact phrase search (as opposed to an “all words” search).

Optionally, one can focus to a specific block of text using the page and rect parameters. Otherwise, point and page can be used to focus on a broad area of text (to focus on a section heading, for example). The page parameter must be present if either point or rect is used. It is also possible to use page alone, to focus on the beginning of a given page.

Examples

This opens the ‘Wyble et al 2009.pdf’ file and searches for the “attentional blink” phrase.

This opens the ‘Yan et al 2007.pdf’ file and focuses on the given area with origin 200 (x) , 400 (y) and size 200 (width), 100 (height).

This opens the same file as before, but instead navigates to a point approximately in the middle of page 2 (or 1, if we start counting from 0).

4.3.2. Refinder

The format of the request is:

sessionId is the unique id assigned to each reading session, as described in subsection 4.1.

Optionally, a specific block of text can be focused upon using the page and rect parameters. Otherwise, point and page can be used to focus on a broad area of text (to focus on a section heading, for example). The page parameter must be present if either point or rect is used. It is also possible to use page alone, to focus on the beginning of a given page.

Examples

Opens the same reading session and in addition focuses to that point on page 2 (if starting from 1).

Opens another reading session and focuses on the given block of text on page 4 (if starting from 1). The given block of text could be something previously annotated, for example.

4.4. Data collection

We, here, explain the mechanisms that are used for collecting data. We first explain how and when a ReadingEvent is generated and which parts of the software are involved. We then discuss the support for eye tracking devices, followed by explaining how an experiment is built using the software.

4.4.1. ReadingEvent generation

The central classes involved in data collection are HistoryManager,
DocumentWindowController, PDFReader, to be described below. Their interaction is visually depicted in Figure 8.

The flow of events begins with Notification events, which are generated by user activity. The notifications are collected by an active DocumentWindowController, which triggers the HistoryManager by identifying itself as the centre of the user’s attention. When the software detects a user has finished reading (e.g. switched to another window), the active controller notifies the HistoryManager again.

Upon first call of the HistoryManager, it sets up a short timer (entryTimer 2 seconds, defined in PeyeConstants.minReadTime, that is keep timing meanwhile the user is reading, preventing noisy data from being recorded. The timer asks the DocumentWindowController to generate a ReadingEvent, containing the viewport information and eye tracking data when available. This event will be sent to DiMe later. The timer stops upon exitTimer which is normally triggered only when eye tracking is off and the user leaves the workstation.

Figure 8. Generation of ReadingEvents. This diagram depicts the flow involved in the generation of ReadingEvents. As detailed in subsection 4.6, this allows PeyeDF to run smoothly even when large amounts of eye tracking data are being collected and re-referenced to text and paragraphs. Events are initially dispatched via application-wide notifications (top right). These notifications are generated by events representing user or eye tracking activity. This is described in more detail in subsubsection 4.4.1.

4.4.2. HistoryManager

The HistoryManager class is a singleton class. That is, only one instance of it is created and is shared across the application. This class acts as a central buffer for behavioural and eye tracking data, which are frequently sent to DiMe. These data are stored in the currentReadingEvent field; this field is cleared and its previous contents are sent to DiMe every time exit() is called.

4.4.3. DocumentWindowController

Every document window displayed by PeyeDF is an instance of DocumentWindowController. It calls the entry() and exit() methods in HistoryManager to record all the information. Upon appearing, the window sets up a number of Notifications, which monitor user and eye tracking activity. Every time the user “starts reading”, the DocumentWindowController calls entry() in HistoryManager passing itself as a parameter. This identifies the current window as the window the user is currently paying attention to. When the user “stops reading” exit() is called.

The “start reading” and “stops reading” actions are defined by the type of Notification that are being triggered. For example, if the window starts moving, loses focus or it is otherwise not visible it is assumed that the user stopped reading. Similarly, exit() is called when the user’s gaze was lost for a relatively long amount of time (defined in PeyeConstants.maxReadTime). Otherwise, when the window stops moving, or a scroll event is terminated, it is assumed that the user started reading, and entry() is called. This sets up a short timer, (entryTimer, 2 seconds) which upon fires calls the HistoryManager.entryTimerFire() method, which signals that the user is actively reading. This generates a ReadingEvent in the background containing PDF information (e.g. viewports), ready to be sent to DiMe once the user stops reading.

4.4.4. PDFReader

Every DocumentWindowController contains a reference to PDFReader. This is a subclass of PDFBase, which represents any instance of a PDF file displayed in PeyeDF. PDFReader is specialised in reading detection and interaction with the user. PDFReader contains the getViewPortStatus() method, called by HistoryManager.entryTimerFire(). PDFReader also contains convenience methods for creating annotations and tagging while reading. The PDFBase superclass contains anything relatable to a PDF, such as annotations, tags, searching support. Each DocumentWindowController controls one instance of PDFReader.

4.5. Paragraph detection

PeyeDF uses the distance of the user from the screen to determine which paragraph is currently being read (or annotated) by the user. Distance is obtained from the eye tracker, if available – otherwise it defaults to 60 cm. When a fixation is detected and it falls within text, the PDFBase.pointToParagraphRect() is called by DocumentWindowController to determine the height covered by 3° of visual angle, which should cover abundantly the area covered by the fovea (Ojanpää et al., 2002). This only covers the area corresponding to the current textual paragraph; in other words, double newlines are skipped at the edges, if present (detected using textual from within the PDF information).

This information is stored in the EyeRectangle class, which is converted into Rects and sent to DiMe within ReadingEvents. When this happens, the splitAndCrop function is called to split EyeRectangles which are too big (as defined by maxHeight). To sum up, this procedure unites whole paragraphs unless they reach maxHeight.

Algorithms and data structures regarding paragraph operations are defined in Paragraphs.swift161616https://github.com/HIIT/PeyeDF/blob/master/PeyeDF/Model/Paragraphs.swift. This includes uniting colliding rectangles, and converting Rects stored in DiMe back to annotated blocks of text.

4.6. Performance

PeyeDF was designed to maximise both usability of the software and reliability of eye tracking data. High usability is achieved by behaving similarly to a lightweight PDF reader and providing tagging and Refinder functionality. Eye tracking and behavioural data are collected using background Grand Central Dispatch171717https://developer.apple.com/guides/threading.html (GCD) queues. The main queue for eye data processing is called eyeQueue in HistoryManager. This has a default priority, which in GCD is third, below userInteractive and userInitiated. Each eye tracker implementation also has its own default priority queue (e.g. LSLManager has a queue field to which all data callbacks are dispatched).

To prevent consuming too much CPU power and reduce unnecessary data consumption, PeyeDF is conservative regarding the creation of events. It is assumed that the user is not reading unless no interaction with the software happened for a predefined amount of time (2 seconds, defined in PeyeConstants.minReadTime) while the window is still currently being focused (i.e. in front).

4.7. Eye tracker support

PeyeDF currently supports three types of protocols to enable eye tracking. These are MIDAS, LSL (for SMI eye trackers) and zeroMQ (for Pupil Labs glasses).

The Extras folder in our repository181818https://github.com/HIIT/PeyeDF/tree/master/Extras, provides supporting files for SMI trackers and Pupil Lab glasses. For SMI, we provide a script that converts SMI network packets into LSL streams. For Pupil Labs, we provide a plugin that allows surfaces (2-D frames) to send fixations over zeroMQ. This is because PeyeDF requires fixation-level information. In case these are not available, a possible alternative is to use the MIDAS framework to calculate them in real time, which we support. This framework is described by (Henelius and Torniainen, 2018).

Assuming fixation-level data can be obtained, implementing support for additional eye trackers requires two steps: 1) creating a class that adopts the EyeDataProvider protocol and 2) adding the given class to the EyeDataProviderType enum.

4.7.1. Adopting the EyeDataProvider protocol

The protocol generalises eye tracking behaviour by providing a number of implemented methods (protocol extensions) that should simply be called by the adopter at the correct times; these are sendLastRaw(), sendFixations(), eyeStateChange() and eyeConnectionChange(). Normally, these are be called in response to receipt of gaze data (i.e. within data callbacks, depending on the specific implementation). The protocol adopter should also provide start() and stop() methods, which control activation and deactivation of the data stream. In addition, it should hold references to the fixationDelegate, available, eyesLost and fixationDelegate fields. The provided adopters (e.g. LSLManager191919https://github.com/HIIT/PeyeDF/tree/master/PeyeDF/Eye%20Tracking/LSLManager.swift can be used as a reference for new implementations.

4.7.2. The EyeDataProviderType enum

The enum is used a reference that informs PeyeDF on the current types of eye trackers supported. Hence, in addition to protocol implementation, a minor sequence of two-line edits to the EyeDataProviderType enum are required to fully integrate the eye tracker into PeyeDF. These are a case statement in the EyeDataProviderType, a new instance of the class under associatedTracker, a string for description and an increased count variable. For example, a tracker called NewTracker would be associated to a .newtracker enum which would call NewTracker(), a description of "New Tracker" while count should be increased from 4 to 5. The current version of EyeDataProviderType can be used as a reference. Completing this step will add the eye tracker to the Preferences > Experiment > Eye Tracker menu and will immediately provide eye tracking data upon selection.

4.8. Experiment creation

PeyeDF can be utilised to conduct user experiments. The approach we recommend is to create a git branch (in github, or a private local branch). Subsequently it is suggested to create an additional target that will be used to run the experiment. It is possible to duplicate the original ‘PeyeDF’ target into another (e.g. ‘PeyeDF Experiment 1’).

Under the Xcode target’s build settings, additional flags can be added to identify the target in the application’s code. That is, under ‘Other Swift Flags’ create a flag using the -Dkeyword syntax. For example, a flag called ExperimentalTarget requires a -DExperimentalTarget flag. One can then add target-specific code using:

#if ExperimentalTarget
    // anything experiment-specific
#endif

This way, patches can be “pushed upstream” or “pulled downstream” independently of experiment-specific implementations. We provide the ‘PeyeDF Questions’ target as an example of how to define experimental targets. This target starts the Refinder and will run through a predefined list of papers, asking a given set of multiple-choice questions. It was utilised to run the second session of the experiment described in the following section.

5. Experiment

The experiment tested the possibility of utilising fixation data collected in the past to predict reading comprehension which takes place in the future. That is, fixation data collected during a first reading of a given passage was utilised to predict performance (answer correctness) that took place during a subsequent reading of the same passage. The experiment was ran utilising PeyeDF, while data analysis was performed in Matlab 2018a.

The study was divided into two sessions. During the first session, participants were instructed to read a sequence of papers and find specific information within each. The second session took place exactly one week after the first; participants revisited the papers, answering multiple-choice questions that referred to the information they had to locate during the first session. We demonstrate that fixation data collected during the first session predicts performance in the second.

5.1. Question creation

We selected four papers, and in relation to each we created a number of questions that referred to a specific passage. We selected papers with the following requirements: they had to be relatively easy to read, while covering topics of general interest (e.g. health). They also had to contain enough text and information to allow the creation of a total of 24 questions. The questions also had to be grouped so that 4 consecutive questions were related to the same topic (and were therefore located within the same section / paragraph). We gave the name “target topic” to the topic covered by a sequence of four related questions. This way, we obtained 6 “target topics” per paper. We then split the 6 target topics into two groups of three. The resulting two “target topic groups” were called A and B. The split was done randomly but did not change across participants (e.g. group A in paper 1 always referred to the same set of questions).

To summarise, each target topic group (A or B) contained 3 target topics, each containing 4 questions. One participant would then be assigned only one target topic group within a paper. Splitting into target topic groups allowed us to assign a different set of questions to every participant, even though they all read the same papers.

The four papers we utilised were:

  1. The Impact of Asthma and Allergic Diseases on Schoolchildren: Are They at Increased Risk of Absenteeism and Poor School Performance? (Bener, 2011)

  2. Acceptability of psychotherapy, pharmacotherapy, and self-directed therapies in Australians living with chronic hepatitis C (Stewart et al., 2013)

  3. The effectiveness of a motivational interviewing primary-care based intervention on physical activity and predictors of change in a disadvantaged community (Hardcastle et al., 2012)

  4. Choice and placebo expectation effects in the context of pain analgesia (Rose et al., 2012)

An additional paper (titled Waterpipe smoking among US university students, (Primack et al., 2012)) was used for practice. This contained only 2 topics with 2 questions each in total, all belonging to the same group (group A).

The resulting set of questions were saved in JSON files and are available in our GitHub repository202020https://github.com/HIIT/PeyeDF/tree/master/Extras/Experiment/Questions.

We used PeyeDF’s tagging functionality to identify the location in the text corresponding to the answer to each question. This information was stored in JSON files, available in our repository212121https://github.com/HIIT/PeyeDF/tree/master/Extras/Experiment/Answer_Location_Tags. Each JSON file contains the answer location for a given paper (1 to 4), group (A or B) and target topic (1 to 3). The answer location corresponds to a page and a rectangle (text box) which encloses the text containing the answer (along with the actual text of the answer).

5.1.1. First session

Each participant was assigned the set of four papers, randomly shuffled and with randomly assigned target topic groups. An assignment could then be represented as 1A, 2B, 4B, 3B, indicating that the participant had to start from paper 1 and would only be shown questions from group A, then continue to paper 2, group B, etc.

Participants were given 15 minutes to freely familiarise themselves with each paper before being asked to perform the information finding task. After this familiarisation phase, participants were asked to find specific information by showing to them instructions on a computer screen. The following example depicts instructions for paper 1, group A, target topics 1 and 2:

  • In Bener2011_asthma.pdf

    • An overall (worldwide) view of asthma (target topic 1)

      • Overall asthma prevalence

      • Factors correlated to asthma

      • Lifestyle disease modifier factors

      • Number of people affected by pollutants

    • Findings about asthma and allergic diseases (target topic 2)

      • Most accepted method to measure asthma prevalence

      • Correlation between age and pulmonary infection

      • Relationship to sex

      • Relationship between rhinitis prevalence and sex

The text shown for each item (identified by asterisk in the example) is called “summary” and is saved in the JSON files available in our repository.

We utilised the practice paper to guide participants in this task, ensuring they understood the procedure correctly.

Participants were asked to simply look up the information, without putting any additional effort in memorising it. Participants were instructed to proceed to the next paper after finding all the necessary information. They were also asked to advise the experimenter immediately in case they found it difficult to locate the information they were asked to find (pilot testing was used to verify that all information could be found autonomously by participants). Once finished, they were instructed to advise the experimenter.

5.1.2. Second session

The second session was run using PeyeDF’s Refinder functionality (previously shown in Figure 4). This feature automatically displayed the papers the given participant read during the first session, in order. The Refinder funcionality was modified using the ‘PeyeDF Questions’ target222222https://github.com/HIIT/PeyeDF/tree/master/PeyeDF/Questions. This modification displays a multiple choice question below the Refinder. Participants had to select one out of three possible answers and confirm their answer before proceeding to the next question. They performed this task in the same order as session one.

The following example refers to paper 1, group A, target topic 1, question 1:

  • In western countries, how high can the prevalence of asthma and allergies be (among children)?

    • 1/3

    • 1/4

    • 1/5

The first item of the Question JSON files always corresponds to the correct answer. Answers were displayed to participants in random order. Participants were trained for this task using the practice paper, along with the experimenter.

Note that questions do not exactly match the ‘summary’ participants were shown during the first session. Our aim was to induce participants into re-reading the text, rather than simply rely on their memory. This is because we were interested in assessing whether fixation data collected in the past predicts reading comprehension in the future (correlations between fixation duration and memory have already been demonstrated in (Hollingworth et al., 2001), for example).

Response correctness and time passed in relation to each question were recorded and exported to JSON for analysis.

5.2. Participants

In total, we recruited 12 participants. We rejected participants that spent less than 30 minutes or more than 90 minutes to complete the first session. We also rejected participants when eye tracking calibration error was above 1° of visual angle. This resulted in 7 valid participants. We also set a threshold for rejection at 80 % correct answers given in the second session; however, none of the remaining participants were below this threshold.

Each participant was paid with two movie tickets.

5.3. Data analysis

As mentioned in subsection 5.1, we used Tags to store the location within the text (page number and text box) corresponding to each answer. We utilised these to identify all fixations collected in the first session that fell within 3° of visual angle from the centre of a given answer location. This way, we obtained a set of fixation positions and durations in relation to each answer, for each participant. No fixation data from the second session was utilised.

We populated a Matlab table in which each row corresponded to an answer. Each column represented information related to the answer (correctness, time spent to answer) and the question (target topic number, group, paper number). Each column also contained features calculated from eye tracking data such as average fixation duration and travel. We created three Support Vector Machine classifiers that predicted response correctness using a given set of columns; the classifiers and related features (table columns) were:

  • Eye: classifier for gaze data

    • Average fixation duration

    • Median fixation duration

    • Sum of fixation durations

    • Distance of eyes from screen

    • Total travel (sum of saccades)

    • Forward travel (sum of saccades moving left or towards the bottom of the screen)

    • Backward travel (sum of saccades moving upwards or towards the top of the screen)

  • Topic: classifier for question data

    • Paper number (1 to 4)

    • Group identity (A or B)

    • Target topic number (1 to 3)

  • All: classifier that considered all table columns.

All classifiers were trained using leave-one-out cross validation. We utilised the Area Under the ROC Curve (AUC) as the performance measure for each classifier, as this has shown to be appropriate for unbalanced data data (Jeni et al., 2013)

. Our data was unbalanced since the predicted variable (response correctness) was skewed, since 87% of responses were correct.

We utilised a permutation test (Good, 2013) to compute a p-value for each classifier. That is, every classifier was run an additional 1000 times with the predicted variable (response correctness) randomly permuted across table rows. A p-value is obtained by dividing the number of times that a randomised AUC was greater than the true-observed AUC by the number of permutations.

5.4. Results and discussion

Of the tree classifiers, only All was significant (AUC: .62, p = 0.041). Topic (AUC: .37, p > 0.5) and Eye (AUC: .50, p > 0.5) failed to classify response correctness above chance levels. This indicates that gaze data significantly contributed to predictions only when coupled to question-related information. However, question related information alone (the Topic classifier) was not sufficient to predict response correctness. Similarly, gaze data alone is not sufficient. This indicates collecting gaze data for future use, which the main feature of PeyeDF, is valuable for reading-based experiments, as long as some text-related information is also collected. We believe this is because not all target topics were of identical difficulty. Combining topic data with gaze-related data provides enough information to discriminate between topics which were been previously read carefully and were relatively easy to understand, agains topics which were not carefully read and / or were harder to understand.

Note that topic data was linked to the design of the experiment in this example. In more realistic scenarios, topic features could be replaced with natural language metrics (such as number nouns and verbs) (Bird et al., 2009) or approximate location (page number, section) of the text currently read by the user.

During pilot testing, SMI eye tracker reported a pupil size of 0, indicating that pupil size could not be estimated reliably (this could be due to lighting in our lab or the fact that we used a highly interactive application). Gaze-based classification could be further improved by including pupil size measurements, as these correlate to workload (Beatty et al., 2000). Despite this limitation, we demonstrated that fixation data provides valuable information and can significantly help in discriminating between correctly and incorrectly comprehended text.

New frameworks recently released for the macOS platform such as the Basic neural network subroutines (BNSS - part of the Accelerate framework), the Convolutional Neural Networks (CNN - part of the MetalPerformanceShaders framework) or the upcoming Create ML framework could be integrated in the future, making deployment of machine learning applications and experiments integrated within PeyeDF, improving user-friendliness.

Collaborative functions are included in PeyeDF. The experimental design we utilised in the previous section could be adapted for collaborative reading research. In particular, we identified two independent “target topic groups” within a number of papers. Each paper could then be read by two participants, one focusing on group A and the other on group B. Gaze and behavioural data could then be used to investigate wether their features can distinguish between the two participants, for example.

PeyeDF is a desktop application; however, many of its functions are based on PDFKit, which is a framework recently made available for iOS. This means that PeyeDF could be extended to run on tablets or smartphones, given compatible eye-tracking hardware. It could also run as an eye-tracking data viewer only.

PeyeDF enables longer term research than traditional experimental software. That is, machine shutdowns and startups do not interfere with the data, which is stored in DiMe. Hence, it can be used to track how user habits change over time using its integration of behavioural and eye tracking data stored within ReadingEvents. For example, it has been suggested that expertise affects gaze over time, on the same participants (van Gog and Scheiter, 2010). PeyeDF would be suitable for experiments in this directions, as experiments can be run over long periods of time. In educational settings, PeyeDF could be used to assess retention of read material depending on eye tracking data (Rayner et al., 2006b).

The Refinder functionality included in PeyeDF is relatively basic and could be extended to investigate how different implementation affect information retrieval and retention. For example, it would be possible to obtain read paragraph information from ReadingEvents and use it to compute a cloud of potentially related documents as suggested by (Buscher et al., 2012; Buscher and Dengel, 2009).

MIDAS (Henelius and Torniainen, 2018), which is supported by PeyeDF, allows real-time processing of sensor data. It could be used to gather data collected from additional sensors, enabling research on the relationship between physiology and emotions experienced during reading, as suggested by (Graesser and D’Mello, 2012).

Native annotation support in PeyeDF is useful for annotation-directed research. That is, different types of annotation can be proposed to users; subsequent DiMe queries can be used to determine which were the most used annotations and experiments can be designed to assess information retention depending on annotation type.

PeyeDF can also be used for more controlled, short-term studies. PDF pages contained the required stimuli can be created and read by PDF. Full screen support allows to display the required stimuli covering the whole screen, if required. Page movements could then be triggered by code, rather than user activity.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Author Contributions

MF wrote the paper, designed the software and the related experiment. HT contributed to the writing of the article and to the analysis of eye tracking data. NR and GJ supervised the software and experimental design.

Funding

The present work has been supported by the Finnish Funding Agency for Innovation (TEKES) under the project Revolution of Knowledge Work (funding decisions 558/31/2013 and 5159/31/2014).

References

  • (1)
  • Ackerman and Goldsmith (2011) R Ackerman and M Goldsmith. 2011. Metacognitive regulation of text learning: on screen versus on paper. J Exp Psychol Appl 17, 1 (2011), 18–32.
  • Adler et al. (1998) Annette Adler, Anuj Gujar, Beverly L Harrison, Kenton O’Hara, and Abigail Sellen. 1998. A diary study of work-related reading: design implications for digital reading devices Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 241–248.
  • Alamargot et al. (2006) D Alamargot, D Chesnet, C Dansac, and C Ros. 2006. Eye and pen: A new device for studying reading during writing. Behavior Research Methods 38, 2 (2006), 287–299.
  • Ball and Hourcade (2011) Robert Ball and Juan Pablo Hourcade. 2011. Rethinking Reading for Age From Paper and Computers. International Journal of Human-Computer Interaction 27, 11 (2011), 1066–1082.
  • Beatty et al. (2000) Jackson Beatty, Brennis Lucero-Wagoner, et al. 2000. The pupillary system. Handbook of psychophysiology 2, 142-162 (2000).
  • Ben-Yehudah and Eshet-Alkalai (2014) Gal Ben-Yehudah and Yoram Eshet-Alkalai. 2014. The Influence of Text Annotation Tools on Print and Digital Reading Comprehension. In Proceedings of the 9th Chais Conference for the Study of Innovation and Learning Technologies: Learning in the Technological Era, Y. Eshet-Alkalai, A. Caspi, N. Geri, Y. Kalman, V. Silber-Varod, and Y. Yair (Eds.). The Open University of Israel, 28–35.
  • Bener (2011) Abdulbari Bener. 2011. The Impact of Asthma and Allergic Diseases on Schoolchildren: Are They at Increased Risk of Absenteeism and Poor School Performance? In Advanced Topics in Environmental Health and Air Pollution Case Studies. InTech.
  • Biedert et al. (2010a) Ralf Biedert, Georg Buscher, and Andreas Dengel. 2010a. The eyeBook – Using Eye Tracking to Enhance the Reading Experience. Informatik Spektrum 33, 3 (2010), 272–281.
  • Biedert et al. (2010b) Ralf Biedert, Georg Buscher, Sven Schwarz, Manuel Möller, Andreas Dengel, and Thomas Lottermann. 2010b. The text 2.0 framework. In Workshop on Eye Gaze in Intelligent Human Machine Interaction. Citeseer, 114–117.
  • Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media.
  • Blair et al. (2009) MR Blair, MR Watson, RC Walshe, and F Maj. 2009. Extremely selective attention: eye-tracking studies of the dynamic allocation of attention to stimulus features in categorization. Journal of experimental psychology. Learning, memory, and cognition 35, 5 (2009), 1196–1206.
  • Borji and Itti (2014) Ali Borji and Laurent Itti. 2014. Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision 14, 3 (2014), 29. https://doi.org/10.1167/14.3.29 arXiv:/data/journals/jov/932817/i1534-7362-14-3-29.pdf
  • Brennan et al. (2008) SE Brennan, X Chen, CA Dickinson, MB Neider, and GJ Zelinsky. 2008. Coordinating cognition: the costs and benefits of shared gaze during collaborative search. Cognition 106, 3 (2008), 1465–1477.
  • Buscher and Dengel (2009) Georg Buscher and Andreas Dengel. 2009. Gaze-based filtering of relevant document segments. In International World Wide Web Conference (WWW). 20–24.
  • Buscher et al. (2012) Georg Buscher, Andreas Dengel, Ralf Biedert, and Ludger V. Elst. 2012. Attentive documents: Eye tracking as implicit feedback for information retrieval and beyond. ACM Trans. Interact. Intell. Syst. 1, 2 (2012), 1–30.
  • Campbell and Maglio (2001) Christopher S Campbell and Paul P Maglio. 2001. A robust algorithm for reading detection. In Proceedings of the 2001 workshop on Perceptive user interfaces. ACM, 1–7.
  • Chen et al. (2014) Guang Chen, Wei Cheng, Ting-Wen Chang, Xiaoxia Zheng, and Ronghuai Huang. 2014. A comparison of reading comprehension across paper, computer screens, and tablets: Does tablet familiarity matter. J. Comput. Educ. 1, 2-3 (2014), 213–225.
  • Cherubini et al. (2010) Mauro Cherubini, Marc-Antoine Nüssli, and Pierre Dillenbourg. 2010. This is it!: Indicating and looking in collaborative work at distance. Journal of Eye Movement Research 3 (2010), 1–20.
  • Djamasbi et al. (2010) Soussan Djamasbi, Marisa Siegel, and Tom Tullis. 2010. Generation Y, web design, and eye tracking. International Journal of Human-Computer Studies 68, 5 (2010), 307–323.
  • Dourish et al. (1999) Paul Dourish, W Keith Edwards, Anthony LaMarca, and Michael Salisbury. 1999. Presto: an experimental architecture for fluid interactive document spaces. ACM Transactions on Computer-Human Interaction (TOCHI) 6, 2 (1999), 133–161.
  • Dragunov et al. (2005) Anton N Dragunov, Thomas G Dietterich, Kevin Johnsrude, Matthew McLaughlin, Lida Li, and Jonathan L Herlocker. 2005. TaskTracer: a desktop environment to support multi-tasking knowledge workers Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 75–82.
  • Duchowski (2002) AT Duchowski. 2002. A breadth-first survey of eye-tracking applications. Behav Res Methods Instrum Comput 34, 4 (2002), 455–470.
  • Dumais et al. (2016) Susan Dumais, Edward Cutrell, Jonathan J Cadiz, Gavin Jancke, Raman Sarin, and Daniel C Robbins. 2016. Stuff I’ve seen: a system for personal information retrieval and re-use ACM SIGIR Forum, Vol. 49(2). ACM, 28–35.
  • Fowler and Barker (1974) Robert L Fowler and Anne S Barker. 1974. Effectiveness of highlighting for retention of text material. Journal of Applied Psychology 59, 3 (1974), 358.
  • Franze et al. (2014) Juliane Franze, Kim Marriott, and Michael Wybrow. 2014. What academics want when reading digitally, Steven Simske and Sebastian Rönnau (Eds.), Vol. the 2014 ACM symposium. ACM Press, New York, New York, USA, 199–202.
  • Good (2013) Phillip Good. 2013. Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Science & Business Media.
  • Graesser and D’Mello (2012) Arthur C. Graesser and Sidney D’Mello. 2012. Moment-To-Moment Emotions During Reading. Read Teach 66, 3 (2012), 238–242.
  • Gurrin et al. (2014) Cathal Gurrin, Alan F Smeaton, and Aiden R Doherty. 2014. Lifelogging: Personal big data. Foundations and Trends(R) in Information Retrieval 8, 1 (2014), 1–125.
  • Gütl et al. (2005) Christian Gütl, Maja Pivec, Christian Trummer, Victor Manuel Garcia-Barrios, Felix Mödritscher, Juergen Pripfl, and Martin Umgeher. 2005. Adele (adaptive e-learning with eye-tracking): Theoretical background, system architecture and application scenarios. European Journal of Open, Distance and E-Learning 8, 2 (2005).
  • Hardcastle et al. (2012) Sarah Hardcastle, Nicola Blake, and Martin S Hagger. 2012. The effectiveness of a motivational interviewing primary-care based intervention on physical activity and predictors of change in a disadvantaged community. Journal of behavioral medicine 35, 3 (2012), 318–333.
  • Harrison (2000) B.L. Harrison. 2000. E-books and the future of reading. IEEE Comput. Grap. Appl. 20, 3 (2000), 32–39.
  • Henelius and Torniainen (2018) Andreas Henelius and Jari Torniainen. 2018. MIDAS: Open-source framework for distributed online analysis of data streams. SoftwareX 7 (2018), 156–161.
  • Hollingworth et al. (2001) Andrew Hollingworth, Carrick C Williams, and John M Henderson. 2001. To see and remember: Visually specific information is retained in memory from previously attended objects in natural scenes. Psychonomic Bulletin & Review 8, 4 (2001), 761–768.
  • Hoppe et al. (2018) Sabrina Hoppe, Tobias Loetscher, Stephanie A. Morey, and Andreas Bulling. 2018. Eye Movements During Everyday Behavior Predict Personality Traits. Frontiers in Human Neuroscience 12 (2018), 105. https://doi.org/10.3389/fnhum.2018.00105
  • Jacob and Karn (2003) Robert JK Jacob and Keith S Karn. 2003. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The mind’s eye. Elsevier, 573–605.
  • Jeni et al. (2013) Laszlo A. Jeni, Jeffrey F. Cohn, and Fernando De La Torre. 2013. Facing Imbalanced Data Recommendations for the Use of Performance Metrics. In 2013 HUMAINE ASSOCIATION CONFERENCE ON AFFECTIVE COMPUTING AND INTELLIGENT INTERACTION (ACII) (International Conference on Affective Computing and Intelligent Interaction). Humaine Assoc; IEEE Comp Soc; Comp Vis; Multimedia Lab; Univ Geneva, Swiss Ctr Affect Sci; GFK Verein; Technicolor; Telono; Brain Prod; Inst Telecom; Telecom ParisTech; Swiss Natl Sci Fdn; Soc Academique Geneve; Amer Assoc Artificial Itelligence, 245–251. https://doi.org/{10.1109/ACII.2013.47} 5th Biannual Conference of the Humaine-Association on Affective Computing and Intelligent Interaction (ACII), Geneva, SWITZERLAND, SEP 02-05, 2013.
  • Koch et al. (2014) Steffen Koch, Markus John, Michael Worner, Andreas Muller, and Thomas Ertl. 2014. VarifocalReader — In-Depth Visual Analysis of Large Text Documents. IEEE Trans. Visual. Comput. Graphics 20, 12 (2014), 1723–1732.
  • Kunze et al. (2013a) Kai Kunze, Shoya Ishimaru, Yuzuko Utsumi, and Koichi Kise. 2013a. My reading life. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication - UbiComp ‘13 Adjunct. ACM Press, New York, New York, USA.
  • Kunze et al. (2013b) Kai Kunze, Hitoshi Kawaichi, Kazuyo Yoshimura, and Koichi Kise. 2013b. Towards inferring language expertise using eye tracking. In CHI ‘13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA ‘13. ACM Press, New York, New York, USA.
  • Lallé et al. (2016) Sébastien Lallé, Cristina Conati, and Giuseppe Carenini. 2016. Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data.. In

    International Joint Conference on Artificial Intelligence

    . 2529–2535.
  • Li et al. (2006) Dongheng Li, Jason Babcock, and Derrick J Parkhurst. 2006. openEyes: a low-cost head-mounted eye-tracking solution. In Proceedings of the 2006 symposium on Eye tracking research & applications. ACM, 95–100.
  • Liu (2005) Ziming Liu. 2005. Reading behavior in the digital environment: Changes in reading behavior over the past ten years. Journal of documentation 61, 6 (2005), 700–712.
  • Mangen et al. (2013) Anne Mangen, Bente R. Walgermo, and Kolbjørn Brønnick. 2013. Reading linear texts on paper versus computer screen: Effects on reading comprehension. International Journal of Educational Research 58 (2013), 61–68.
  • O’Hara et al. (1998) Kenton O’Hara, Fiona Smith, William Newman, and Abigail Sellen. 1998. Student readers’ use of library documents: implications for library technologies Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 233–240.
  • Ojanpää et al. (2002) Helena Ojanpää, Risto Näsänen, and Ilpo Kojo. 2002. Eye movements in the visual search of word lists. Vision Research 42, 12 (2002), 1499–1512.
  • Primack et al. (2012) Brian A Primack, Ariel Shensa, Kevin H Kim, Mary V Carroll, Mary T Hoban, E Victor Leino, Thomas Eissenberg, Kathleen H Dachille, and Michael J Fine. 2012. Waterpipe smoking among US university students. Nicotine & Tobacco Research 15, 1 (2012), 29–35.
  • R.-Tavakoli et al. (2015) Hamed R.-Tavakoli, Adham Atyabi, Antti Rantanen, Seppo J. Laukka, Samia Nefti-Meziani, and Janne Heikkilä. 2015. Predicting the Valence of a Scene from Observers’ Eye Movements. PLOS ONE 10, 9 (09 2015), 1–19. https://doi.org/10.1371/journal.pone.0138198
  • R.-Tavakoli et al. (2016) Hamed R.-Tavakoli, Hanieh Poostchi, Jaakko Peltonen, Jorma Laaksonen, and Samuel Kaski. 2016. Preliminary Studies on Personalized Preference Prediction from Gaze in Comparing Visualizations. In Advances in Visual Computing, George Bebis, Richard Boyle, Bahram Parvin, Darko Koracin, Fatih Porikli, Sandra Skaff, Alireza Entezari, Jianyuan Min, Daisuke Iwai, Amela Sadagic, Carlos Scheidegger, and Tobias Isenberg (Eds.). Springer International Publishing, Cham, 576–585.
  • Rayner et al. (2006a) Keith Rayner, Kathryn H Chace, Timothy J Slattery, and Jane Ashby. 2006a. Eye Movements as Reflections of Comprehension Processes in Reading. Scientific Studies of Reading 10, 3 (2006), 241–255.
  • Rayner et al. (2006b) Keith Rayner, Kathryn H Chace, Timothy J Slattery, and Jane Ashby. 2006b. Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading 10, 3 (2006), 241–255.
  • Richardson et al. (2007) Daniel C. Richardson, Rick Dale, and Michael J. Spivey. 2007. Eye Movements in Language and Cognition. John Benjamins Publishing Company.
  • Rose et al. (2012) Jason P Rose, Andrew L Geers, Heather M Rasinski, and Stephanie L Fowler. 2012. Choice and placebo expectation effects in the context of pain analgesia. Journal of behavioral medicine 35, 4 (2012), 462–470.
  • Santini et al. (2016) Thiago Santini, Wolfgang Fuhl, Thomas Kübler, and Enkelejda Kasneci. 2016. Eyerec: An open-source data acquisition software for head-mounted eye-tracking. In International Conference on Vision Theory and Applications (VISAPP).
  • Schilit et al. (1998) Bill N Schilit, Gene Golovchinsky, and Morgan N Price. 1998. Beyond paper: supporting active reading with free form digital ink annotations. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 249–256.
  • Schneider and Pea (2013) Bertrand Schneider and Roy Pea. 2013. Real-time mutual gaze perception enhances collaborative learning and collaboration quality. Intern. J. Comput.-Support. Collab. Learn. 8, 4 (2013), 375–397.
  • Shaffer et al. (2015) Timothy R. Shaffer, Jenna L. Wise, Braden M. Walters, Sebastian C. Müller, Michael Falcone, and Bonita Sharif. 2015. iTrace: enabling eye tracking on software artifacts within the IDE to support software engineering tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2015. ACM Press, New York, New York, USA.
  • Sharma et al. (2015) Kshitij Sharma, Patrick Jermann, and Pierre Dillenbourg. 2015. Displaying Teacher’s Gaze in a MOOC: Effects on Students’ Video Navigation Patterns. In Design for Teaching and Learning in a Networked World. Springer, 325–338.
  • Sjöberg et al. (2016) Mats Sjöberg, Hung-Han Chen, Patrik Floréen, Markus Koskela, Kai Kuikkaniemi, Tuukka Lehtiniemi, and Jaakko Peltonen. 2016. Digital Me: Controlling and Making Sense of My Digital Footprint. In Proceedings of the 5th International Workshop on Symbiotic Interaction.
  • Sogo (2013a) H Sogo. 2013a. GazeParser: an open-source and multiplatform library for low-cost eye tracking and analysis. Behavior Research Methods 45, 3 (2013), 684–695.
  • Sogo (2013b) H Sogo. 2013b. GazeParser: an open-source and multiplatform library for low-cost eye tracking and analysis. Behav Res Methods 45, 3 (2013), 684–695.
  • Stein and Brennan (2004) Randy Stein and Susan E Brennan. 2004. Another person’s eye gaze as a cue in solving programming problems. In Proceedings of the 6th international conference on Multimodal interfaces. ACM, 9–15.
  • Stewart et al. (2013) Benjamin JR Stewart, Deborah Turnbull, Antonina A Mikocka-Walus, Hugh AJ Harley, and Jane M Andrews. 2013. Acceptability of psychotherapy, pharmacotherapy, and self-directed therapies in Australians living with chronic hepatitis C. Journal of clinical psychology in medical settings 20, 4 (2013), 427–439.
  • Taipale (2014) Sakari Taipale. 2014. The affordances of reading/writing on paper and digitally in Finland. Telematics and Informatics 31, 4 (2014), 532–542.
  • Traxler et al. (2002) Matthew J Traxler, Robin K Morris, and Rachel E Seely. 2002. Processing Subject and Object Relative Clauses: Evidence from Eye Movements. Journal of Memory and Language 47, 1 (2002), 69–90.
  • van Gog and Scheiter (2010) Tamara van Gog and Katharina Scheiter. 2010. Eye tracking as a tool to study and enhance multimedia learning. Learning and Instruction 20, 2 (2010), 95–99.
  • Vertegaal (1999) Roel Vertegaal. 1999. The GAZE groupware system: mediating joint attention in multiparty communication and collaboration. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 294–301.
  • Vosskühler et al. (2008) A Vosskühler, V Nordmeier, L Kuchinke, and AM Jacobs. 2008. OGAMA (Open Gaze and Mouse Analyzer): open-source software designed to analyze eye and mouse movements in slideshow study designs. Behav Res Methods 40, 4 (2008), 1150–1162.
  • Wilson and Russell (2003) Glenn F Wilson and Christopher A Russell. 2003. Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks. Human Factors 45, 4 (2003), 635–643.
  • Wood and Bulling (2014) Erroll Wood and Andreas Bulling. 2014. Eyetab: Model-based gaze estimation on unmodified tablet computers. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 207–210.
  • Zhang and Meur (2018) A. T. Zhang and B. O. Le Meur. 2018. How Old Do You Look? Inferring Your Age from Your Gaze. In 2018 25th IEEE International Conference on Image Processing (ICIP). 2660–2664. https://doi.org/10.1109/ICIP.2018.8451219