Retrieving Event-related Human Brain Dynamics from Natural Sentence Reading

03/29/2021 ∙ by Xinping Liu, et al. ∙ University of Tasmania 0

Electroencephalography (EEG) signals recordings when people reading natural languages are commonly used as a cognitive method to interpret human language understanding in neuroscience and psycholinguistics. Previous studies have demonstrated that the human fixation and activation in word reading associated with some brain regions, but it is not clear when and how to measure the brain dynamics across time and frequency domains. In this study, we propose the first analysis of event-related brain potentials (ERPs), and event-related spectral perturbations (ERSPs) on benchmark datasets which consist of sentence-level simultaneous EEG and related eye-tracking recorded from human natural reading experiment tasks. Our results showed peaks evoked at around 162 ms after the stimulus (starting to read each sentence) in the occipital area, indicating the brain retriving lexical and semantic visual information processing approaching 200 ms from the sentence onset. Furthermore, the occipital ERP around 200ms presents negative power and positive power in short and long reaction times. In addition, the occipital ERSP around 200ms demonstrated increased high gamma and decreased low beta and low gamma power, relative to the baseline. Our results implied that most of the semantic-perception responses occurred around the 200ms in alpha, beta and gamma bands of EEG signals. Our findings also provide potential impacts on promoting cognitive natural language processing models evaluation from EEG dynamics.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

page 5

page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Preserving human languages by text has been influencing the human cognitive growth, and it is also considered as the primary medium for preservation of natural language processing (NLP). However, research on how human brains retrieve and process the meaning from text-based languages is relatively few. Previous research Mason and Just (2006) has used custom reading materials to explore particular facets of linguistic material such as event-related potentials (ERPs) and eye motions. Later on, using Electroencephalography (EEG) in conjunction with eye-tracking has become significant measures to study the temporal dynamics of natural sentence reading, because EEG signals have excellent time resolution and comparatively low costs to be recorded Dimigen et al. (2011), Loberg et al. (2018), Hollenstein and Zhang (2019). For example, a recent study Pfeiffer et al. (2020) indicated that text-based emotion analyze could be significantly enhanced by computing fixation-related potentials (FRPs) as one of ERPs on a neurocognitive dataset named ZuCo 1.0 Hollenstein et al. (2018) that recorded from human readers during naturalistic reading. To measure event-related synchronisations, Roach and Mathalon (2008) quantified the timeline of event-related spectral perturbations (ERSPs) that could generate brain dynamic changes in amplitude of the broadband EEG frequency spectrum.

However, the studies above only focused on wording or naive reading tasks without comprehensive event-related brain dynamic analyze, so it is still unknown how ERP and ERPS changes in human natural sentence reading. In this work, we investigated EEG brain dynamics of ERPs and ERSPs through both channel and component forms for the total of 18 subjects during a recent natural sentence reading task, ZuCo 2.0 (Hollenstein et al., 2020). This task supports to analyse the differences in cognitive processing between natural reading, and also allows us to precisely extract the EEG signals for sentence-level processing and detecting human retrieving lexical and semantic information in natural sentence reading.

Figure 1:

Natural Sentence Reading Experiment. (a) Natural reading phases. (b) Each sentence onset and finished state and switching to the next sentence by control pad. The sentence reading and sentence switching were performed alternately, and each sentence switching lasted 100 ms, while sentence reading lasted from 1.2 second to 12 seconds. (c) Location map of multi-channel EEG.

2 Related Work

ERPs Frank et al. (2013, 2015) are electrical potential responses or reaction measured in relation to an event, such as reading each sentence in our study. For example, N200 is a negative wave in connection with a baseline because it occurs at around 200 to 350 ms after the onset of the first word in a sentence, but it may cause some mismatch issues after an infrequent change in the case of visual stimuli Brattico et al. (2006). The recent study Pfeiffer et al. (2020) has indicated that text-based emotion analyze can be significantly enhanced by using FRPs to analyze human readers during naturalistic reading, but it only focused on the potentials of word-based reading. Also, Hale et al. (2018), Jat et al. (2019) applied ERPs as a neuroimage feature to retrieve the lexical and semantic information from the natural language content for designing cognitive word embedding evaluation in NLP, but still, there has some significant loss possibly due to not precise patterns. In terms of ERSPs, one study zhang2017investigation investigated emotion and working memory analyze to reading, but it did not touch the natural sentence reading tasks.

3 Method

3.1 Experiment Paradigm and Data

The natural sentence reading experiment and data was performed and collected in ZuCo 2.0 (Hollenstein et al., 2020), which recorded from 18 healthy subjects who are all native English speakers originating from Canada, USA, UK or Australia. The sentences were displayed one by one at the same position on the monitor during the natural reading paradigm. The text was displayed on a light grey backdrop in black with font size 20-point Arial, resulting in a letter height of 0.8 mm or 0.674 . During the entire reading task, a sum of 80 letters or 13 words were presented per line, and the subjects who participate in this natural reading experiment read each sentence at their own speed.

In the task, the 349 sentences were displayed one by one at the same position on the monitor during the natural reading task for each subject. The recording devices include both 128-channel EEG and eye-tracking that used to collect brain signals and eye motions when subjects conducted natural sentence reading, as shown in Fig. 1. In this work, one trial was considered as reading each sentence, and there are a total of 6,282 trials (18 subjects multiplied by 349 sentences in the natural reading paradigm).

3.2 Data Processing and analyze

EEG Processing

The raw EEG data were loaded by EEGLAB (Delorme and Makeig, 2004)

with the removal of bad electrodes, keeping 105 EEG channels. We then removed electromyogram (EMG) and electrooculogram (EOG) artifacts and minimised other artifacts by setting a finite impulse response filter 1-50Hz. Next, we conducted the EEG electrodes re-referencing by the vertex electrode Cz, and the re-referenced EEG data were measured by Z Score formula normalisation. Furthermore, we conducted the data segmentation to extract all trials where two time-locking events (sentence onset event and sentence finished event) were selected. What is more, the trial limits were 100 ms before starting sentence reading (-100 ms, act as the baseline) to 1200 ms after the reading event (+1200 ms), because the sentence interval sets 100 ms and the minimum interval between sentence onset and sentence finished is 1223 ms.

In addition, to represent synchronous brain activities, decomposing the pre-processed data by Independent Component analyze (ICA) is an effective approach to investigate the diversity of source information typically contained in EEG data. In this study, we merged a total of 18 pre-processed single-subject EEG data and conducted the statistical comparisons by running the merged-subjects ICA decomposition to find the potential similar independent component sources across 18 subjects.

EEG analyze: ERP/ERSP in channel-based and component-based forms

The 1-D event latency scalp maps and 2-D ERP/ERSP images could produce to intuitively discover the brain dynamics of human natural sentence reading in both forms of channel-based and component-based EEG patterns, respectively. In terms of ERPs, to analyse the channel-based ERPs, we chose the most relevant channel to the corresponding events from the scalp map where the power is concentrated among a total of 18 subjects. Meanwhile, to analyse component-based ERPs, we calculated and selected the power spectrum for isolated components to identify which component contributes the most to the ERPs. For ERSPs, similarly, we determined the channel which is most relevant to the corresponding events and analysed spectral perturbations, and then calculated and selected a component that the source contributes the most to the task. Please note that the significance level sets 0.01 with removing the baseline for plotting the ERSPs in this study.

4 Results

Our results show the comprehensive group analyze and visualisation of 18 subjects of EEG dynamics for natural sentence reading, including 1-D event latency potentials plots and 2-D ERP and ERSP images in channel-based and component-based forms.

Channel/Component 1-D event latency potentials

As shown in Fig. 2, from 1-D event latency potentials among all subjects, we found highest peaks at 162 ms from the onset natural sentence reading, which may indicate that a significant EEG dynamic (particular in reduced power in the occipital region) approaching 200 ms (N200) and this finding also supported by evidence for the early latency of lexical and semantic information retrieval in visual word recognition (Hauk et al., 2012, Pfeiffer et al., 2020). Thus, we assume that the human start retrieves lexical and semantic information near-simultaneously at the central occipital region (Oz channel) within 200 ms of the sentence reading onset.

Figure 2: Channel-based 1-D event latency potientials with the scalp map among a total of 18 subjects for natural sentence reading.

Following above, we also extracted 7 most significant component maps from 1-D event latency potentials, as shown in Fig. 3, which showed the occipital component contributed most to the task (marked as rank 1). The results in the section are used for determining that the occipital region is the most contribution area related to the natural sentence reading task, because the occipital results displayed the strongest response in the text-based retrieval of sentiment information than that of the other regions of the brain.

Figure 3: Components-based 1-D event latency potientials with the scalp map among a total of 18 subjects for natural sentence reading.

Channel/Component 2-D ERP Results

As shown in Fig. 4-A, we located the channel Oz as the central occipital region to plot channel-based ERP results, which present the negative power (marked as the blue colour) around 200 ms in short reaction time (between 1-2000 trials). However, from the occipital component-based ERP image as shown in Fig. 4-B, we observed the positive power (marked as the red colour) around 200 ms in medium (between 2000-4000 trials) and long (between 4000-6000 trials) reaction times. This is a new finding that may be beneficial for retrieving the diverse patterns of brain visual reading processing.

Figure 4: 2-D ERP results for natural sentence reading in channel-based (Oz) and component-based (occipital) forms.

Channel/Component 2-D ERSP Results

In terms of the channel-based and component-based ERSPs in the occipital region, they demonstrated a similar performance around 200 ms, as shown in Fig. 5. The Oz-channel ERSP as presented in Fig. 5-A, at the time of around 200ms, demonstrated increased high alpha (10-12Hz), high beta (25-30Hz), and high gamma (40-50Hz) power, and decreased low beta (13-25Hz) and low gamma (30-40Hz) power, relative to the baseline (the interval of sentence reading). In terms of the occipital component-channel ERSP around 200 ms, Fig. 5-B presented increased high alpha (10-12Hz) and high gamma (40-50Hz) power, and decreased low beta (13-25Hz) and low gamma (30-40Hz) power, relative to the baseline (the interval of sentence reading).

Furthermore, ERSP results in the occipital region around 200 ms primarily consist with the longer latencies covering from 300 ms to 1200 ms (the ending point of each sentence reading), particularly in the enhanced high gamma (40-50Hz) and the reduced low beta (13-25Hz) and low gamma (30-40Hz) power spectral, relative to the baseline, suggesting the patterns for retrieving lexical and semantic information in the natural sentence reading task.

Figure 5: 2-D ERSP results for natural sentence reading in channel-based (Oz) and component-based (occipital) forms.

5 Conclusion

Our study successfully identified the spatiotemporal neural dynamics of sentiment processing during the naturalistic reading of English sentences. Combining high-density EEG and eye-tracking data, we provide clear evidence that essential processes of lexical and semantic information retrieval occur around the 200 ms of each sentence onset, by retrieving EEG patterns of negative power and positive power in short and long reaction times and increased high gamma and decreased low beta and low gamma power. We believe that these EEG dynamics found from this study for human language understanding could benefit for semantic representations of sentence reading and cognitive evaluation modelling for NLP.

References

  • E. Brattico, M. Tervaniemi, R. Naatanen, and I. Peretz (2006) Musical scale properties are automatically processed in the human auditory cortex. Gene Expression Patterns 1117, pp. 162–174 (English). External Links: Document, ISSN 1567-133X Cited by: §2.
  • A. Delorme and S. Makeig (2004)

    EEGLAB: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis

    .
    Journal of neuroscience methods 134 (1), pp. 9—21. External Links: Document, ISSN 0165-0270, Link Cited by: §3.2.
  • O. Dimigen, W. Sommer, A. Hohlfeld, A. M. Jacobs, and R. Kliegl (2011) Coregistration of eye movements and eeg in natural reading: analyses and review. Journal of experimental psychology. General 140 (4), pp. 552—572. External Links: Document, ISSN 0096-3445, Link Cited by: §1.
  • S. L. Frank, I. Fernandez Monsalve, R. L. Thompson, and G. Vigliocco (2013) Reading time data for evaluating broad-coverage models of english sentence processing. Behavior Research Methods 45 (4), pp. 1182–1190. Cited by: §2.
  • S. L. Frank, L. J. Otten, G. Galli, and G. Vigliocco (2015) The erp response to the amount of information conveyed by words in sentences. Brain and Language 140, pp. 1–11. Cited by: §2.
  • J. Hale, C. Dyer, A. Kuncoro, and J. Brennan (2018) Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 2727–2736. External Links: Link, Document Cited by: §2.
  • O. Hauk, C. Coutout, A. Holden, and Y. Chen (2012) The time-course of single-word reading: evidence from fast behavioral and brain responses. NeuroImage 60 (2), pp. 1462–1477. External Links: Link, Document Cited by: §4.
  • N. Hollenstein, J. Rotsztejn, M. Troendle, A. Pedroni, C. Zhang, and N. Langer (2018) ZuCo, a simultaneous eeg and eye-tracking resource for natural sentence reading. Scientific Data 5, pp. . Cited by: §1.
  • N. Hollenstein, M. Troendle, C. Zhang, and N. Langer (2020) ZuCo 2.0: a dataset of physiological recordings during natural reading and annotation. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 138–146 (English). External Links: Link, ISBN 979-10-95546-34-4 Cited by: §1, §3.1.
  • N. Hollenstein and C. Zhang (2019) Entity recognition at first sight improving NER with eye movement information. CoRR abs/1902.10068. External Links: Link, 1902.10068 Cited by: §1.
  • S. Jat, H. Tang, P. P. Talukdar, and T. M. Mitchell (2019)

    Relating simple sentence representations in deep neural networks and the brain

    .
    CoRR abs/1906.11861. External Links: Link, 1906.11861 Cited by: §2.
  • O. Loberg, J. Hautala, J. A. Hämäläinen, and P. H. T. Leppänen (2018)

    Semantic anomaly detection in school-aged children during natural sentence reading – a study of fixation-related brain potentials

    .
    PLoS ONE 13 (12), pp. e0209741. External Links: Document, Link Cited by: §1.
  • R. A. Mason and M. A. Just (2006) Chapter 19 - neuroimaging contributions to the understanding of discourse processes. In Handbook of Psycholinguistics (Second Edition), M. J. Traxler and M. A. Gernsbacher (Eds.), pp. 765 – 799. External Links: ISBN 978-0-12-369374-7, Document, Link Cited by: §1.
  • C. Pfeiffer, N. Hollenstein, C. Zhang, and N. Langer (2020) Neural dynamics of sentiment processing during naturalistic sentence reading. NeuroImage 218, pp. 116934. External Links: ISSN 1053-8119, Document, Link Cited by: §1, §2, §4.
  • B. J. Roach and D. H. Mathalon (2008) Event-related eeg time-frequency analysis: an overview of measures and an analysis of early gamma band phase locking in schizophrenia. Schizophrenia bulletin 34 (5), pp. 907–926. Cited by: §1.