A Novel Framework for Visual Motion Imagery Classification Using 3D Virtual BCI Platform

02/04/2020 ∙ by Byoung-Hee Kwon, et al. ∙ Korea University 0

In this study, 3D brain-computer interface (BCI) training platforms were used to stimulate the subjects for visual motion imagery and visual perception. We measured the activation brain region and alpha-band power activity when the subjects perceived and imagined the stimuli. Based on this, 4-class were classified in visual stimuli session and visual motion imagery session respectively. The results showed that the occipital region is involved in visual perception and visual motion imagery, and alpha-band power is increased in visual motion imagery session and decreased in visual motion stimuli session. Compared with the performance of visual motion imagery and motor imagery, visual motion imagery has higher performance than motor imagery. The binary class was classified using one versus rest approach as well as analysis of brain activation to prove that visual-related brain wave signals are meaningful, and the results were significant.



There are no comments yet.


page 2

page 3

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Brain-computer interface (BCI) is a technology that enables direct communication between the user’s brain and a computer and control of devices to reflect the user’s intentions [14]. Electroencephalography (EEG) equipment is widely used in BCI systems because it is non-invasive, and has high time resolution [23].

The BCI system controls the device according to the user’s intention [19], [10] providing appropriate motor function for paralyzed patients. If the purpose of BCI is to control the device with user’s intention [8], steady state visually evoked potential (SSVEP) [4], [17], evoked related potentials (ERPs) [18], and motor imagery (MI) [11], [21], [13] have used traditionally. However, these paradigms are difficult for users to intuitively accept. ERPs and SSVEP have an obvious limitation that brain signals should be generated through external stimuli and collected to identify user intentions. To compensate for these limitations, previous studies collected and analyzed brain signals that reflect user intentions using MI [3]

. MI is an endogenous-based paradigm and users imagine muscle movement. Accordingly, the area near sensory motor of brain that is related to movement is activated and this area is analyzed in various methods. In fact, many users imagined MI tasks in wrong way, and even if they are familiar with a proper way to imagine MI tasks, it is difficult to imagine muscle movement. MI also requires a long training time to be used in real online environments. This causes fatigue to users and hinders them in accepting high-quality brain signals. While the degree of freedom is very limited when controlling a device using MI, this study suggests the possibility of controlling a variety of high-level movements.

Visual imagery allows users to control the device using internal imagination and allows for more intuitive imaginations than MI. Visual motion imagery can be classified based on analyzing the alpha wave areas. The intensity of alpha-band power varies when users are aware of visual stimuli and performing visual imaginations, which allows them to classify the two states [20]. The potential of visual motion imagery as a control strategy in comparison to motor imagery was explored by Neuper et al[2]. In a recent study, Koizumi et al. clarified differences in brain activity between imagery and perception [12]. Through this study, we have confirmed that perception and imagery are clearly distinct.

The purpose of this study is to compare the brain activation differences and activation regions between visual motion imagery and visual perception through EEG, and to classify each class in visual motion imagery. Many research groups have been investigated to control the arm prosthesis or the robotic arm based on BCI system. Meng et al. [7] have investigated the feasibility of control a robotic arm in three-dimensional space using sensorimotor rhythm. However, the stimuli presented to users in previous studies were not three-dimensional stimuli and the imagery task was not intuitive.

Fig. 1: 3D virtual BCI platform-based stimuli given to users in a perception session. Each stimulus is given to the user for five seconds and the movement is designed to have different meanings. The four above stimuli were shown only in the perception session, so that no aftereffect was left on the stimuli.

Fig. 2: The paradigm of visual perception and visual motion imagery experiment. The user first conducted a session on the stimulus (a) and then a session on the imagery (b). An auditory stimulus of less than 0.5 seconds was given at the beginning of the rest phase and visual stimuli phase. Auditory stimuli were given corresponding auditory stimuli for each class.

Therefore, this study presented visual stimuli in three-dimensional space that users can intuitively accept and visual motion imagery as an intuitive imagination. We have hypothesized that provide more realistic stimuli to users results in better imaginations and can classify them. If the brain signals produced by visual imagery can be classified [16] as we have established a hypothesis, then users can control BCI-based robots such as robotic arms.

Ii Methods

Ii-a Data Acquisition

Five subjects between the ages of 25 and 30 participated in the experiment. The subjects who participated in the experiments were naïve BCI users. All participants were right-handed, and had normal vision and reported no medical or psychological disorders. We used BrainAmp amplifier and BrainVision Recorder for EEG data acquisition. EEG data were collected using 64 Ag/AgCl electrodes following 10-20 international system. The data were collected at 1,000 Hz only with 60 Hz notch filtering. The impedance of all electrodes were maintained below 10k during the acquisitions. The protocols and environments were reviewed and approved by the Institutional Review Board at Korea University [1040548-KU-IRB-17-172-A-2].

Ii-B Experimental paradigm

The visual stimulation paradigm was used as a guide to the visual motion imagery tasks. The video for visual stimulation was created using Unity3d and Blender (Blender 3D Engine: www.blender.org). Each video for stimulation consisted of eating food, pouring water, picking up a phone, and opening doors as illustrated in Fig. 1. The data of each session was controlled by MATLAB (MathWorks) using the Psychtoolbox (Psychtoolbox-3 distribution; http://www.psychtoolbox.org) [15]. The experiment consists of two sessions, a visual perception session, and a visual motion imagery session that take place on the same day (Fig. 2). In the visual stimuli session, the subject was stimulated for 5 seconds using each stimulation video. In an imagery session proceeded after a stimulation session, only a black screen is presented to the subject to minimize the side effects of visual stimulation. In visual motion imagery session, all subjects imagined each class according to the auditory cue of each class. In each session, the number of trials per class is 50. The subjects were instructed to minimize eye flickering and body movement during each session. The experiment was conducted in a dark environment so that the subjects could concentrate on perception and imagination, and the distance between the monitor and the subject’s face was maintained at about 60 cm.

Ii-C Data Analysis

The data analysis was performed offline using BBCI toolbox and the EEGLAB toolbox (version 14.1.2b). To performance evaluation of visual motion stimuli and visual motion imagery, we used filtered EEG signals. Raw EEG signals filtered from 8 to 13 Hz with band-pass filter. The features of EEG signals should be extracted with common spatial pattern (CSP) [9]. Alpha band power was used in previous studies on visual motion imagery classification [5]. Thus, alpha band energy was extracted and filtered by CSP. To classify the visual motion stimuli and the visual motion imagery, we used the regularized linear discriminant (RLDA) as a classifier [6].

EEG signals were band-pass filtered in [8-13] Hz, using a third order Butterworth filter for analyzing alpha-band power activity based on spatial information. Using BBCI toolbox, topographies of pre-processed data were visualized. We analyzed the spatial information of the visual stimuli data and visual motion imagery data in four time zones: (01,000) ms, (1,0002,000) ms, (2,0003,000) ms, and (3,0004,000) ms.

In order to understand the variance of alpha-band power, we performed channel time-frequency of the imagery and stimulation data. Variation of spectral power according to the event-related changes at each time during the trials, and at each frequency, was analyzed using the event-related spectral perturbation (ERSP) method

[1]. ERSP analyses were performed for frequencies ranging from 3 to 50 Hz for all channels, using 200 time points. The baseline for calculating ERSP taken from the last 500 ms of the rest phase before imagery or stimuli phase.

Fig. 3: The activation of alpha wave in visual motion imagery and visual perception. (a) refers to the alpha wave activation in the visual imagery session and is activated near the occipital lobe over time. The following figure (b) shows the alpha wave activation in the visual perception session, and the power is decreased in the occipital lobe.

Fig. 4: The event-related spatial perturbation of visual motion imagery. It shows that the power of alpha waves increases over time.

To control BCI applications successfully, such as robotic arm control, we have attempted to classify [22] four imagery classes using a reduced number of channels and features. For analysis, we removed the effects of auditory stimuli before using the data. Since the auditory cue was given to the subjects during the visual motion imagery session, the initial 0.5 s was excluded after the auditory cue and a 0.5 to 4 s segment of the data was used for the analysis.

Iii Results and discussions

Iii-a Neurophysiology Analysis

Fig. 3 showed the visualization of the alpha-band power activity of each channel as spatial information. For the occipital channels’ cluster, we found a significant main effect for the alpha-band power during visual motion imagery and visual perception session. The alpha-band power increase significantly during visual motion imagery session and decreased during visual perception session. When divided into 1,000ms intervals, the alpha-band power increased more clearly over time in the imagery session, and in the case of a perception session, the alpha-power band decreased regardless of time. This means that a characteristic brain signal was showed near the occipital region after a certain period of time after the onset of imagination and, in the case of perception, a characteristic brain signal was initiated directly from the occipital lobe because no preparatory step is required. Fig. 4 showed the ERSP of alpha-band power activity in visual motion imagery session using channel Oz. The graph on the left side of Fig. 4 showed the power distribution over the entire frequency range of the trial averaged data. The power increased in the alpha band, which means that characteristic brain signals evoked at frequencies related to visual imagery [20], [12]. This result is consistent with the result that alpha band power increases in the occipital lobe over time when visual related imagery derived through Fig. 3.

Iii-B Performance Evaluation

Table i@ showed a comparison of the classification accuracies of visual related brain signals and motor related brain signals. The performance of visual motion imagery was 33.03%, and the result was over 25% chance level in four classes. The reason for the low visual motion imagery performance of sub4 is that user does not receive proper instruction, not a BCI illiteracy problem. This is because the performance for the visual perception of sub4 is much higher than that of other users, which means that it is possible to classify through visual-related brain waves. The results showed better performance than the motor imagery, which is commonly used in BCI-based control technologies such as robotic arms. We suggest the possibility of being used as an alternative application for BCI-based device control. Indeed, in a recent study, K. Koizumi and colleagues controlled drone through visual motion imagery in three-dimensional space and yielded meaningful results [11]. We suggest that through visual motion imagery, which is more intuitive than other paradigms, patients with disabilities can access BCI-based device control more easily.

Table ii@ showed the performance comparisons to prove that subjects are imagining correctly in the visual imagery phase through one versus rest approach. Binary classification performances based on each class are 79.62%, 81.36%, 80.16%, and 81.42%, and it is an acceptable result. In one versus rest approach, there is little performance difference between classes, which means that the behavior or shape expressed in the class does not significantly affect classification. This can be inferred that an increase in the number of classes does not cause confusion among classes due to the confusion of visual motion. But as Table i@ shows, performance decreases as the number of classes increases. This is a critical problem when classifying multiple classes as machine learning in BCI domain. We expect to solve this problem using deep learning methods, and if the prediction is correct, we can increase the number of classes. This means that paralyzed patients can control BCI-based devices to suit their intentions.

Task Sub1 Sub2 Sub3 Sub4 Sub5 Average
TABLE I: The performance comparison of the executed session
Task Sub1 Sub2 Sub3 Sub4 Sub5 Average
Picking up
a phone
TABLE II: The result of one versus rest approach

Iv Conclusion and Future Work

In this study, brain signals related to visual perception and visual motion imagery were analyzed by brain region and frequency. And we classified visual related imagery using 3D BCI training platform. As a result, performance exceeded the chance level for all classes. In addition, visual motion imagery brings out a better performance than motor imagery and proves that a solid visual imagination is possible through a one versus rest approach. However, for application, the performance of visual motion imagery (33.03% based on four classes in this study) is still low. We can present the possibility that there is a deep learning architecture suitable for classification rather than machine learning for EEG classification, and it can improve future performance.

In Fig. 4, we can see a similar aspect to event-related desynchronization (ERD) and event-related synchronization (ERS) in motor imagery. We are going to analyze neurophysiological analysis of what this means in visual imagery.

After questioning the inconvenience to the subjects about the experiment, the paradigm proposed in this study had the problem that each class’s memory becomes blurred over time in the visual motion imagery session. Each session has 50 trials per class, and subjects should perform a total of 200 trials in one session. Thus, in the visual motion imagery session where no visual stimulus is given, the memory of the stimuli seen in the previous visual perception session is dimmed. Also, the paradigm divided into both different sessions can give fatigue and pressure to subjects, so we should develop a more concise and effective paradigm consisting of sessions made by stimuli and imagery. A comfortable environment can be created, which can improve the performance by acquiring good brain signals.

V Acknowledgment

The authors thanks to J.-H. Cho, B.-H. Lee, and D.-Y. Lee for setting experiment environment and K.-H Shim for developing the experimental codes.


  • [1] A. Delorme and S. Makeig (2004)

    EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis

    J. Neurosci. Methods 134, pp. 9–21. Cited by: §II-C.
  • [2] C. Neuper, R. Scherer, M. Reiner, and G. Pfurtscheller (2005) Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Cogn. Brain Res. 25, pp. 668–677. Cited by: §I.
  • [3] D.-O. Won, H.-J. Hwang, D.-M. Kim, K.-R. Müller, and S.-W. Lee (2017) Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 26, pp. 334–343. Cited by: §I.
  • [4] D.-O. Won, H.-J. Hwang, S. Dähne, K.-R. Müller, and S.-W. Lee (2015) Effect of higher frequency on the classification of steady-state visual evoked potentials. J. Neural Eng. 13, pp. 016014. Cited by: §I.
  • [5] H.-I. Suk and S.-W. Lee (2011) Subject and class specific frequency bands selection for multiclass motor imagery classification. Int. J. Imag. Syst. Tech. 21, pp. 123–130. Cited by: §II-C.
  • [6] I. D. Rus, P. Marc, M. Dinsoreanu, R. Potolea, and R. C. Muresan (2017) Classification of EEG signals in an object recognition task. In Proc. IEEE. Intell. Comput. Commun. Process (ICCP), pp. 391–395. Cited by: §II-C.
  • [7] J. Meng, S. Zhang, A. Bekyo, J. Olsoe, B. Baxter, and B. He (2016) Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci. Rep. 6, pp. 1–15. Cited by: §I.
  • [8] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan (2002) Brain-computer interfaces for communication and control. Clin. Neurophysio. 113, pp. 767–791. Cited by: §I.
  • [9] J.-H. Cho, J.-H. Jeong, K.-H. Shim, D.-J. Kim, and S.-W. Lee (2018) Classification of hand motions within EEG signals for non-invasive BCI-based robot hand control. In Conf. Proc. IEEE Int. Syst. Man Cybern. (SMC), pp. 515–518. Cited by: §II-C.
  • [10] J.-H. Jeong, K.-H. Shim, J.-H. Cho, and S.-W. Lee (2019) Trajectory decoding of arm reaching movement imageries for brain–controlled robot arm system. In Int. Conf. Proc. IEEE Eng. Med. Biol. Soc. (EMBC), pp. 23–27. Cited by: §I.
  • [11] J.-H. Kim, F. Bießmann, and S.-W. Lee (2014) Decoding three-dimensional trajectory of executed and imagined arm movements from electroencephalogram signals. IEEE Trans. Neural Syst. Rehabil. Eng. 23, pp. 867–876. Cited by: §I, §III-B.
  • [12] K. Koizumi, K. Ueda, and M. Nakao (2018) Development of a cognitive brain-machine interface based on a visual imagery method. In Conf. Proc. IEEE Eng. Med. Biol. Soc. (EMBC), pp. 1062–1065. Cited by: §I, §III-A.
  • [13] K.-T. Kim, H.-I. Suk, and S.-W. Lee (2018) Commanding a brain-controlled wheelchair using steady-state somatosensory evoked potentials. IEEE Trans. Neural Syst. Rehabil. Eng. 26, pp. 654–665. Cited by: §I.
  • [14] L. F. Nicolas-Alonso and J. Gomez-Gil (2012) Brain computer interfaces, a review. Sensors 12, pp. 1211–1279. Cited by: §I.
  • [15] M. Kim, G. Wu, Q. Wang, S.-W. Lee, and D. Shen (2015)

    Improved image registration by sparse patch-based deformation estimation

    Neuroimage 105, pp. 257–268. Cited by: §II-B.
  • [16] M.-H. Lee, S. Fazli, J. Mehnert, and S.-W. Lee (2015) Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recognit. 48, pp. 2725–2737. Cited by: §I.
  • [17] N.-S. Kwak, K. R. Müller, and S.-W. Lee (2017)

    A convolutional neural network for steady state visual evoked potential classification under ambulatory environment

    PLoS One 12, pp. e0172578. Cited by: §I.
  • [18] S.-K. Yeom, S. Fazli, K. R. Müller, and S.-W. Lee (2014) An efficient ERP-based brain-computer interface using random set presentation and face familiarity. PLoS One 9, pp. e111157. Cited by: §I.
  • [19] T. M. Vaughan, W. Heetderks, L. Trejo, W. Rymer, M. Weinrich, M. Moore, A. Kübler, B. Dobkin, N. Birbaumer, E. Donchin, E. Wolpaw, and J. Wolpaw, and others (2003) Brain-computer interface technology: a review of the Second International Meeting. IEEE Trans. Neural Syst. Rehabil. Eng. 11, pp. 94–109. Cited by: §I.
  • [20] T. Sousa, C. Amaral, J. Andrade, G. Pires, U. J. Nunes, and M. B. Castelo (2017) Pure visual imagery as a potential approach to achieve three classes of control for implementation of BCI in non-motor disorders. J. Neural Eng. 14, pp. 046026. Cited by: §I, §III-A.
  • [21] T.-E. Kam, H.-I. Suk, and S.-W. Lee (2013) Non-homogeneous spatial filter optimization for ElectroEncephaloGram (EEG)-based motor imagery classification. Neurocomputing 108, pp. 58–68. Cited by: §I.
  • [22] X. Zhu,H.-I. Suk, S.-W. Lee, and D. Shen (2016)

    Canonical feature selection for joint regression and multi-class identification in Alzheimer’s disease diagnosis

    Brain imaging and behavior 10, pp. 818–828. Cited by: §II-C.
  • [23] Y. Chen, A. D. Atnafu, I. Schlattner, W. T. Weldtsadik, M.-C. Roh, H.-J. Kim, S.-W. Lee, B. Blankertz, and S. Fazli (2016) A high-security EEG-based login system with RSVP stimuli and dry electrodes. IEEE Trans. Inf. Forensics Secur. 11, pp. 2635–2647. Cited by: §I.