Coimagination method has been proposed as a novel method to support interactive communication for activating three (3) cognitive functions: episodic memory, division of attention, and planning function, which decline at an early stage of mild cognitive impairment (MCI) (Otake et al., 2009b). It is hypothesized that the activation of these cognitive functions is effective for the prevention of dementia (Otake et al., 2009b; Otake, 2009; Otake et al., 2011). Also, the generation of social networks among individuals helps in protecting individuals against dementia (Otake et al., 2009b). In the past, there have been numerous experiments with the coimagination method in care institutions in Japan, from the typical coimagination method to modified versions. In these studies, several observers and measurers were needed to facilitate and to capture and record data during the coimagination method, which were difficult at times. To address these challenges, we propose an initial design and development of an automated coimagination support system. We aim to have an automated coimagination support system that can be used easily by either healthy participants or elderly participants by voice. Specifically, in this paper, we want to test our initial proposed key features.
The automated support system can accept participants’ registration.
The automated support system can search images for the participants using Google Custom Search API. The image will be used as a visual aid during the participant’s speaking round.
The automated support system can record audio during the session. This can be used to analyze the session afterward.
The automated support system can be used with a voice interface.
In this paper, we want to conduct a preliminary performance study measuring the proposed key features. We will test first with healthy participants to fine-tune our automated system before we conduct a preliminary performance study with elder participants, who will be the actual end-users of our automated coimagination support system.
2. Coimagination Method
2.1. Key Principles
This section describes the principles followed by the coimagination method, procedures, and variations done in previous studies.
Coimagination method should contribute to the generation of social networks among subjects through communication (Otake et al., 2009b).
Coimagination method should evoke feelings and thoughts, not just memories (Otake, 2009).
Coimagination method should give equal time for each participant to talk about their topic, to ask and answer questions from other participants, and to give comments and reactions (Otake, 2009).
Coimagination method should have measures for effectiveness (Otake et al., 2009b).
2.2. Typical Coimagination Method Procedures
Coimagination program consists of five (5) sessions. Each session is held for about one (1) hour per week.
Each session has a unique, predetermined theme. Themes can include but not limited to “favorite food”, “favorite things”, “health and food”.
The average number of participants is six (6).
Each participant brings three (3) images, on the average.
A session has two (2) rounds. The first round is for brief speech, where each participant talks about their topics using the images. The second round is for questions and answers from other participants to each other.
The fifth session is for memory tasks. Images presented throughout the four (4) sessions are displayed randomly to each participant. The participants guess the owner of the image and the theme of the image.
2.3. Previous Implementations of Coimagination Method
Coimagination method has been conducted in the past with different configurations and different analysis. There were programs held with older adults with cognitive decline, compared with a typical program held with healthy older adults (Tsukawaki et al., 2011). In this scenario, the number of participants changed from six (6) to four (4) (Tsukawaki et al., 2011). Time for each round and the number of images had been adjusted based on the participants’ cognitive function, physical fitness, and concentration (Tsukawaki et al., 2011). Furthermore, the question-and-answer round was held immediately after the brief speech round so as not to forget (Tsukawaki et al., 2011). There were also programs held in multiple facilities from different Japan prefectures (Otake et al., 2012). A personal coimagination support system for a single participant was also explored to address elders, not in care home facilities and lack social interaction among other elders (Otaki and Otake, 2017). Most importantly, different measures were being analyzed and evaluated, in addition to analyzing the results of the memory tasks. For instance, the number of comments which evoked reactions, such as laughter and wonder, were counted (Otake, 2009). The intensity of cognitive activities was also measured using social network analysis (Otake et al., 2011). Conversation sessions were transcribed to analyze different aspects such as common topics through grouping to examine knowledge creation (Otake et al., 2009a) and duplication of certain words and phrases by each participant to reflect the quality of interaction (Otake et al., 2013b). Different audio features were also analyzed such as overlaps during conversation (Otake et al., 2013a; Otake and Yamaguchi, 2013) and detection of laughter (Nergui and Otake, 2013; Nergui et al., 2013) to assist in group conversations.
2.4. Motivation for the Proposed Automated Support System
Given the context in the previous subsection, the coimagination method can be configured and analyzed differently, yet we want to make sure that the principles of the coimagination method are kept intact. We want to provide an automated tool for facilitators, observers, and measurers to easily conduct the coimagination method and analyze the different facets of the method. We intend that the automated support system will help facilitators connect more with the participants rather than handling administrative tasks in conducting the coimagination method. Moreover, we want the automated support system to be deployed in nursing care facilities and can be easily used by elders. In this way, we keep them engaged among themselves using the framework set by the coimagination method.
3. Automated Coimagination Support System
3.1. System Architecture
The initial automated coimagination support system is mainly built using Web technologies. Ruby on Rails (RoR) is used as the primary web framework, with PostgreSQL for the database. Audio recordings were stored in the server’s file system and not in the database. For speech processing, Web Speech (Application Programming Interface) API and Web Audio API were used to recognize speech commands and to record audio, respectively. Google Custom Search API is used to search images for participants, given specific keywords. The web application is deployed to Heroku, a cloud platform-as-a-service (PaaS). During the coimagination session, any laptop connected to the Internet and with the latest Google Chrome browser can be used to run the automated system. An external monitor is used for a bigger display. We also used an 8-channel microphone array for the audio input, but an internal laptop microphone can also be used. Figure 1 shows an overview of the system architecture.
3.2. User Interface
In this section, the automated coimagination support system is presented with user interface screens. The automated system can be operated by voice commands via Web Speech API and microphone, or by clicking using the mouse. The automated coimagination support system is used as follows.
The initial home screen presents two options to register their names or start the session to the participants as seen in Figure 2.
Participants are asked for their names, as shown in Figure 3. The system will ask confirmation if it recognized the name correctly. The participant can say “save” or “yes” or click “Save” to confirm. In the Japanese version, “hai” is also accepted. After every participant has registered their names, saying or clicking “back” returns the screen to the home screen (Figure 2), where the session can be started by saying or clicking “session”.
In Figure 4, an active session can be selected by saying the session’s theme, for example, “favorite food”.
Similar to step 2, each participant is asked for their response to the theme of the session. For example, the participant says “fried chicken”. The system uses the keyword “fried chicken“ as an input to Google Custom Search API. The system automatically searches images for “fried chicken“ as if one searches using Google Search Engine. The system displays the top image result, as shown in Figure 5.
To compensate for the lack of preparation time, each participant is given an arbitrary period of five (5) minutes to think about their topic. When the participant feels ready, the button can be pressed to proceed to the speaking round. This is seen in Figure 6.
Each participant is given one (1) minute thirty (30) seconds to speak. The enlarged image is shown on the left side; while, the timer, name of the participant speaking, the session’s theme, and the participant’s topic are shown on the right side, as shown in Figure 7. This step repeats until the last participant is finished speaking, following the original coimagination method (Otake, 2009).
Similar to step 5, participants are given a preparation period before the question-and-answer round. The only difference in this step is that no time limit is provided and the button has to be pressed to proceed.
In this configuration, the question-and-answer round is given one (1) minute and thirty (30) seconds for each participant. The same screen setup is used in step 6.
After the last participant has finished the question-and-answer round, the system shows a thank-you message.
3.3. System Configuration
The automated coimagination support system can accept certain parameters to configure its user interface. The language can switch between English and Japanese. In effect, the labels and accepted spoken commands are adjusted based on the selected language. For example, the keyword “hai” is accepted in exchange for the keyword “yes” or keyword “tōroku” is accepted in exchange for the keyword “register”. The themes for each session can also be added. The themes, as well as participants, can also be activated or deactivated to show or hide in the system during a session, respectively. The period for preparation, speaking round and question-and-answer can be changed easily using the system.
4.1. Experiments Setup
A trial run (or alpha test) with healthy University students and a preliminary performance study were conducted to test the developed system. The trial run was conducted with four (4) healthy male University students, with an average age of 24. The speaking round and the question-and-answer round were given five (5) minutes. Preparation time for both rounds was not set because this feature was only added after the trial run. The addition of preparation time feature shall be explained in the next section. The trail run is meant to fine-tune the automated support system before the actual preliminary performance experiment with actual end-users.
The preliminary performance experiment (or beta test) was conducted at an elderly nursing care facility in Kitakyushu City, Fukuoka, Japan. There were four (4) elderly participants (one male, three female) with an average age of 84. All four elders have dementia but with varying severity. Speaking round and question-and-answer round were reduced to one (1) minute and thirty (30) seconds, with a preparation time of at most five (5) minutes before speaking round. The theme for both experiments is “favorite food.” Table 1 summarizes the trial run and the preliminary experiment, while Figures 8 and 9 show images with healthy participants and elderly participants, respectively.
|Trial Run||Preliminary Experiment|
|Attribute||Healthy University students||Elders with varying severity of dementia|
|Average Age (in years old)||24||84|
|Gender Composition||4 male||1 male, 3 female|
|Preparation Time before Speaking Round||Not applicable||5 minutes|
|Speaking round period||5 minutes||1 minute 30 seconds|
|Question-&-Answer round period||5 minutes||1 minute 30 seconds|
4.2. Experiments Measure
This paper focuses on testing the performance of the proposed automated coimagination support system. Specifically, we want to measure the following proposed key features:
The automated support system can accept participants’ registration.
The automated support system can search images for the participants using Google Custom Search API.
The automated support system can record audio during the session.
The automated support system can be used with a voice interface.
Table 2 shows the key features with the participants’ input and expected output. It also shows the total number of attempts and the success rate. The total number of attempts is counted from video recordings during the preliminary experiment. Some participants only took one (1) attempt at a particular feature, while some participants took more than one (1) attempt. The success rate is computed based on the total successful attempts overall attempts.
|Key Features||Participant Input||Expected Output||Total number of attempts||Success Rate (%)|
|Participant Registration||Participant’s name||System should record the name of the participants||4||100.00%|
|Image Search||Participant’s response to the session’s theme||System should show the correct image.||4||100.00%|
|Audio Recording: Speaking Round||Participant’s voice during his/her discussion||System should record the participant’s voice in an audio file during this round.||4||100.00%|
|Audio Recording: Question-and-Answer Round||Participants’ voices during their discussion||System should record the participants’ voices in an audio file during this round.||4||100.00%|
|Voice Interface: Participant Registration||Spoken participant’s name||System should record the name of the participants||8||0.00%|
|Voice Interface: Image Search||Spoken participant’s response to the session’s theme||System should show the correct image.||9||67.50%|
The primary purpose of the trial run with University students is to fine-tune the automated support system before the actual preliminary experiment with actual end-users, i.e., elders. The automated system has been used easily since the participants are University students who are at least familiar with voice-based applications. However, we noticed that five (5) minutes for both speaking and question-and-answer round was too long. Participants were neither talking for a certain amount of time nor talking about the theme of the session. Due to these observations, we decided to reduce the time for the second experiment with elders. In previous coimagination experiments, time can be adjusted based on the composition of the participants (Tsukawaki et al., 2011). Nonetheless, the system has successfully fulfilled its goals – registering and recording the coimagination session, supporting image preparation using Google Custom Search API, recording audio for both rounds, and navigating the system using the voice interface.
During the preliminary experiment, there were more observations to be considered. First, the elders responded to the system in complete and polite sentences, denoted by the use of “I am …”, and “desu” (which in the Japanese language denotes certain politeness). This is the reason why the system could not correctly register the participant’s name by voice interface. After failing two (2) times, we decided to type their names instead. The input field has been added as a fail-safe feature.
The same happened when asked for their favorite food. They responded with a sentence pattern “I like …”. Two (2) out of the four (4) participants’ voice-based response was properly recognized. Thus, image searching by voice has been accomplished. However, for the other two (2) participants, we assisted them in their response by voice instead of typing their responses.
Another possible reason for the low success rate of the voice interface happened when the system could also hear other participants talking. The system could not determine the end of the sentence. Thus, the recognized input is a mixture of the recognized voice from two or more participants.
In the aspect of correctly providing an image based on certain input (i.e.,“What is your favorite food?”), the system successfully returned the correct image during the preliminary experiment. It happened during the development phase where we tested a few “favorite food”, but the system did not return a correct intended image. For example, “naruto” is Japanese fish cake. But because of the popularity of the fictional character “Naruto”, the system returned an incorrect image. Fortunately, during the preliminary experiment, the participants did not respond like this.
Regarding the changes between the trial run and the preliminary experiment, the elders still showed a period of silence during speaking round and question-and-answer round, despite the decrease in time. We, together with the nursing care facility staff, had to ask questions or make short comments so that participants continue to talk. Participants who are listeners during the speaking round were sometimes making comments. In general, the system seemed to engage the participants to communicate with each other.
5.1. Comparison Between the Typical Coimagination Method and the Automated Coimagination Support System
The automated coimagination support system was developed to closely follow the typical coimagination method, as much as possible. Changes have been made to address the previously mentioned challenges.
In a typical coimagination method, the participants are informed about the theme of the session. Thus, they can find and bring their own pictures to be presented during the session. In the automated method, the participants inform the system of their response to the theme. Then, the system finds a picture from the Internet using Google Custom Search API.
In relation to the previous one, the automated method has a preparation period before the speaking round of each participant. The preparation preiod was added as a cautionary feature for the participants. They can collect their thoughts about the topic, remember the feelings and events related to their topic. The participant can proceed to the speaking round if they feel they are ready.
In a typical coimagination method, a human facilitator operates a laptop with a large screen. Before the session, the facilitator scans the images into the laptop and arranges them according to seat order. During the session, the facilitator chooses the images based on whose turn it is, enlarges a specific image based on the participant’s request, and can ask short questions to assist the participant in their topic. In other studies, additional observers were needed to measure certain things e.g., the number of comments or reactions during the session. In the automated method, we aim that the participants can operate the system by themselves. If the system fails to function properly, as it happened during the preliminary experiments, manual input via keyboard can be done by their supervisor. Moreover, the automated system assists the supervisor in data collection. In this way, listening and responding to the participants is the main purpose of the supervisor.
In this paper, we presented the initial design and development of an automated coimagination support system that can register and record coimagination sessions, search for images for the participants using Google Custom Search API, and record audio during the session and can be used with a voice interface. The automated system can be configured easily in terms of language support, the period for each round and phase, and the addition, activation, and deactivation of sessions and users. We first tested the developed system with University students as a trial run. Then, we tested the re-configured system with actual end-users during the preliminary experiment. In this performance study, we observed and analyzed how well the system performed based on the preliminary experiment’s video recording. We have successfully demonstrated three (3) out of the four (4) key features that we propose. Our future versions can include noise cancellation and better speech separation for improved voice recognition due to the voice interface challenge. Additionally, natural language processing (NLP) tools such as intent classification can be added to handle the complete sentences provided by the participants. We also plan to add multiple images per participant in a session and taking a picture of one’s own images to handle the issue on the correct image.
The authors would like to thank the Ministry of Education, Culture, Sports, Science and Technology Japan (MEXT) for the scholarship grant and Sawayaka Club of Kitakyushu for allowing us to conduct our experiment. This work was supported by JSPS KAKENHI Grant Number JP16H06534.
- Nergui and Otake (2013) Myagmarbayar Nergui and Mihoko Otake. 2013. Laughter detection for an assisting tool of group conversation. methods 8, 9 (2013), 10.
- Nergui et al. (2013) Myagmarbayar Nergui, Lin Pai-hui, Gota Nagamatsu, Mikio Waki, and Mihoko Otake. 2013. Automatic detection of laughter using respiratory sensor data with smile degree. In Proceedings of the International Conference on Advances in Computing, Communications and Informatics, Vol. 1. 8–11.
- Otake (2009) Mihoko Otake. 2009. Coimagination method: sharing imagination with images and time limit. In Proceedings of the International Reminiscence and Life Review Conference, Vol. 2009. 97–103.
et al. (2009a)
Mihoko Otake, Motoichiro
Kato, Shuichi Iwata, Toshihisa Takagi,
and Hajime Asama. 2009a.
Knowledge Creative Activity Support towards
Prevention of Dementia via Coimagination Method. In
The 23rd Annual Conference of the Japanese Society
for Artificial Intelligence
. The Japanese Society for Artificial Intelligence, 2F1NFC41–2F1NFC41.
- Otake et al. (2009b) Mihoko Otake, Motoichiro Kato, Toshihisa Takagi, and Hajime Asama. 2009b. Coimagination method: Communication support system with collected images and its evaluation via memory task. In International Conference on Universal Access in Human-Computer Interaction. Springer, 403–411.
- Otake et al. (2011) Mihoko Otake, Motoichiro Kato, Toshihisa Takagi, and Hajime Asama. 2011. The coimagination method and its evaluation via the conversation interactivity measuring method. In Early Detection and Rehabilitation Technologies for Dementia: Neuroscience and Biomedical Applications. IGI Global, 356–364.
- Otake et al. (2012) Mihoko Otake, Yoko Koyanagi, Yukie Doi, Mitsuhiro Tsujihata, Takayo Tasaki, Muneaki Noguchi, Akira Abe, and Eiko Nagata. 2012. Development of Coimagination Method Support System named “Fonobono Panel” Utilized at Multiple Facilities. In The 26th Annual Conference of the Japanese Society for Artificial Intelligence. The Japanese Society for Artificial Intelligence, 2A1NFC613–2A1NFC613.
- Otake et al. (2013a) Mihoko Otake, Myagmarbayar Nergui, Seong-eun Moon, Kentaro Takagi, Tsutomu Kamashima, and Kazuhiro Nakadai. 2013a. Development of a sound source localization system for assisting group conversation. In International Conference on Intelligent Robotics and Applications. Springer, 532–539.
- Otake et al. (2013b) Mihoko Otake, Myagmarbayar Nergui, Takashi Otani, and Jun Ota. 2013b. Duplication analysis of conversation and its application to cognitive training of older adults in care facilities. Journal of Medical Imaging and Health Informatics 3, 4 (2013), 615–621.
- Otake and Yamaguchi (2013) Mihoko Otake and Kenta Yamaguchi. 2013. Analysis of Overlap during Group Conversation of Active Older Adults. In The 27th Annual Conference of the Japanese Society for Artificial Intelligence. The Japanese Society for Artificial Intelligence, 3C1IOS1b5–3C1IOS1b5.
- Otaki and Otake (2017) Hikaru Otaki and Mihoko Otake. 2017. Interactive Robotic System Assisting Image Based Dialogue for the Purpose of Cognitive Training of Older Adults. In 2017 AAAI Spring Symposium Series.
- Tsukawaki et al. (2011) Akio Tsukawaki, Yoshiyasu Tadenuma, Yukiko Sato, Katsutoshi Negishi, Yoshie Taguchi, Akiko Maekawa, Sumiko Nagai, Hideko Takeshita, Seiji Kuroda, Yoshiki Myojin, Makato Onitake, Yoshinori Hasegawa, and Mihoko Otake. 2011. Development of Support Service for Care Welfare Facilities based on Coimagination Method towards Prevention and Recovery from Dementia. In The 25th Annual Conference of the Japanese Society for Artificial Intelligence. The Japanese Society for Artificial Intelligence, 1A1NFC1a1–1A1NFC1a1.