Vision is an important sensory modality for humans. Many activities of daily living (ADLs), such as cooking and eating, can be difficult without visual support. Jones et al. (Jones et al., 2019) revealed that people with visual impairments tend to have poor nutritional status, which is often linked to problems with buying, preparing, and eating healthy food. People with visual impairments may have an aversion to cooking due to difficulty accessing visual information and cues during the cooking process (Bilyk et al., 2009; Kostyra et al., 2017). This has resulted in people with visual impairments more frequently eating outside at restaurants or preparing frozen food that may be calorie-rich. According to the aforementioned Canadian study (Bilyk et al., 2009), eight out of nine participants stated they “disliked or hated cooking” because of the time it takes to cook without vision.
Christine Ha, the first blind contestant of MasterChef, won the third season of the show in 2012 and described the importance of cooking to people with visual impairments (Wikipedia, 2021; Ha, 2020). To assist visually impaired individuals to cook independently, there have also been training guidelines released from blind communities (VisionAware, ) and cooking related assistive technologies that are commercially available (Henkler, 2020). For example, people with visual impairments could use voice commands to set timers or use a speaking kitchen thermometer to check the temperature of a steak. However, little research has explored the practices of people with visual impairments in cooking and how they leverage different assistive devices for cooking. Furthermore, there has been little to guide HCI researchers on what stages or steps in the cooking process may benefit most from support via technology. In our research, we explore the following research questions:
RQ1: What are current cooking approaches and techniques employed by people with visual impairments?
RQ2: What are the key challenges, concerns, and risks encountered while preparing meals?
RQ3: What are potential opportunities for assistive technologies to support people with visual impairments to cook independently?
To first understand the current cooking experiences of people with visual impairments (RQ1), we conducted a content analysis of 122 YouTube videos that feature visually impaired individuals preparing meals. We describe 12 different activities essential to cooking that were summarized from the video analysis. Based on the findings from the video analysis, we then conducted semi-structured interviews with 12 visually impaired people who have experience cooking to better understand RQ2 and RQ3. The interview findings further illuminate challenges encountered before, during, and after cooking including: utilizing tools, information access, touching and feeling, safety and consequence, precision and ambiguity, organizing and tracking, item and quality inspection, and collaborative cooking and communication. We then discuss the potential opportunities to support people with visual impairments while cooking (e.g., zero-touch interactions for cooking, status tracking and safety monitoring, and collaborative cooking).
2. Background and Related Work
2.1. Eating and Cooking for People with Visual Impairments
People with visual impairments’ Activities of Daily Living (e.g., eating and mobility) and Instrumental Activities of Daily Living (e.g., preparing and making food) are affected by the loss of vision (Bhowmick and Hazarika, 2017). Jones et al. (Jones et al., 2019) conducted a survey study with 101 visually impaired people and found 65% of the participants stated that their visual impairments made cooking difficult. Due to the difficulty of cooking, Bilyk et al. (Bilyk et al., 2009) found that people with visual impairments tend to eat outside or prepared food, which affects healthy eating behaviors. To enable efficient preparation of meals, Kostyra et al. (Kostyra et al., 2017) further showed that assistive technologies, such as having equipment with a voice editor, devices informing about the cooking process, and sensors supporting pouring fluids, may enable efficient preparation of meals for people with visual impairments. Therefore, it is important to explore existing cooking practices and challenges for people with visual impairments and understand specific cooking processes or steps that certain assistive technologies may help people with visual impairments in cooking.
2.2. Enabling Technology for People with Visual Impairments
Cooking usually requires people with visual impairments to interact with different interfaces or devices. The traditional way to enable people with visual impairments to interact with electronic appliances is by adding tactile markers to them. Beyond this traditional method, prior research also explored using computer vision(Guo et al., 2016; Fusco et al., 2014; Morris et al., 2006; Tekin et al., 2011), voice interactions (Abdolrahmani et al., 2018; Branham and Mukkath Roy, 2019), and 3D printed tactile marking (Guo et al., 2017; He et al., 2017) to better support people with visual impairments interacting with different interfaces. For example, VizLens leveraged computer vision and crowdsourcing to enable people with visual impairments to interact with different interfaces, such as a microwave oven (Guo et al., 2016). Guo et al. (Guo et al., 2017) further introduced a crowdsourced fabrication pipeline to help blind people independently make physical interfaces accessible through adding a 3D printed augmentation of tactile buttons overlaying the original panel.
Beyond making appliance interfaces accessible, prior research has also explored various approaches to improve the accessibility of mobile devices for people with visual impairments which might help with the cooking process, such as gestural interactions (e.g., (Kane et al., 2008; Azenkot et al., 2012; Li et al., 2017)) and screen readers (e.g., (Rodrigues et al., 2015; Inc., 2021; Leporini et al., 2012; LLC, 2021)). For example, Talkback (LLC, 2021), and VoiceOver (Inc., 2021) enable people with visual impairments to explore interface elements on mobile devices through audio feedback. The feasibility of using mobile devices further allows people with visual impairments to interact with other IoT devices (Zhou et al., 2017; Saquib et al., 2017). However, it is unknown how people with visual impairments tend to interact with mobile devices during cooking or utilize their mobile devices to interact with different kitchen appliances, and associated challenges and barriers.
2.3. Technology for Cooking
In terms of cooking processes, there has been prior research that explored learning procedures of cooking (Kato and Hasegawa, 2013) and different cooking techniques (Kusu et al., 2017; Hamada et al., 2005). For example, Kato and Hasegawa (Kato and Hasegawa, 2013) introduced an interactive sauteed cooking simulator that could visualize different cooking states (e.g., temperature changes, browning from burns). This system could help users to better manage their cooking skills, such as how to cook medium-rare meat (Kato and Hasegawa, 2013). Kusu et al. (Kusu et al., 2017) further proposed a method to calculate a cooking recipe’s difficulty level during searching and recommend recipes that match the user’s cooking skills. Although prior research has explored how to help people with cooking activities, there lacks research and understanding of what cooking-related learning procedures people with visual impairments have adopted and what the existing challenges are during these learning processes. In our work, we showed cooking practices of 12 different cooking procedures through a YouTube video analysis and uncovered eight themes of cooking challenges through interviews with people with visual impairments.
3. YouTube Video Analysis: Cooking Practices for People with Visual Impairments
To understand existing cooking practices and potential risks for people with visual impairments, we conducted a YouTube video analysis—searching, filtering, and analyzing YouTube videos related to cooking practices by people with visual impairments—inspired by prior research on leveraging the richness of YouTube video contents to understand accessibility needs (Anthony et al., 2013). Our video analysis consisted of two main steps: 1) searching for YouTube videos related to cooking practices for people with visual impairments; 2) analysis and coding procedures.
|Blind Cooking, Blind Person Cooking, Blind Chef, Legally Blind Cooking, Blind Cooking Food, Blind Cooking Dinner, Blind in the Kitchen, Visually Impaired Cooking, Visually Impaired Person Cooking, Visually Impaired Chef, Visual Impairment Cooking|
3.1. Search Protocol
In the video searching process, we looked for videos focused on cooking practices for people with visual impairments. To search for relevant videos, three researchers independently combined visual impairment related keywords (e.g., blind, visually impaired, visual impairment) and cooking related keywords (e.g., cook, cooking, chef, kitchen). To come up with these, our researchers first started with basic searches (e.g., blind cooking) and gradually included other keyword combinations from candidate video titles or descriptions. Because each search may generate hundreds of results, we then followed the same approach as Komkaite et al. (Komkaite et al., 2019) by stopping our search for videos after the whole page of results started to be irrelevant.
In total, we initially created a video dataset of 136 relevant videos found by March 28th, 2021. We then filtered out videos if: 1) the person who cooked in the video did not have visual impairments; 2) the person only heated frozen food; 3) it has poor audio and video quality or did not show the person cooking; 4) videos were duplicated. We then ended up filtering 14 videos and created the final video dataset of 122 videos (V1 - V122). Among the 122 videos in our dataset, most videos were uploaded in 2020 (35), while others were uploaded in 2021 (34), 2019 (11), 2018 (10), 2013 (9), 2014 (7), 2017 (6), 2016 (5), 2015 (3), and 2012 (2). The average length of videos was 698 seconds (ranging from 77 seconds to 2694 seconds).
3.2. Video Content Analysis
To code the videos, three researchers first open-coded (Charmaz, 2006) the videos independently. Then, the coders met and discussed their codes. When there was a conflict, they explained their rationale for their code to each other and discussed to find a resolution. Eventually, they reached a consensus and consolidated the list of codes. Afterward, we performed affinity diagramming (Hartson and Pyla, 2012) to group the codes and identify emerging themes according to the Cooking Guidelines for people with visual impairments (VisionAware, ). Overall, we established 12 themes and 37 codes. The following section describes the findings based on the 12 themes.
|Preparation||General Safe Cooking Practices|
|Cutting and Chopping|
|Cooking||Placing Pans on a Burner|
|Testing Food for Doneness|
|Tools, Environment, and Recognition||Tools and Small Appliances|
|Knowing the Kitchen Space|
|Tell Things Apart|
3.3. Results: Cooking Practices of People with Visual Impairments
From the YouTube video analysis, we uncovered unique cooking practices and procedures of different people with visual impairments (Table 2).
3.3.1. General Safe Cooking Practices
For general safe cooking tips and practices, we found that people with visual impairments usually start everything with gentle actions. This includes using a slow cooker rather than an open grill to avoid fire flare ups (V18), always cooking on low to medium heat to avoid injury (V80), and putting all food and ingredients in a cold pan first before putting the pan on the burner (V4, V6). In V80, the person with visual impairments mentioned in his video:
“…For people with visual impairments like me, we might have to make different adjustments on the pan with our hands or utensils while cooking, I often start with using my hand to feel the temperature and cook at low to medium temperature on my pan…”
Beyond starting with gentle actions, we found that people with visual impairments tend to use hearing and hand-feeling to compensate for vision needs during cooking. In terms of leveraging hands for feeling during cooking, we found that people with visual impairments hold their hands over the pan to gauge temperature (e.g., V11, V23) and use hands to guide food onto utensils (V15). For hearing, we found that people with visual impairments leveraged sound to tell if oil is hot or not (V90, V94). For example, one cook with visual impairments put a small droplet of water into a pan of oil to hear if it was hot (V90). Furthermore, we found that it is important for people with visual impairments to memorize the environment in the kitchen, including the kitchen layout (V1, V17, V46) and which knobs correspond to which burner on the stove (V9).
3.3.2. Cutting and Chopping
For cutting and chopping, we learned that many people with visual impairments tried using alternate ways to avoid knives, such as using a garlic masher, a vegetable chopper (Fig. 1(a)), a food processor (e.g., V103, V114), scissors (e.g., V84, V94), or even by buying pre-minced garlic (V9, V13). Moreover, we found that people with visual impairments sort cut and uncut food into different specific locations on the cutting surface (V1, V32). In V1, the person commented on her practices in the video: “For me, I always place the cut food at the top right corner of my cutting board, so I can keep track of the food that I still need to cut.” We also found that people with visual impairments have different practices of using the knife to cut objects. For example, we found people perform cutting by 1) having fingers on both sides of the knife to position where to cut (e.g., V23, V28) (Fig. 1(d)), 2) putting the whole hand on top of the knife (e.g., V20, V27) (Fig. 1(b)), or 3) tucking the fingertips under and in towards the palm of the hand and cutting with the other hand (e.g., V16, V112) (Fig. 1(c)).
To measure food, we first found that there is a common practice for people with visual impairments to use hands to weigh meat and measure spices (e.g., V12, V35). We further revealed that people with visual impairments highly rely on using measuring cups with hand assistance to measure liquids (Fig. 2). To accurately use measuring cups, people with visual impairments either have to memorize the measurements (V51) or use a braille-labeled set (V10). Finally, visually impaired cooks also prefer using existing containers or utensils to measure food, such as using the same can to get a 1:1 ratio of water and soup (V89) and using the broad side of a knife to measure the size of meat (V96).
For pouring, we found that people with visual impairments use an extra-large bowl to reduce the difficulty of pouring (V53, V56) and usually pour over the sink to make sure spills do not matter (V10, V26). To know how much people are pouring, we revealed that people tend to use their finger over the spout and/or in containers to know the exact amount (e.g., V16, V21). Another common practice we discovered is “making sure you poured everything” to ensure all of the ingredients they want are added to their dish. More specifically, we found that people first add water to food processor or bowl to get any leftover ingredients (V35) and then wipe bowls with fingers to make sure they are empty after pouring (e.g., V30, V59).
3.3.5. Placing Pans on a Burner
From the video analysis, we recognized that people with visual impairments prefer using traditional electric stove top than modern glass top or gas top. The key reason was the ease of placing pans on the stove top. As a practice, we found that people with visual impairments tend to either use listening or feeling to know whether the pan is on the burner. For example, V19 showed that visually impaired individuals listen to sizzle to tell whether the pan is on the burner or not after turning on the fire (V19). As another option, we found people either use hand to feel to make sure the pan is at the center of the stove top (V96) or use the back of a spatula to feel if the pan is on the burner (V19).
In baking, we found that visually impaired people have to use their hands to feel batter texture, check the batter readiness, and shape dough (V41, V88). Specifically, we found that visually impaired cooks prefer to fold the dough over parchment paper (V92), use their hands to shape the dough (V121), and create barriers when rolling it out (V82). Furthermore, we revealed the importance of keeping things clean during baking. For example, V29 mentioned the importance of wiping edges of the pan before baking to prevent the burning of anything that accidentally got on the edges. Finally, we found people with visual impairments use caution with the oven when baking, such as by pulling the oven shelf out slightly to avoid burns (V40).
3.3.7. Turning Food
For turning food, we learned that people with visual impairments have different preferences on choosing tools to accomplish the task. This includes using a large two-prong fork (V55), using two utensils (V83), and using tongs and spatulas for (V33, V78). While turning food, people with visual impairments have different practices of keeping track of food locations. For example, V22 showed that the person always memorizes how much food she put on the grill and roughly where it is. V16 further showed that people with visual impairments use one hand to feel for food locations and then use the other hand to turn the food with a spatula (Fig. 3).
From the video analysis, we uncovered that using hands to spread butter or jam is a common practice for people with visual impairments to make sure everything is evenly distributed (e.g., V43, V63). V88 further showed the practices of using single-use plastic gloves for spreading. Beyond using hands, V45 showed the approach of using utensils to spread jam evenly on bread.
3.3.9. Tools and Small Appliances
In terms of tools and small appliances for cooking, we found that visually impaired individuals often use a virtual assistant to set timers. This includes using Siri (e.g., V9, V13), Alexa (V1), Google (V4), and analog timers (V44). In V44, the visually impaired person rotated the analog timer using her hand position to gauge the correct time. Furthermore, we uncovered that speaking tools are widely used for different purposes, such as using a barcode reader to know what is in a can (V5), Aira (Corp, 2021) for identifying cans, screen readers to read recipes off of a phone (V29), video magnifiers to read boxes (V17), talking scales (V41) and speaking thermometers (e.g., V42, V115). We further found that people with visual impairments tend to use a lot of specific tools for specific tasks. For instance, V36 showed that the cook used an automatic can opener and listened for when it was done. Other examples include using a tea strainer to rinse off rice (V84) and spatula and chopsticks for mixing (V95).
3.3.10. Testing Food for Doneness
To test whether the food is done, we found that people with visual impairments mostly use five different methods. First, they feel to check food doneness, such as feeling food texture with a spoon (e.g., V9, V86), feeling for fried food to float to the top with a utensil (V90), and feeling cooking food with hands (e.g., V16, V39). We also learned that visually impaired people used measures for checking doneness. For example, they initiated a timer (e.g., V19, V54) or used a thermometer to check whether the food is done (V19, V121). Moreover, we found people with visual impairments also used listening to tell when food sounds done (e.g., V11, V34) and use smell to check for food doneness (e.g., V67, V100). For example, V84 mentioned that visually impaired people could use sound to know whether water is boiling. Finally, we found that people who learned cooking before losing their vision judge food doneness on their intuition and prior memories from when they were sighted (V91).
3.3.11. Knowing the Kitchen Space
From the YouTube video analysis, we discovered that people with visual impairments put a lot of effort to organize the kitchen space to reduce cooking barriers, such as having separate space for cooking and for ingredients (V10), keeping the container of utensils handle side down so they can feel the ends (V25), keeping measuring cups nested (V28) and always placing utensils back to their original places (V3). Furthermore, we realized visually impaired individuals have different practices to feel the kitchen space. For example, V22 showed that people use a spatula to feel around a hot grill to get to know the space.
3.3.12. Tell Things Apart
To tell ingredients or tools apart, we found a common practice for people with visual impairments is adding labels on buttons or objects. For instance, visually impaired individuals add Braille labels on spice jars (V27, V47) or bump dot stickers for recognizing buttons (V50). Another approach to tell things apart is by recognizing the size or shape of containers. For example, the person from V72 put each ingredient in a different type of bowls to keep track of different ingredients. Furthermore, people also rely on smell to tell things apart, such as smelling to tell the differences between spices (V35, V65). Finally, we found that some people with visual impairments prefer putting their containers in certain orders. For example, some people prefer to alphabetically organize spices in their drawers (V50).
4. Interview Study: Challenges of Cooking for People with Visual Impairments
In the YouTube video analysis, we uncovered existing practices for people with visual impairments in different cooking procedures. We then leveraged these findings and conducted semi-structured interviews with visually impaired people who have experience cooking to better understand their existing cooking challenges.
|Participant||Age||Gender||Vision Impairment Description||Learned Cooking Before or After Vision Loss|
|P1||25||Male||Totally Blind, Congenital||After|
|P2||33||Male||Legally Blind, Congenital||After|
|P3||32||Non-binary||Legally Blind, Congenital||After|
|P4||19||Male||Totally Blind, Congenital||After|
|P5||36||Male||Totally Blind, Acquired (7 years)||Before|
|P6||24||Female||Legally Blind, Congenital||After|
|P7||22||Male||Totally Blind, Congenital||After|
|P8||35||Male||Legally Blind, Acquired (1 year)||Before|
|P9||28||Female||Totally Blind, Congenital||After|
|P10||31||Female||Legally Blind, Congenital||After|
|P11||55||Male||Legally Blind, Congenital||After|
|P12||48||Female||Legally Blind, Congenital||After|
4.1. Method: Semi-structured Interview with People with Visual Impairments
To understand perceptions and challenges of cooking for people with visual impairments, we conducted semi-structured interviews with 12 visually impaired people who have experience cooking (Table 3). Our participants have an average age of 32, with a range from 19 to 55 years old. Five of them are totally blind, and the rest are legally blind. Furthermore, ten of our participants are congenitally blind and learned cooking after the vision loss, and the other two cooked before losing their vision. Participants were recruited through social platforms (e.g., Reddit, Twitter, Facebook). To participate in our interview, participants needed to 1) be 18 or above; 2) have visual impairments; 3) have experience cooking; 4) be able to communicate in English. The interviews were conducted through Zoom and took around 60 - 75 minutes for each participant. Participants who completed the interview were compensated by a $20 Amazon gift card. The entire recruitment and study procedure was approved by the institutional review board (IRB).
In the interview, we first asked participants about their demographic information, general reasoning and perceptions about cooking, and any barriers they encountered when they started cooking. Based on our findings of different cooking procedures and practices in the YouTube video analysis, we asked our participants to discuss their perceptions and challenges around different cooking procedures, such as measuring, pouring, and cutting and chopping. Afterward, we asked participants to discuss their experiences and challenges with assistive cooking technologies, cooking with other people, and the modification of kitchen appliances for accessibility purposes.
Two researchers independently performed open-coding (Charmaz, 2006) on the interview transcripts. Then, the coders met and discussed their codes. When there was a conflict, they explained their rationale for their code to each other and discussed to resolve the conflict. Eventually, they reached a consensus and consolidated the list of codes. Afterward, they performed affinity diagramming (Hartson and Pyla, 2012) to group the codes and identify emerging themes. Overall, we established eight themes and 26 codes. The results introduced in the findings are organized based on eight themes.
In this section, we introduce the challenges of cooking derived from our interviews in eight themes: utilizing tools, information access, touching and feeling, safety and consequences, precision and ambiguity, organizing and tracking, item and quality inspection, and collaborative cooking and communication.
4.2.1. Utilizing Tools
From the interviews, we found that people with visual impairments encountered various challenges when utilizing tools (e.g., cooking utensils, appliances). In the video analysis, we found that people with visual impairments rely on either braille markers or bump dots to interact with an interface (Section 3.3.12). In the interview, seven participants commented on the concerns of messing up tools with dirty hands and food during cooking. For example, P1 commented on the problems of messing up braille tactile markers on kitchen appliances:
“…It is very common that I might accidentally have something on my hand, especially when I make a dough, then when I touch my Braille markers or bump dots, it just stays on the marker and affects me when I want to read the markers…”
From the video analysis, we learned that people with visual impairments usually utilize many tools for different purposes (Section 3.3.9). In the interview, we found that eight of our participants struggled with having too many tools for different purposes than are necessary. For example, P12 mentioned that she had to use a vegetable holder on one hand to hold the vegetable and use another hand to peel it. P4 further commented on his situation of even using syringes for cooking:
“…I have too many tools or cookware for different purposes. I even used syringes to measure and add soy sauce to my dishes. But having too many things really freak me out, it made me look for things all the time…”
During the interview, five participants mentioned that existing tools lack confirmation and feedback, especially for our participants who learned cooking before vision loss and were used to leverage vision for confirmation and feedback. For example, P8 mentioned the challenges of just using his spatula to tell where food is and whether the food is done:
“…Feel through tools is hard, I cannot just use my spatula to do everything for me, it is really hard for me to use it to tell if the food is ready by using tools to feel the texture. That is why I said feel is difficult without hand, that is why we need more help with adding more sensors at the head of tools to tell me different status, and it can also accomplish basic needs…”
4.2.2. Information Access
In the interview, participants overall complained about information access problems with kitchen appliances’ guides and recipes. First, three participants mentioned the lack of details for guide manuals and instructions of kitchen appliances. For example, P11 commented on the need for more vocal descriptions about the interface layout:
“…I cannot understand existing manuals and instructions for kitchen appliances like pressure cookers, it definitely needs more vocal descriptions about the position of different parts and the interface layout. Especially that the current pressure cookers or rice cookers have so many buttons…”
Beyond the manuals, we found our participants complained that recipe content does not correspond to how people with visual impairments cook. This includes cookbooks that have too many pictures (P9), wording or cooking language varying among different recipes which confused visually impaired cooks (P7), online recipes containing many figures and ads (P1, P3), and the way current recipes explain different steps requiring lots of vision support (P8). P8 continued with his experiences with recipes like a guessing game:
“…The descriptions in many recipes are very hard to correspond to the ways that I cook now. I was reading a recipe, and it told me to cook the piece of chicken until it turns brown and then flip it over. It does not transfer to people with visual impairments, it is like a guessing game…”
Furthermore, we realized that existing recipe structures are not friendly to people with visual impairments. Participants commented that different recipes they found online tend to have different recipe structures, such as the cook’s notes, nutrition facts, ingredients and cooking steps. As a consequence, our participants complained that they always have to go back to check the recipe (P1) and do not know where to start the recipe (P1, P3). P6 then further talked about her frustration of starting over from the beginning of the recipe because of the non-standardized recipe structure:
“…I use voiceover for apple devices, it typically read the whole recipe. However, many recipes do not follow a specific structure which made me easily miss information. Once I lose my place, I have to use voiceover to read from the beginning again…”
Besides the need for better recipe structures, we also found that our participants had a difficult time interacting with the recipe while cooking. Challenges include problems interacting with phones or tablets while cooking (P1), irrelevant ads and pictures (P10), and worries of getting both physical recipes and electronic devices dirty while cooking (P10). P10 continued:
“…I just cannot find a better way to interact with both my braille recipes and electronic recipes by getting them dirty and have a better interactive interface while cooking. Because my hands often have oil or sauce when I am cooking my dishes…”
Therefore, it is necessary for future research to consider how to better support cooking related information access and better interactive methods while cooking.
4.2.3. Touching and Feeling
In the YouTube video analysis, we mentioned that people with visual impairments leverage their hands’ feeling for measuring (Section 3.3.3), placing pans on a burner (Section 3.3.5), baking (Section 3.3.6), and testing food for “doneness” (Section 3.3.10). From the interview, we found that there exist various challenges and concerns about this from people with visual impairments in cooking. First, we found that hand feeling is not sufficient to tell all the information about certain cooking procedures. For example, P12 commented: “it can be difficult to feel the liquid level if the liquid is room temperature”. Therefore, P5 complained that he had to put oil in the fridge first and then use a finger to feel the level. P1 further mentioned his difficulties in using his hand to feel the pouring speed of liquid:
“…When I use my fingers to lead the liquid to pour into a container, I can tell somethings flows over my finger, but I cannot tell how fast that is, and it usually ended up with having too much or too little that got poured, because I do not have another hand to feel the weight of the container, I have to use one hand to pour and use one hand to lead the liquid…”
Another concern from our participants is that using hand feeling during cooking might not be appropriate or unsafe. As an explanation, P5 mentioned his experiences of messing up the decoration of a cake by using his hands. Moreover, P9 and P12 mentioned that using hands when cooking might cause them to burn themselves by accidentally touching the hot pan. P12 continued:
“…Using hands to feel the food on a hot pan is not always easy to do, you have to be very very slow and gentle, especially when you want to feel the doneness of the food and try to flip it. I have burned myself many times…”
Furthermore, we found that visually impaired individuals complained that using hands to feel objects affects the efficiency of cooking, that is, the person cannot use their hand, which is used for feeling and touching, to do other things. As an example we mentioned previously, the person has to use one hand to pour and one hand to lead the liquid to the container, which forces them to place the container at a stable position (P10). Therefore, future research should consider how to enable hand and finger sensing to help with knowing more information during different cooking procedures and explore alternative methods to substitute the need for hand touch and feel in cooking.
4.2.4. Safety and Consequence
From the interview, we found that all participants showed strong concerns of getting burned during different cooking procedures, which need heat detection assistance while cooking. For example, taking things out of the oven puts the visually impaired person in danger (P8), and the plate and dish for baking can be very hot even after taking it out of the oven (P7). P8 explained the danger of exposed heat sources:
“…A lot of kitchen appliances have exposed heat sources, it can be very dangerous, such as my toaster or the steamer. After I burned myself once, I always pay 200% attention when I am cooking…”
Beyond burning, we found participants reported the difficulty of handling accidents while cooking. This includes accidentally knocking things over (P3), it being hard to know if there is an oven fire or not (P10), and accidentally dropping things and making a mess (P5). P5 continued:
“…I had experiences making scrambled eggs and spilled the egg all over the stovetop. However, I did not notice the mess after cooking, and my partner told me this situation the next day. I need something to alert me or help me to be aware of these things happened…”
Finally, we found that seven participants mentioned that other people walking by could potentially cause safety concerns, especially for participants who learned cooking before their vision loss. For example, P8 mentioned: “you may turn around, and someone might stay behind you, and you hold a knife…It is a factor that is out of your control with another person while cooking.” Thus, future research should explore how to help people with visual impairments detect heat to prevent burns, track accidents, and people walking by.
4.2.5. Precision and Ambiguity
In the YouTube video analysis, we introduced the need for measuring (Section 3.3.3) while cooking. From the interview, we found that making precise steps while cooking can be a huge barrier for people with visual impairments, such as getting the right amount of baking soda while baking, because using a hand to feel the level in a measuring cup of powders is challenging (P1). P7 further mentioned the problem with always getting more during spreading:
“…Because it is hard to know how much butter I already spread on my bread, I always ended up with taking more than needed…”
P11 further continued with commenting on the difficulty of decorating the food:
“…Decorating the cake or dishes is extremely hard for me. It took a lot of practice to get used to frosting or adding decorations. And it is very easy to get messed up…”
To reduce cooking risks, we showed different safe cooking tips in Section 3.3.1. However, we found that our participants complained about the long learning curve of new methods which makes people stick with their familiar way of cooking. P7 mentioned: “every person has their own of doing things.” P3 further commented on the hardness of adopting advice from other people:
“…It is really difficult to get advice from other people when I am already used to doing things in a certain way. I know my current way of cutting and chopping might not be the recommended way by blind cooking guidelines, but I am already used to it, I tried using the recommended method of cutting and chopping, and I cut myself while learning it…”
According to participants’ responses, another challenge is following cooking steps or procedures precisely according to a recipe or instructions. P10 mentioned that a simple mistake during cooking could end up ruining the dish, such as measuring baking soda and tracking the leavening time during baking. All participants mentioned the correlation between the tedious work of following all cooking steps and procedures and why they do not like to cook. Therefore, future research should explore different tracking and measuring methods to help people with visual impairments follow steps easier and should create various teaching techniques to reduce the learning curve of different safe cooking methods.
4.2.6. Organizing and Tracking
In Section 3.3.11, we showed that people put a lot of effort into organizing their kitchen space to reduce barriers. From the interviews, we uncovered various challenges in organizing the kitchen and tracking objects. First, we found that participants complained about the challenge of finding previously used objects or cookware due to the mental load during cooking. P1 commented: “It is really easy for me to forget where my used knives are.” P10 also mentioned that it is hard to find previously used ingredients. P3 then commented on the need of having a system that continuously tracks the kitchen environment:
“…Cooking itself is a very complicated task that requires people to do multi-tasking and keep tracking of different things. For me, I sometimes forgot where my used knives are, and I had to be very conscious about my actions to prevent being cut by touching my used knife. Therefore, I think I need a tracker that can track the positions of different utensils and cookware for convenience and safety…”
Furthermore, we found that cooking is not a simple binary task that will guarantee successful and tasty dishes if you follow the recipe. We also realized people with visual impairments have a hard time tracking the status of tasks in progress. Five participants commented that it is hard to know when things are done. They have to constantly check the food. P1 further commented on this problem and tended to overcook the food:
“…Knowing whether the food is done is such a complicated project for me without any vision. To make sure I do not get diarrhea afterward, I often overcooked the food, such as shrimp. Although I can make sure it is totally cooked, but it affected the texture and the taste of the food…”
We also found that participants have challenges tracking and organizing parts of the food, such as organizing the cut and uncut parts of a large volume of food, and memorizing which parts of the food on the pan need to be flipped (P10). Therefore, P1 said that she had to wash her hands a lot and touch everything to check the food. P6 continued with her challenges:
“…The pan is big. If only a portion of the food on the pan needs to be turned over, it would be very difficult. I usually had to use hand and smell to check if the food at the center of the pan is burned or not…”
Finally, participants mentioned that they often have a hard time knowing and remembering where to clean while cooking. Specifically, six participants complained about remembering where they need to clean afterward, or they have to clean things right after use. P4 further commented that it is also very common for him to not know or not be aware of where he has to clean. Overall, future works should explore how to help people with visual impairments to better organize and track things through cooking.
4.2.7. Item and Quality Inspection
From the interview, we realized that people with visual impairments often have concerns about inspecting an item or food. First of all, we found that participants have a hard time checking food quality, such as whether the bread is moldy. P5 commented on the difficulty of checking food quality just with smell and touch, even with computer vision assistance:
“…Knowing the food quality is hard just by smelling and touch. Such as knowing whether a piece of bread is moldy or whether bananas or pineapples are ripe. I even tried with some apps that could object recognition, but I realized it just told me the object it is…”
Furthermore, we found that our participants complained about the task of telling whether things got peeled or cleaned thoroughly. P8 commented on cleaning and peeling non-smooth vegetables:
“…Different vegetables have different difficulties in cleaning and peeling. Vegetables like carrots are very easy to peel. But non-smooth vegetables are hard to tell if they got peeled or cleaned completely. Ginger is a hard one, also spots on potatoes are hard too…”
In the interviews, nine participants reported having a hard time inspecting an accidental mix of unwanted things, such as eggshells and bugs in flour (P1). P1 continued: “I had many times of having eggshells in my scrambled eggs without noticing, it was disgusting.” Overall, we showed the existing challenges of items and quality inspection. Future research should explore how to bridge the gap of the challenges we introduced to people with visual impairments in cooking.
4.2.8. Collaborative Cooking and Communication
From interviewing people with visual impairments, we found that ten out of twelve participants mentioned that they do not like cooking with another person because of continuous communication needs. “Cooking with another person needs me to constantly speak out things.” said P4. P3 further complained about collaborative cooking processes:
“…I have to verbally confirm steps and actions with another person all the time. Sometimes they might just do something in a way that I do not like, and they might not inform me because cooking and talking is not common for any person…”
In the interview, we found that eight participants leveraged apps for remote assistance while cooking, such as Aira (Corp, 2021) or Be My Eyes (Wiberg, 2021). However, our participants complained about making sure the item is in the camera’s field of view. P1 continued:
“…Remote assistance definitely needs more improvements, I often do not know whether the thing I am talking about is in the camera’s field of view. More importantly, my hands are often dirty while cooking, and I always tried not to touch my electronic device. This leads to more concerns about using remote assistance…”
Moreover, we found that our participants complained about appliance modification barriers in a communal space. P8 commented on this problem: “the biggest thing about using a shared kitchen, even the simple one like adding bump dots or braille markers, it is hard for me to just add these markers on different appliances, because they are shared.” Therefore, future research should explore how to improve the social acceptability of adding tactile markers on appliances or other alternative ways for people to interact with different appliances.
5. Discussion and Future Work
From the results of our interviews with people with visual impairments, we uncovered existing challenges and gaps for visually impaired people in cooking. In this section, we further discuss zero-touch interactions for cooking, status tracking and safety monitoring, collaborative cooking, and contextual inquiry and learning process in-depth.
5.1. Zero-touch Interactions for Cooking
From the results, we showed that our participants often rely on touching and feeling when cooking (e.g., measuring, checking for food doneness, spreading), and interacting with touch-based electronic devices might be inconvenient because they often have food on their hands. As an input method, prior research has explored speech (Choi et al., 2020; Branham and Mukkath Roy, 2019; Abdolrahmani et al., 2020; Storer et al., 2020) and mid-air gestures (Dim et al., 2016) as input options for people with visual impairments. However, many of the existing interaction paradigms still require people to physically touch the interfaces (e.g., (Ye et al., 2014)), especially to explore elements on the screen. Although visually impaired individuals can currently leverage voice control to input certain commands (setting a timer), future research should explore how to best integrate zero-touch interactions with electronic devices alongside existing practices utilizing touch to manipulate ingredients and cooking tools.
5.2. Status Tracking and Safety Monitoring
From our interviews, we highlighted that people with visual impairments have various difficulties in tracking the status of tasks in progress (Section 4.2.6), protecting themselves from getting burned during different cooking procedures (Section 4.2.4), and handling accidents while cooking (Section 4.2.4). Prior work has explored using thermography to detect heat sources for energy auditing (e.g., (Mauriello et al., 2017, 2019)). A similar approach could be applied in the kitchen to detect heat hazards. Beyond heat detection, prior research also explored using computer vision approaches to opportunistically capture actions and provide proactive reminders to users with visual impairments (Kianpisheh et al., 2019). Similar approaches using computer vision or audio (Laput et al., 2018) could be applied in the kitchen to track object locations or track user’s actions and status of cooking steps (e.g., boiling water).
5.3. Collaborative Cooking
The majority of our participants complained about cooking with other people because they have to constantly speak about their actions and ask about other people’s actions. Cooking while constantly talking might not be a natural behavior for people both with and without visual impairments. However, three participants did mention the benefit of having another person to accomplish tasks that rely on vision, such as checking expiration dates and food doneness (P10). Prior research has explored collaborative behaviors for people with visual impairments for other purposes, such as gaming (Gonçalves et al., 2020), use of tangible interfaces (Chibaudel et al., 2020), music composition (Omori and Yairi, 2013), and creating accessible home environments (Branham and Kane, 2015). Therefore, future research should explore the collaborative cooking behaviors for people with visual impairments and further create guidelines and training methods to reduce safety concerns as well as both mental and physical effort.
5.4. Contextual Inquiry and Learning Process
In this paper, we described our research approach using YouTube Video analysis to understand more about the practices of cooking by people with visual impairments. In general, video analyses of this type have the benefit of allowing researchers to gain a broad understanding of the practices of assistive technology use for many different users. This has been exemplified in prior work in HCI and Accessibility (e.g., (Anthony et al., 2013; Hourcade et al., 2015)). In future research, a contextual inquiry study is well suited to gather a deeper understanding of individual users’ practices and challenges (e.g., (Dosono et al., 2015; Li et al., 2019; Krome et al., 2016; Kianpisheh et al., 2019)), which may then be aggregated to understand themes across those individuals. As it relates to the research opportunities we discussed above, a contextual inquiry study may enable future researchers to deeply explore the practices and challenges of zero-touch interactions, status and safety monitoring systems, and collaborative cooking. For example, conducting a contextual inquiry study at a participant’s kitchen could identify more status and safety monitoring concerns that might not be recognized by the participants with visual impairments and are not included in the camera’s field of view in YouTube videos. An example of this might be watching a visually impaired cook show how she cuts vegetables in front of the camera while, off-camera, cooking chicken in a pan. In this case, neither the audience nor researchers may recognize any potential safety concerns with the pan or other objects in the surroundings from the video. Beyond only interviewing visually impaired individuals, we realized it would also be beneficial for future research to conduct interviews with teachers or trainers who work at community centers and help people with visual impairments with their activities of daily living and meal preparation. This may generate more common practices of cooking by people with visual impairments.
All of the participants in our study were either legally blind or totally blind. We think people with low vision may have different cooking strategies and challenges that utilize their visual perceptive abilities. In our study, we focused on understanding the existing practices and challenges of visually impaired people in their cooking. However, we did also find that people with visual impairments have different learning experiences with cooking. Therefore, people might encounter different challenges at different stages of learning to cook.
In this paper, we describe the results of content analyses with 122 YouTube videos featuring visually impaired people preparing meals and uncovered unique cooking practices for people with visual impairments (e.g., use hands to weigh meat and measure spices, use sound and smell to test if the food is done). We then present findings from semi-structured interviews with 12 visually impaired participants and highlight existing challenges encountered while cooking from their perspectives (e.g., tracking the status of tasks in progress, telling whether ingredients are peeled or cleaned thoroughly). We then discussed potential opportunities to integrate technology with the existing practices for cooking, status tracking, safety monitoring, and collaborative cooking. Overall, our findings provide guidance for future research exploring various assistive technologies to help people cook without relying on vision.
- ” Siri talks at you” an empirical investigation of voice-activated personal assistant (vapa) usage by individuals who are blind. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 249–258. Cited by: §2.2.
- Blind leading the sighted: drawing design insights from blind users towards more productivity-oriented voice interfaces. ACM Transactions on Accessible Computing (TACCESS) 12 (4), pp. 1–35. Cited by: §5.1.
- Analyzing user-generated youtube videos to understand touchscreen use by people with motor impairments. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1223–1232. Cited by: §3, §5.4.
- PassChords: secure multi-touch authentication for blind people. In Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility, pp. 159–166. Cited by: §2.2.
- An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends. Journal on Multimodal User Interfaces 11 (2), pp. 149–172. Cited by: §2.1.
- Food experiences and eating patterns of visually impaired and blind people. Canadian Journal of Dietetic practice and research 70 (1), pp. 13–18. Cited by: §1, §2.1.
- Collaborative accessibility: how blind and sighted companions co-create accessible home spaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 2373–2382. Cited by: §5.3.
- Reading between the guidelines: how commercial voice assistant guidelines hinder accessibility for blind users. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility, pp. 446–458. Cited by: §2.2, §5.1.
- Constructing grounded theory: a practical guide through qualitative analysis. sage. Cited by: §3.2, §4.1.
- ” If you’ve gone straight, now, you must turn left”-exploring the use of a tangible interface in a collaborative treasure hunt for people with visual impairments. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–10. Cited by: §5.3.
” Nobody speaks that fast!” an empirical study of speech rate in conversational agents for people with vision impairments. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13. Cited by: §5.1.
- Home - aira : aira. Note: https://aira.io/(Accessed on 04/08/2021) Cited by: §3.3.9, §4.2.8.
- Designing mid-air tv gestures for blind people using user-and choice-based elicitation approaches. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems, pp. 204–214. Cited by: §5.1.
- “I’m stuck!”: a contextual inquiry of people with visual impairments in authentication. In Eleventh Symposium On Usable Privacy and Security (SOUPS 2015), pp. 151–168. Cited by: §5.4.
- Using computer vision to access appliance displays. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility, pp. 281–282. Cited by: §2.2.
- Playing with others: depicting multiplayer gaming experiences of people with visual impairments. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–12. Cited by: §5.3.
- Vizlens: a robust and interactive screen reader for interfaces in the real world. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 651–664. Cited by: §2.2.
- Facade: auto-generating tactile interfaces to appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 5826–5838. Cited by: §2.2.
- The blind cook – christine ha’s adventures. Note: http://www.theblindcook.com/(Accessed on 03/14/2021) Cited by: §1.
- Cooking navi: assistant for daily cooking in kitchen. In Proceedings of the 13th annual ACM international conference on Multimedia, pp. 371–374. Cited by: §2.3.
- The ux book: process and guidelines for ensuring a quality user experience. Elsevier. Cited by: §3.2, §4.1.
- TacTILE: a preliminary toolchain for creating accessible graphics with 3d-printed overlays and auditory annotations. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 397–398. Cited by: §2.2.
- How does a blind person cook? - the blind guide. Note: https://theblindguide.com/how-does-blind-person-cook/(Accessed on 03/14/2021) Cited by: §1.
- Look, my baby is using an ipad! an analysis of youtube videos of infants and toddlers using tablets. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1915–1924. Cited by: §5.4.
- Accessibility - vision - apple (ca). Note: https://www.apple.com/ca/accessibility/vision/(Accessed on 03/16/2021) Cited by: §2.2.
- An analysis of the impact of visual impairment on activities of daily living and vision-related quality of life in a visually impaired adult population. British Journal of Visual Impairment 37 (1), pp. 50–63. Cited by: §1, §2.1.
- Slide rule: making mobile touch screens accessible to blind people using multi-touch interaction techniques. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, pp. 73–80. Cited by: §2.2.
- Interactive cooking simulator: showing food ingredients appearance changes in frying pan cooking. In Proceedings of the 5th international workshop on Multimedia for cooking & eating activities, pp. 33–38. Cited by: §2.3.
- Face recognition assistant for people with visual impairments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3 (3), pp. 1–24. Cited by: §5.2, §5.4.
- Underneath the skin: an analysis of youtube videos to understand insertable device interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12. Cited by: §3.1.
- Food shopping, sensory determinants of food choice and meal preparation by visually impaired people. obstacles and expectations in daily food experiences. Appetite 113, pp. 14–22. Cited by: §1, §2.1.
- Contextual inquiry of future commuting in autonomous cars. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 3122–3128. Cited by: §5.4.
Calculating cooking recipe’s difficulty based on cooking activities.
Proceedings of the 9th Workshop on Multimedia for Cooking and Eating Activities in conjunction with The 2017 International Joint Conference on Artificial Intelligence, pp. 19–24. Cited by: §2.3.
- Ubicoustics: plug-and-play acoustic activity recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pp. 213–224. Cited by: §5.2.
- Interacting with mobile devices via voiceover: usability and accessibility issues. In Proceedings of the 24th Australian Computer-Human Interaction Conference, pp. 339–348. Cited by: §2.2.
- FMT: a wearable camera-based object tracking memory aid for older adults. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3 (3), pp. 1–25. Cited by: §5.4.
- BrailleSketch: a gesture-based text input method for people with visual impairments. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 12–21. Cited by: §2.2.
- Get started on android with talkback - android accessibility help. Note: https://support.google.com/accessibility/android/answer/6283677?hl=en(Accessed on 03/16/2021) Cited by: §2.2.
- Thermporal: an easy-to-deploy temporal thermographic sensor system to support residential energy audits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–14. Cited by: §5.2.
- Exploring novice approaches to smartphone-based thermographic energy auditing: a field study. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 1768–1780. Cited by: §5.2.
- Clearspeech: a display reader for the visually handicapped. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14 (4), pp. 492–500. Cited by: §2.2.
- Collaborative music application for visually impaired people with tangible objects on table. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–2. Cited by: §5.3.
- Getting smartphones to talkback: understanding the smartphone adoption process of blind users. In Proceedings of the 17th international acm sigaccess conference on computers & accessibility, pp. 23–32. Cited by: §2.2.
- BlinDar: an invisible eye for the blind people making life easy for the blind with internet of things (iot). In 2017 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), pp. 71–75. Cited by: §2.2.
- ” All in the same boat”: tradeoffs of voice assistant ownership for mixed-visual-ability families. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14. Cited by: §5.1.
- Real-time detection and reading of led/lcd displays for visually impaired persons. In 2011 IEEE Workshop on Applications of Computer Vision (WACV), pp. 491–496. Cited by: §2.2.
-  () Safe cooking techniques for cooks who are blind or have low vision - visionaware. Note: https://visionaware.org/everyday-living/essential-skills/cooking/safe-cooking-techniques/(Accessed on 04/05/2021) Cited by: §1, §3.2.
- Be my eyes - see the world together. Note: https://www.bemyeyes.com/(Accessed on 04/10/2021) Cited by: §4.2.8.
- Christine ha - wikipedia. Note: https://en.wikipedia.org/wiki/Christine_Ha(Accessed on 03/14/2021) Cited by: §1.
- Current and future mobile and wearable device use by people with visual impairments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3123–3132. Cited by: §5.1.
- An iot system design for blind. In 2017 14th Web Information Systems and Applications Conference (WISA), pp. 90–92. Cited by: §2.2.