SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches

09/07/2021 ∙ by Haoran Xie, et al. ∙ JAIST 0

In this work, we propose an interactive general instruction framework SketchMeHow to guidance the common users to complete the daily tasks in real-time. In contrast to the conventional augmented reality-based instruction systems, the proposed framework utilizes the user sketches as system inputs to acquire the users' production intentions from the drawing interfaces. Given the user sketches, the designated task instruction can be analyzed based on the sub-task division and spatial localization for each task. The projector-camera system is adopted in the projection guidance to the end-users with the spatial augmented reality technology. To verify the proposed framework, we conducted two case studies of domino arrangement and bento production. From our user studies, the proposed systems can help novice users complete the tasks efficiently with user satisfaction. We believe the proposed SketchMeHow can broaden the research topics in sketch-based real-world applications in human-computer interaction.



There are no comments yet.


page 5

page 6

page 7

page 8

page 10

page 12

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In our daily lives, numerous tasks required special skills or adequate experience, which usually frustrate the common and amateur users to make any attempt. In addition, the daily tasks may be tedious and complex procedures, and it is difficult and challenging to conduct creative activities. In the research fields of creativity support in human-computer interaction, the previous works mainly focused on the graphical user interface in the computer environment. In this work, we aim to help the common user to complete the designated tasks under the computed guidance in real-world applications.

To solve this issue, the common approaches are spatial computing techniques of instruction systems in a workspace, such as augmented reality approaches with head-mounted display and projection, see-through displays such as tablet [2, 12]. Considering the system stability and cognitive loads of the instruction systems, we adopt the spatial augmented reality technology with projector-camera system [5]. Especially, the depth camera was used for sensing the depth information for instruction generation rather than the standard web cameras. In the previous instruction works [5, 23], the users are asked to complete the designated task with the given target where the requirement of satisfying the users’ design intention is absent. In this work, we combine the creativity support and instruction systems to help users to conduct tasks with their own designs. To achieve this research purpose, we utilized the freehand user sketches as system input which are difficult to be handled due to the ambiguous and abstract visual representations.

Therefore, we propose an interactive instruction system, SketchMeHow that can provide the users with projection-based guidance in real-time with spatial augmented reality technique. As shown in Figure 1, the system input of SketchMeHow may include freehand line drawing or color illustration. With the help of user guidance in projector-camera systems, the completed tasks can be achieved at low time costs and high success rates.

Figure 1: The proposed interaction framework of SketchMeHow.

To verify the proposed SketchMeHow framework, we conducted two usage applications for task instruction from user sketches: domino block arrangement and bento production. In the block arrangement system, the stroke input is analyzed with the individual block, and the gradient colors are projected on the blocks for guiding the correct positions. Finally, the domino blocks can be collapsed successfully. In the bento production system, we adopt both stroke and color of user sketches as input. The food ingredients are recommended based on user input and arranged with the projected color guidance. The accomplished bento is in compliance with the user sketches well. From our user study and subjective questionnaires, we verified that the proposed framework of SketchMeHow can provide the interactive projection guidance to help non-skilled users to complete theirs desired target in specific tasks. The proposed systems are effective and useful compared with the conventional approaches without guidance. We believe SketchMeHow can facilitate sketch-based systems for real-world applications.

2 Related Work

2.1 Sketch-based Interface

Freehand sketches can present the users’ design intention and have been adopted intensively in two-dimensional content design. Teddy proposed the sketch-based interface to construct 3D polygonal models from freehand drawn sketches [11]. A sketch-based interface was usually adopted in the image and shape retrieval [3] and drawing guidance [26]

. Recently, the deep learning algorithms were widely developed in the sketch user interface, such as facial image editing

[19] and drawing [10], anime image editing [15], image translation [6]

and normal map estimation

[7]. Besides static applications of image editing, the physical dynamics were controlled with user sketches such as hair [27] and fluid [9]. In this work, we especially focus on the sketch interfaces for task instructions with augmented reality approaches.

2.2 Sketch-based Guidance

It is challenging and useful to adopt the sketch-based design for real-world applications with interactive user guidance. Sketch and run used the sketch-based interface for robot control based on a top-down camera view [21]. Lighty proposed the guidance system of room lights by sketching a target illumination area on tablet devices [16]. However, these proposed systems were designed for automatic machine control where the user guidance was absent such task instruction. To solve this issue, Sketch2Domino proposed a sketch-based guidance system for domino toppling production [17]. Sketch2Bento proposed the guide system for ingredient arrangement of designing lunch boxes [24]. In this work, we aim to propose a general instruction framework for sketch-based user guidance in real-world applications.

2.3 Projection-based Guidance

The spatial augmented reality adopts interactive projection for various real-world applications. Illuminating clay was proposed as a tangible interface for landscape design [18], Flagg et al. worked on the implementation to guide oil painting using the projection techniques [4]. There are several works to adopt projection-based guidance in fabrication purposes, such as sculpting [20]. For large-scale fabrication, BalloonFab guided the user to create the large-scale balloon art [25]. A similar approach was used in newspaper sculpture [14]. For daily activities, the projection guidance can also be helpful in solving the puzzle of Rubik’s cubes [1], calligraphy practice [8], and sports training [22]. For room-scale projection, Roomalive was proposed for multiple projector-camera systems [13]. In contrast to these projection-based guidance approaches, we first adapt user sketches as system control to meet user design intentions.

3 Framework Overview

Figure 2: The proposed interaction framework of SketchMeHow.

Figure 2 shows the proposed framework of SketchMeHow which is composed of three modules on the separated platforms: system input, task instruction analysis, and projection-based guidance. For the input module, a drawing user interface is developed with the stoke and coloring function with pen-based interaction on a tablet or real workspace. For the task instruction analysis module, the input hand-drawn sketches are cleaned with stroke smoothing to exclude the infeasible instructions. According to the requirements of specific works, we divide the whole sketch instruction into separated sub-tasks with the elemental objects, such as individual blocks in element construction. Finally, the locations of each task object are calculated based on the subtasks. For the projection-based guidance module, we adopt the projector-camera system with the depth camera. The system calibration is conducted based on multi-pin correction to adjust the projector and camera coordinates. Then, the current workspace is detected with depth differences from the captured depth maps. With the calculated task instruction, the guidance information is projected onto the workspace overlapped with the calculated operations. After the iterative processes of subtasks, the desired productions can be achieved with guidance.

3.1 Drawing Interface

The sketching interface can be developed based on the graphical user interface or tangible physical interface. The examples of user interfaces are shown in Figure 3. For the graphical drawing interface, the canvas is used for stroke-based interaction as Figure 3(a). Besides the canvas, the user can select tools from the toolbars includes pencil, eraser, line, and different shapes. In addition, the user can select the stroke sharpness and stroke color. For the physical drawing interface, it is convenient for the user to draw with a pen-type input device. A simple solution is to use the retro-reflective marker on the pen tip in a depth camera-projector system as Figure Figure 3(b). In our case studies, we implemented both graphical and physical drawing interfaces in the SketchMeHow framework.

Figure 3: Drawing interface can be implemented as graphical user interface on computer (a) or the 3D spatial interface on real desk (b).

3.2 Spatial Augmented Reality

Due to the absence of head-mounted devices, it is more convenient in instruction guidance systems using interactive projection [5]. As shown in Figure 4(a), a projector-Camera System is usually adopted in the setup of spatial augmented reality system. The analyzed results from the drawing interface are projected on the workspace. In the prototype systems in our case study, we used a depth camera to capture the depth map or image recognition in infrared frames. An example setup of the proposed system is shown in 4(b). We used the pin-points tool to ensure the appropriate calibration of projector-camera system.

Figure 4: Spatial augmented reality system (a); an example setup for bento design (b).
Figure 5: Projection guidance for task instruction (a) and the system usage of the proposed SketchMeHow framework with bento production (b).

3.3 User Guidance

In our prototype systems, the depth data of captured depth map is used to guide the arrangement of instruction elements. For the projector calibration, the guidance image projected onto the workspace is distorted to match the coordinates of the depth view. In addition, we stored the pixel positions for each color in the scaled guidance image to provide guidance for each sub-tasks. The guidance image is projected onto the workspace with different colors to help complete tasks as shown in Figure 5(a). We map the stored color list to the depth data to obtain the areas, and the user can place the objects under the corresponding tasks. For example, a black color denotes the area where the object should not be placed. In addition, the green color refers to the correct placement, and the red color for incorrect placement. An example of the usage of the proposed instruction system is shown in Figure 5(b).

4 Case Study

To verify the proposed SketchMeHow framework, we conducted two case studies of domino arrangement and bento production.

4.1 Domino Arrangement

To support designing a complex chain reaction such as domino blocks, we implemented a sketch-based instruction system for the domino placement according to the user customized sketch in this case study [17]. The prototype system applied interactive projection-mapping technology to guide users to places the domino blocks. Users were allowed to design the blueprint of the arrangement by the freehand sketches without considering the physical limitations of domino placement following the rules of chain reaction.

4.1.1 System Overview

Figure 6: The prototype system of the proposed domino arrangement system.

As shown in Figure 6, users can draw the sketch for the intention design of the domino arrangement. The proposed system analyzed the stroke data of input sketches to make the domino blocks collapse successfully in a valid arrangement where the vertices are excessively dense or the angles are quite acute. After that, the users can locate the domino blocks following the projection guidance to confirm the target positions.

4.1.2 System Environment

Similar to the example setup in Figure 4, we set the working space (70 cm-high desk) with a laser projector (LG HF80JG, 2000 lumen), and a depth camera (Microsoft Kinect V2, the resolution of depth image is 512 x 424). The projector and camera were set vertically 85.1 cm and 77.8 cm above the desk. The size of the projection area was 57.2 cm x 32.1 cm. In this study, standard domino blocks (23 mm wide, 46 mm high and 8 mm thick) were used which was certified by the Japan Domino Association.

4.1.3 Projection Guidance

The users can use a pen stick with the marker to draw a sketch directly on the projection area. The arrangement proposal according to the drawn target is projected as visual guidance of the valid domino’s positions, so that the user can arrange the domino blocks in the order following the design. For the correction of the valid domino’s position, a gradient-colored circle is projected at the center of the placed domino blocks for the user to confirm whether they have been placed correctly. The colors of the circles are set in gradient colors in terms of the distance from the target position. The circle on the top of the block is in red color if the distance is larger than 2.0 cm from the target position. The circle turns yellow about 1 cm away. The circle turns green when is correctly placed. Note that the target positions of domino are shown by white rectangles from the beginning.

4.1.4 User Study

We conducted a comparison study to verify the effectiveness of the proposed system. we compared the domino arrangement with and without the proposed system. In conventional approach without the systems, the participants were asked to place the domino blocks following the hand-drawn sketches on a printed paper without explicit guidance on the intervals between the domino blocks. The criterion for a successful design was that all the domino blocks could collapsed with one simple push. 12 graduate student participants (11 males, 1 female) joined the comparison study.

For our prototype system, the participants were asked to arrange the domino blocks with the help of the visual guidance. The user can start the arrangement from any location, because the arrangement order is non-trivial. Note that we adopted the same blueprint design used in the conventional approach. Both operational time costs and the results were recorded. In addition, we conducted a questionnaire for the subjective evaluations of the prototype system. The questionnaire had four questions: a) Which approach is easier for arranging domino blocks? b) Which perfection do you have confidence in? c) Which result gave you a sense of accomplishment? d) Which approach was more interesting?

Figure 7: The prototype system of the proposed bento production system.

4.2 Bento Production

In this case study, we implemented an ingredient placement support system for bento (lunch box) production based on the user hand-drawn sketches. The prototype system allows users to make the bento following creative intentions without relying on existing recipes.

4.2.1 System Overview

As illustrated in Figure 7, the user first sketches the desired arrangement image with the drawing interface. Then, the system provides the projection guidance to lunch box from the sketch input. Under the help of the production guidance, the user can complete the ingredient arrangement for desired bento. Finally, the real-time guidance is achieved based on the current working progress.

4.2.2 System Environment

As shown in Figure 4, we set the similar setup for domino arrangement where the desk height is 77 cm, a projector (EPSON dreamio EF-100, 2000 lumen), a Kinect V2 depth camera, and a pen tablet (Wacom Intous CTL-6100WL). The projector was placed 89 cm above the desk, while the height of camera was 80cm. The size of the lunch box was 20 cm 13.5 cm 4 cm. Because the light reflection may influence the captured infrared image, we placed a kitchen paper inside the box. We adjusted the canvas size of drawing interface to match the size ratio of the lunch box to be used. We set the canvas size to be 600px 405px as shown in Figure 3.

4.2.3 Projection Guidance

The color selection is decided according to the colors of the prepared food ingredients. We investigated the common food ingredients which are often used in character bento production and determined the color of the ingredients correlated to the sketch. The ingredients used in the prototype system were broccoli, fried egg, crab stick, and sausage. Therefore, the drawing interface allows the use of green, yellow, orange, and pink colors according to the above ingredients.

We use the depth sensor to detect the positions of food ingredients placed by the user. The depth map of the workspace without any ingredients is used as the environment map. In the comparison with the depth map with food ingredients, the environment map is excluded for noisy data. In our implementation, we decided the depth difference of 8.0 mm as the threshold value to detect the placement of food ingredients. If 70% of the projected color area has been filled with food ingredients, the arrangement is considered to be correct with the projected green color. On the contrary, the red color will be projected when the ingredients are placed outside of the target area. The arrangement is considered to be completed if the area of red color is less than 20% of the non-target area. The arrangement area then switches to the next step with a different color. If the subtasks of all colors are completed, a white color is projected.

4.2.4 User Study

We conducted a comparison study to verify the effectiveness of the arrangement support with the prototype system. 12 participants were recruited in two groups: Group A with the proposed system; Group B without the proposed system. All the participants were asked to arrange a lunch box. We evaluated the time costs and the levels of completion with a questionnaire in five-point scale (“did you complete your lunch box the way you wanted?”). For Group B, the participants created bento by drawing the blueprint on paper as a reference. For Group A, the participants used the guidance of the proposed system. All the participants were asked to fill a questionnaire about the sketching interface and the arrangement support system after the task. In our questionnaire survey, we designed three questions items with 5-Point Likert scale (1 for strongly disagree, 5 for strongly agree). The questions include: Q1: Did you follow the guidance?; Q2: Easy to observe the guidance; Q3: Easy to understand the guidance.

5 Results

In this section, the task instruction and evaluation results of two case studies developed in SketchMeHow framework are discussed, and we explore the further potential application with the proposed framework.

5.1 Domino Arrangement

Figure 8: The collapsed results (c) of domino arrangement (b) under user sketches (a).

Figure 8 shows the final arrangements of the domino matched the participants’ sketches well. The operational time cost of with/without the domino arrangement system is shown in Figure 9(a) . The average time cost using our system was 140 seconds (min 109 s, max 185 s). The average time cost without the system was 155 seconds (min 88 s, max 231 s). The variation and average of the processing time costs were reduced using the proposed system. We counted the number of participants who accidentally knocked down the domino blocks during arrangement which had 4 using the proposed system, and 8 without the system. Therefore, it is verified that SketchMeHow can not only shorten the arrangement time cost but also provide stability.

Figure 9: Arrangement time cost (a) and the results of questionnaire (b).

Figure 9(b) shows the results of the questionnaire in 5-Point Likert scale with the prototype system. It is verified that the proposed system has higher evaluation than without the system except for question c). In the case of the sense of accomplishment, the visual guidance that asked the participants to follow the projected positions may make participants feel boring. This issue can be improved by providing users more freedom of the modification in stroke drawing and domino blocks arrangement.

5.2 Bento Production

Figure 10: Users sketches and the arranged bento box using the proposed system.

Figure 10 shows examples of bento produced in the proposed SketchMeHow framework. Figure 11 shows the comparison results of production time cost of with/without the proposed system and level of completion (5 for the highest degree of completion). The average production time cost of Group B is 280.2 seconds, while that of Group A with the proposed system was 250.6 seconds. We found that the production time cost decreased with the proposed system. For the level of completion, the average score for Group B was 2.4, while Group A was 3.3. Group B achieved lower scores because the participants had difficulty to arrange ingredients without any guidance. The participant spent more time in production because the participants may hesitate to adjust the positions of ingredients to match the sketch.

Figure 11: From left to right: the production time costs (a), level of completion (b), and the questionnaire results (c).

Figure 10(c) shows The results of questionnaire that the scores of Q2 (4.1) and Q3 (4.3) were relatively higher than Q1 (3.6), which confirmed most of the participants were satisfied with the system guidance and the prototype system. Scores for Q1 was above 3.0, so the participants can follow the guidance well. Meanwhile, we found that the size of the lunch box is small and lack of ingredient choice in the study. The user may fail to fit the ingredients to the prompted position due to the inaccuracy between the food size and the prompted area.

5.3 Discussion

5.3.1 Cooperative Instruction System

In contrast to the instruction systems with head-mounted displays and tablets [5, 12], SketchMeHow can support not only individual use and also cooperative working styles. It is possible for multiple users to collaborate in sketch drawing and task completion.

5.3.2 Real World Applications

Not limited to the prototype systems developed in this work, we believe various real world applications may benefit from the proposed SketchMeHow framework, such as assembly tasks of hand craft and furniture construction, and fabrication work of hand-made objects and arts.

5.3.3 Large Scale System

In our case study, the desk-based tasks were adopted for framework verification. However, it is also feasible to support large scale instruction system, such as architecture-scale fabrication. For large-scale systems, the volumetric projection calibration may be required for accurate visual guidance, such as layered projection mapping technique [25].

6 Conclusion

In this work, we proposed a general sketch-based task instruction framework, SketchMeHow with the users’ hand-drawn sketches as system inputs. The framework consists of the drawing interfaces in graphical and physical styles, the computation platform for task analysis, and the projection guidance with projector-camera systems. To verify the proposed SketchMeHow framework, we conducted two case studies of domino arrangement and bento production. In terms of the user studies in two prototype systems, it is verified that SketchMeHow framework can help novice users in task instruction in low computation cost and high completion levels. The participants were satisfied with the usage and visual guidance provided by the prototype systems.

Based on the free comments in our case studies, the participants reported that the accuracy of recognition should be improved. In future, we plan to adopt depth sensor with higher resolution of depth images than the Kinect, such as the RealSense D455 depth camera. We believe that a higher resolution depth sensor can also reduce the noise data in prototype implementation. Under the SketchMeHow framework, the instruction system with user sketches can improve both the work efficiency and user satisfaction of our daily activities.


The authors thank the anonymous reviewers for their valuable comments. We greatly thank Yuki Mishima, Yamato Igarashi, Ryoma Miyauchi, Masahiro Okawa, Haruka Kanayama, Shogo Yoshida, Shogo Okada for idea discussion and experiment design. We would thank Ze Wang for sharing the codes of drawer interface. This project has been partially funded by JAIST Research Grant, Hayao Nakayama Foundation for Science & Technology and Culture Grant A and JSPS KAKENHI grant JP20K19845, Japan.


  • [1] S. Ajisaka, S. Hara, M. Matsuchi, S. Luo, S. Yoshida, H. Xie, and K. Miyata (2020) Learning rubik’s cube through user operation history. In 2020 Nicograph International (NicoInt), Vol. , pp. 43–46. External Links: Link, Document Cited by: §2.3.
  • [2] J. Blattgerste, P. Renner, B. Strenge, and T. Pfeiffer (2018) In-situ instructions exceed side-by-side instructions in augmented reality assisted assembly. In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, PETRA ’18, New York, NY, USA, pp. 133–140. External Links: ISBN 9781450363907, Link, Document Cited by: §1.
  • [3] M. Eitz, R. Richter, T. Boubekeur, K. Hildebrand, and M. Alexa (2012) Sketch-based shape retrieval. ACM Transactions on Graphics (TOG) 31 (4), pp. 31:1–31:10. External Links: Link, Document Cited by: §2.1.
  • [4] M. Flagg and J. M. Rehg (2006) Projector-guided painting. In Proceedings of the 19th annual ACM symposium on User interface software and technology, pp. 235–244. Cited by: §2.3.
  • [5] M. Funk, T. Kosch, and A. Schmidt (2016) Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. UbiComp ’16, New York, NY, USA, pp. 934–939. External Links: ISBN 9781450344616, Link, Document Cited by: §1, §3.2, §5.3.1.
  • [6] A. Ghosh, R. Zhang, P. K. Dokania, O. Wang, A. A. Efros, P. H. Torr, and E. Shechtman (2019) Interactive sketch & fill: multiclass sketch-to-image translation. In

    Proceedings of the IEEE International Conference on Computer Vision (ICCV)

    Long Beach, CA, USA, pp. 1171–1180. External Links: Document Cited by: §2.1.
  • [7] Y. He, H. Xie, C. Zhang, X. Yang, and K. Miyata (2021) Sketch-based normal map generation with geometric sampling. In International Workshop on Advanced Imaging Technology (IWAIT) 2021, M. Nakajima, J. Kim, W. Lie, and Q. Kemao (Eds.), Vol. 11766, pp. 261 – 266. External Links: Document, Link Cited by: §2.1.
  • [8] Z. He, H. Xie, and K. Miyata (2020) Interactive projection system for calligraphy practice. In 2020 Nicograph International (NicoInt), Vol. , pp. 55–61. External Links: Link, Document Cited by: §2.3.
  • [9] Z. Hu, H. Xie, T. Fukusato, T. Sato, and T. Igarashi (2019)

    Sketch2VF: sketch-based flow design with conditional generative adversarial network

    Computer Animation and Virtual Worlds 30 (3-4), pp. e1889. Note: e1889 cav.1889 External Links: Document, Link, Cited by: §2.1.
  • [10] Z. Huang, Y. Peng, T. Hibino, C. Zhao, H. Xie, T. Fukusato, and K. Miyata (2021) DualFace: two-stage drawing guidance for freehand portrait sketching. CoRR abs/2104.12297. External Links: Link, 2104.12297 Cited by: §2.1.
  • [11] T. Igarashi, S. Matsuoka, and H. Tanaka (1999) Teddy: A sketching interface for 3d freeform design. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), New York, USA, pp. 409–416. Cited by: §2.1.
  • [12] F. Jasche, S. Hoffmann, T. Ludwig, and V. Wulf (2021) Comparison of different types of augmented reality visualizations for instructions. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA. External Links: ISBN 9781450380966, Link, Document Cited by: §1, §5.3.1.
  • [13] B. Jones, R. Sodhi, M. Murdock, R. Mehra, H. Benko, A. Wilson, E. Ofek, B. MacIntyre, N. Raghuvanshi, and L. Shapira (2014) RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, New York, NY, USA, pp. 637–644. External Links: ISBN 9781450330695, Link, Document Cited by: §2.3.
  • [14] S. Li, S. Yoshida, K. Arihara, K. Nakashima, Y. Peng, H. Xie, T. Sato, and K. Miyata (2021) Skeleton-based interactive fabrication for large-scale newspaper sculpture. In 2021 Nicograph International (NicoInt), Vol. , pp. 74–81. External Links: Link, Document Cited by: §2.3.
  • [15] S. Luo, H. Xie, and K. Miyata (2021) Sketch-based anime hairstyle editing with generative inpainting. In 2021 Nicograph International (NicoInt), Vol. , pp. 7–14. External Links: Link, Document Cited by: §2.1.
  • [16] S. Noh, S. Hashimoto, D. Yamanaka, Y. Kamiyama, M. Inami, and T. Igarashi (2013-05) Design and enhancement of painting interface for room lights. The Visual Computer 30, pp. 467–478. External Links: Document Cited by: §2.2.
  • [17] Y. Peng, Y. Mishima, Y. Igarashi, R. Miyauchi, M. Okawa, H. Xie, and K. Miyata (2020) Sketch2Domino: interactive chain reaction design and guidance. In 2020 Nicograph International (NicoInt), Vol. , pp. 32–38. External Links: Link, Document Cited by: §2.2, §4.1.
  • [18] B. Piper, C. Ratti, and H. Ishii (2002) Illuminating clay: a 3-d tangible interface for landscape analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’02, New York, NY, USA, pp. 355–362. External Links: ISBN 1581134533, Link, Document Cited by: §2.3.
  • [19] T. Portenier, Q. Hu, A. Szabó, S. A. Bigdeli, P. Favaro, and M. Zwicker (2018) FaceShop: deep sketch-based face image editing. ACM Transactions on Graphics (TOG) 37 (4). External Links: Document Cited by: §2.1.
  • [20] A. Rivers, A. Adams, and F. Durand (2012-11) Sculpting by numbers. ACM Trans. Graph. 31 (6). External Links: ISSN 0730-0301, Link, Document Cited by: §2.3.
  • [21] D. Sakamoto, K. Honda, M. Inami, and T. Igarashi (2009) Sketch and run: a stroke-based interface for home robots. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, New York, NY, USA, pp. 197–200. External Links: ISBN 9781605582467, Link, Document Cited by: §2.2.
  • [22] Y. Sano, K. Sato, R. Shiraishi, and M. Otsuki (2016) Sports support system: augmented ball game for filling gap between player skill levels. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces, pp. 361–366. Cited by: §2.3.
  • [23] E. Smith, G. Semple, D. Evans, K. McRae, and P. Blackwell (2020) Augmented instructions: analysis of performance and efficiency of assembly tasks. In Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications, J. Y. C. Chen and G. Fragomeni (Eds.), Cham, pp. 166–177. Cited by: §1.
  • [24] H. Wang, H. Kanayama, Y. Peng, S. Yoshida, H. Xie, S. Okada, and K. Miyata (2021) Sketch2Bento: sketch-based arrangement guidance for lunch boxes. In 2021 Nicograph International (NicoInt), Vol. , pp. 114–114. External Links: Link, Document Cited by: §2.2.
  • [25] H. Xie, Y. Peng, N. Chen, D. Xie, C. Chang, and K. Miyata (2019) BalloonFAB: digital fabrication of large-scale balloon art. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA ’19, New York, NY, USA, pp. 1–6. External Links: ISBN 9781450359719, Link, Document Cited by: §2.3, §5.3.3.
  • [26] J. Xie, A. Hertzmann, W. Li, and H. Winnemöller (2014) PortraitSketch: face sketching assistance for novices. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST), New York, USA, pp. 407–417. External Links: Document Cited by: §2.1.
  • [27] J. Xing, R. H. Kazi, T. Grossman, L. Wei, J. Stam, and G. Fitzmaurice (2016) Energy-brushes: interactive tools for illustrating stylized elemental dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST ’16, New York, NY, USA, pp. 755–766. External Links: ISBN 9781450341899, Link, Document Cited by: §2.1.