Aerial haptic feedback is a popular topic in research fields such as real-world-oriented interaction, augmented reality (AR), and virtual reality (VR). Several methods have therefore been proposed to realize aerial haptic feedback which include phenomena such as magnetic forces, ultrasound, and air vortices.
An aerial haptic display has several advantages. First, it projects a force over a distance without physical contact or wearable devices. Second, it has a high programmability. In particular, it can be set and rearranged at an arbitrary position in a three-dimensional (3D) space, as it does not require physical actuators.
In this study, new aerial interactions were evaluated. The aim of the study was the development of a new aerial haptics system to express a wide range of feedback. The proposed system (Figure 1) combines aerodynamic and acoustic fields. The aerodynamic vortex  from the aerodynamic field and the focused ultrasound  from the acoustic field were used to develop the device.
The tactile sensations of single and multiple fields were then compared. Through a user study, it was found that the aerodynamic vortex and focused ultrasound do not influence each other. By combining different types of forces, the proposed system can display various textures. Based on the reports in the literature, this is an early study that combines multi-field physical quantities to render haptic textures.
2 Related Work
2.1 Aerial Haptics Feedback
Several methods have been proposed for aerial haptic feedback without physical contact or wearable devices. The technologies employed without wearable devices are based on aerodynamic vortices, focused ultrasound, laser induced plasma, and magnetic forces. These technologies have been applied to touch panels  and VR  systems .
Ultrasonic technology, which uses ultrasound, can provide tactile sensations in mid-air without the need for a user actuator  . The position of the focal point can be changed using a phased array transducer as it represents tactile behavior. The rendering of volumetric haptic shapes can also be achieve using focused ultrasound . MidAir  reflects a virtual image in the air and provides tactile feedback using an ultrasonic speaker based on the virtual image and finger location. HaptoClone  enables real-time interaction with floating volumetric images using haptic feedback. This method is insufficient, given that the focused ultrasound force is very weak and only limited focal points can be generated.
In , an air jet was used to produce contactless haptic feedback with a low accuracy. In , virtual objects were represented by air jets from an array of nozzles. Vortex rings  have also been used as a non-contact haptic feedback system, and air vortices have been used to produce impact in midair  . These previous approaches mainly used vortex rings to add multi-modal sensation to a conventional display. Ueoka et al. evaluated the manner in which people perceive haptic stimuli generated by air vortex rings and how the stimuli affect their emotional states when stressed. However, both these methods have a low fidelity due to the non-focusing stimulating area.
The tactile presentation of a magnetic field involves both direct and indirect presentation. For direct tactile feedback, a magnet can be placed on the finger  . The authors in , presented magnetic based sensing in addition to haptic feedback. Zhang et al.  and Berkelman et al.  rendered a 3D model in mid-air using an electromagnet array. In direct tactile presentation, powerful tactile feedback can be achieved without touching the screen.
2.2 Cross-Field Haptics
This study combines multiple haptics technologies, thus overcoming their individual drawbacks and improving the interaction width.
The wUbi-Pen  is a pencil-type tactile interface that consist of a vibrator, linear vibrator, speaker, and pin array. It provides functions such as feedback drag, drop, and movement. Minamizawa et al.  developed a device based on tactile presentation, which that combines one-point kinesthetic feedback and multipoint tactile feedback. The accuracy of the feedback was then improved by combining haptics technologies. Impacto  was designed to render the haptic sensation of hitting and being hit in the VR environment. They combined tactile stimulation with electrical muscle stimulation. Cross-Field Aerial Haptics  involves the drawing of a tactile interface in the air by combining ultrasonic waves and laser plasmas. Hashizume et al.  developed a touch type haptic device that combines magnetic and electrostatic fields. Their report also includes a description of their implementation method.
Cross-field haptics is not a widely studied field. Aerodynamic vortex and focused ultrasound are simultaneously utilized. Using both fields helps eliminate their drawbacks and it is intended to provide a wider tactile presentation. The aerodynamic vortex provides tactile sensations over large distances and forces. The focused ultrasound delivers distinguishable high-resolution tactile sensations.
3.1 Aerodynamic Haptics
An air vortex (Figure 3, right) is a ring of air that typically has a toroidal shape and can travel at high speeds over large distances. Vortex rings (Figure 2, upper) can be formed by pushing air using a piston through a circular aperture, or hole. The quality of the formed vortex is dependents on the volume of air pushed, the velocity of the piston, and the diameter of the aperture.
Gharib et al.  defined the stroke ratio as a ratio of the length of the theoretical cylindrical slug of air pushed out of the nozzle to aperture of diameter :
The stroke ratio characterizes the stability of the vortex as it exits the aperture, and it is used to define the formation number. A typical value for the formation number is within the range of 3.6-4.5 for several various vortex systems.
Thus, for stable vortices, the following is true:
3.2 Ultrasound Haptics
Ultrasonic haptics are based on acoustic radiation pressure, which exerts a force on the surface of the skin (Figure 2, lower). Ultrasonic haptics can be applied to the skin for a long time-period; however, they are relatively weak (10-20 mN). The sensation is similar to that of a laminar air flow within a narrow area.
The time delay for the -th transducer is given by:
where and are the distances from the focal point to the -th reference and -th transducers, respectively; and is the speed of sound in the air. The focal point can be moved by recalculation and the setting of the time delays for the next coordinates.
Haptic images are generated by an acoustic phased array system(Figure 3, middle). Haptic image Hi is the summation of the time series of the focal points:
where is the ultrasonic focal points, is the acoustic pressure, and is the time duration.
The amount of time required to produce a tactile presentation using an aerodynamic vortex is different from that required using the focused ultrasound. The focused ultrasound advances in the air at the speed of sound. Therefore, the focused ultrasound can achieve a near simultaneous tactile presentation with the generation of signals. The speed and delay of the aerodynamic vortex are greater than those of ultrasound. According to , the vortex speed is given by
where is the time required for the displaced air to move through the aperture. The average speed of the aerodynamic vortex device used in this study was . A delay of 30mms was applied to the focused ultrasound to ensure that the aerodynamic vortex and focused ultrasound reach the user simultaneously.
4 Experiments and Results
In this section, a discussion on the user experiments for the evaluation proposed haptic system is presented.
4.1 Experiment of generated force
The magnitude of the force generated by an ultrasonic wave and air cannon were considered. The precision electronic balance was set vertically and placed 15 cm from the ultrasonic device and air cannon (Figure 4). The mass displayed on the precision electronic balance was then converted to force. The air cannon was output at 30 Hz, and the power supply voltage was varied from 5 to 17.5 V in increments of 2.5 V. In addition the change in the magnitude of the generated force was examined. The ultrasonic wave was output at a modulation frequency of 50 Hz, and the output intensity was changed from 0 to 600 in increments of 100. Moreover, the change in the magnitude of the generated force was investigated. To convert the output intensity to the output force, a conversion could be carried out using . The ultrasonic focal length was set as 15 cm.
Figure 5 present the results, in which both the ultrasonic wave and air cannon increase the force generated in proportion to the output intensity.
4.2 Experiment of double-point threshold
The user study was conducted to investigate spatial resolution. The double-point threshold for acoustic radiation pressure induced by focused ultrasound and air vortex pressure was evaluated. Five people participated in the user study (20.2 years old on average, with one female and four males). The participants were isolated from visual information using blindfolds. Participants then placed their hands on a table positioned 15 cm away from the haptic device(Figure 7 right). The platform could be moved with an accuracy of 0.1 mm, as participants were told not to move their hands. The output force was set with reference to Figure 5.
The method of limits was used to measure the double-point threshold. First, the standard stimulus was applied to the palm of the hand of the participant. The standard stimulus is a 10 s ultrasound wave and five air vortex cycles. After administering the standard stimulus, the platform was moved and the comparative stimulus was administered. After administering the comparative stimulus, each participant gave one of the following answers: ”(1) the two points are divided,” (2) ” the two points not are divided,” or (3) ”I do not know.” The distance between the standard and the comparative stimuli was approximated until the participant answered ”the two points are not divided” or ”I do not know” (descending series). The distance between the standard and comparative stimuli was then increased until the participant answered with ”the two points are divided” (ascending series). The test for the descending and ascending series were carried out twice.
The experiments were conducted when only ultrasonic waves were applied, when only an air cannon was used, when an air cannon was used with a constant ultrasonic wave, and when an ultrasonic wave was applied with a constant air vortex. The ultrasonic waves were generated at modulation frequencies of 50 Hz and 200 Hz. The focal length of the ultrasonic device was set as 15 cm, and the output force was set as 5.73 mN. The participants were stimulated with ultrasound for 5 s. The output force of the air cannon was set as 7.67 mN, and the stimulation was applied five times. To provide a constant air vortex, the air cannon was implemented at 15 Hz.
Results : The results are shown in Figure 6. In the case of only ultrasonic waves (Figure 6 (a) and (b)), the double-point threshold was approximately 6 mm regardless of the modulation frequency. The double-point threshold for an air cannon only (Figure 6 (c)) was 11 mm. The double-point threshold of an air cannon while an ultrasonic wave was constantly provided (Figure 6 (d) and (e)) was not much different from that of an air cannon only. The double-point threshold of the ultrasonic wave while an air vortex was constantly provided (Figure 6 (f) and (g)) was approximately 3 mm larger that of the ultrasonic only. In the case in which the air cannon was affected, the variation of the double-point threshold by the participant was large.
4.3 Experiment of perceptual threshold
The user study was conducted to evaluate the perceptual threshold for acoustic radiation pressure induced by the focused ultrasound and air the pressure vortex. Seven people participated in the user study (19.6 years old on average with two females and five males). The participants were isolated from visual information using blindfolds, and auditory information was eliminated using headphones with white noise (Figure7, left). Participants placed their hands on a table positioned 15 cm away from the haptic device(Figure 7 right). The output force was set with reference to Figure 5.
Focused ultrasound : The focused ultrasound haptic stimulation was applied to the right palm of each participant. Moreover, vibrotactile stimulation modulated by 200-Hz and 50-Hz rectangular waves was applied. The output force was set as one of six values (min:0.70 mN, max:10.9 mN) near the thresholds. The ultrasound output time was 200ms. Each force condition was applied once (i.e., one trial) and the number of trials was 10 per participant. The order of trials was randomized. In each trial, the participants were asked whether they perceived stimuli on their forefingers.
Aerodynamic vortex : The aerodynamic vortex haptic stimulation was applied to the right palm of each participant. The output force was set to one of six values (min:0.66 mN, max:13.7 mN) near the thresholds. Each force condition was applied once (i.e., one trial) and the number of trials was 10 per participant. The order of trials was randomized. In each trial, the participants were asked whether they perceived the stimuli on their forefingers.
Cross-field : Three types of user studies were conducted on the cross-field. First, focused ultrasound and an aerodynamic vortex tactile stimulus were applied simultaneously. The output force of the air cannon was set as 7.67 mN. The vibrotactile stimulation of ultrasound was set as 200 Hz and 50 Hz. Second, the air vortex perceptual threshold in space with a constant ultrasound was evaluated. The output force of the ultrasound was maintained at 9.7 mN. The output force of the air vortex was set to one of six values (min:0.66 mN, max:13.7 mN). Third, the focused ultrasound perceptual threshold in space with a constant aerodynamic vortex was evaluated. The output force of the air vortex was set as 7.67 mN and the air cannon was actuated to 20Hz. The output force of ultrasound was set as one of six values (min:0.70 mN, max:10.9 mN) and the vibrotactile stimulation was modulated to 200 Hz and 50 Hz.
|The ultrasound||The perceptual|
|vibrotactile stimulation||threshold of haptic|
Results : The results are presented in Figure 8 and Table 1. The perception rate on the vertical axis indicates the ease of tactile sensation. When the perception threshold was 100%, the examinee could perceive the tactile sensation.
In the case wherein only the ultrasound was applied, the perception threshold was nearly 100% when the output exceeded 4 mN. In the case wherein only the air cannon was used, the perception threshold increased as the voltage increased. In addition, when the output force exceeded 11 mN, the participants could perceive feel the tactile sensation. When the air cannon was operated with an ultrasonic wave, sensing when the output force reached 2 mN was difficult. However, when the output force was higher than 2 mN, the results were the same as that wherein only the air cannon was used. When ultrasonic waves were applied with an air cannon, the perception threshold was generally low compared with the case wherein only ultrasonic waves was applied. In particular, when the modulation frequency was 50 Hz, the perception threshold was less than 20 %. When presented simultaneously, participants could recognize a perception threshold of 95 % or greater.
5 Discussion and Conclusion
From the experiments conducted on double-point thresholds, the double-point threshold was found to increase the when air cannon and ultrasonic wave were combined. In addition, from the experiments on perceptual thresholds, the air cannon was easy to perceive in the ultrasonic presentation state; however, the ultrasonic tactile stimulus was difficult to perceive in the air cannon presentation state. This is because the magnitude of the force that the air cannon can apply is larger than the ultrasonic tactile stimuli. It is necessary to appropriately adjust the magnitude of the applied force. However, when the tactile stimulus was simultaneously applied, neither tactile sense could be perceived. When using Sonovortex, it is effective to present the air cannon tactile sense in the ultrasonic presentation state, or to simultaneously present the air cannon and the ultrasonic.
In this study, a method was developed that combines multiple tactile technologies. This method generates a tactile sensation using an ultrasonic device and air cannon. In addition, the ranges of possible resolutions and thresholds were discussed. Cross-field helps eliminate the drawbacks of each field and provide a wider tactile presentation. An aerodynamic vortex provides tactile sensations over large distances and with considerable force levels. The focused ultrasound delivers distinguishable high-resolution tactile sensations.
However, there are still several drawbacks. Both an air cannon and ultrasonic device generate environmental noise. In particular, given that the air cannon produces a loud sound, using it in a quiet space such as a hospital or company office is difficult. However, this does not present a problem in noisy spaces such as shops and towns. Moreover, to use Sonovortex, an air cannon and phased array must be deferred. For wearing, Sonovortex is heavy and large. However, Sonovortex is expected to be incorporated into the environment and used for digital signage and amusement.
We would like to thank University of Tsukuba for supporting this work. We are also thankful to all the members of the Digital Nature Group at University of Tsukuba for their discussions and feedback.
-  (2012-03) Co-located 3d graphic and haptic display using electromagnetic levitation. In 2012 IEEE Haptics Symposium (HAPTICS), pp. 77–81. External Links: Cited by: §2.1.
-  (2013) UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST ’13, New York, NY, USA, pp. 505–514. External Links: Cited by: §2.1.
-  (1987) Reliability of two-point discrimination measurements. The Journal of hand surgery 12 (5), pp. 693–696. Cited by: §4.2.
-  (1988) The formation of vortex rings. Physics of Fluids 31 (12), pp. 3532–3542. External Links: Cited by: §3.1.
-  (2013) AirWave: non-contact haptic feedback using air vortex rings. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’13, New York, NY, USA, pp. 419–428. External Links: Cited by: §2.1, §3.3.
-  (2017-06) Cross-field haptics: multiple direction haptics combined with magnetic and electrostatic fields. In 2017 IEEE World Haptics Conference (WHC), Vol. , pp. 370–375. External Links: Cited by: §2.2.
-  (2010-07) Noncontact tactile display based on radiation pressure of airborne ultrasound. IEEE Transactions on Haptics 3 (3), pp. 155–165. External Links: Cited by: §1, §2.1, §4.1.
-  (2013) Magnetic field based near surface haptic and pointing interface. In Human-Computer Interaction. Interaction Modalities and Techniques: 15th International Conference, HCI International 2013, Las Vegas, NV, USA, July 21-26, 2013, Proceedings, Part IV, M. Kurosu (Ed.), pp. 601–609. External Links: Cited by: §2.1.
-  (2008) WUbi-pen: windows graphical user interface interacting with haptic feedback stylus. In ACM SIGGRAPH 2008 New Tech Demos, SIGGRAPH ’08, New York, NY, USA, pp. 42:1–42:4. External Links: Cited by: §2.2.
-  (1992) Vortex rings. Annual Review of Fluid Mechanics 24 (1), pp. 235–279. External Links: Cited by: §3.1.
-  (2014-11) Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans. Graph. 33 (6), pp. 181:1–181:10. External Links: Cited by: §2.1.
-  (2015) Impacto: simulating physical impact by combining tactile stimulation with electrical muscle stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, UIST ’15, New York, NY, USA, pp. 11–19. External Links: Cited by: §2.2.
-  (2016) HaptoClone (haptic-optical clone) for mutual tele-environment by real-time 3d image transfer with midair force feedback. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, New York, NY, USA, pp. 1980–1990. External Links: Cited by: §2.1.
-  (2010-03) Simplified design of haptic display by extending one-point kinesthetic feedback to multipoint tactile feedback. In 2010 IEEE Haptics Symposium, pp. 257–260. External Links: Cited by: §2.2.
-  (2014) HaptoMime: mid-air haptic interaction with a floating virtual screen. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, New York, NY, USA, pp. 663–667. External Links: Cited by: §2.1, §2.1.
-  (2016) Cross-field aerial haptics: rendering haptic feedback in air with light and acoustic fields. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, New York, NY, USA, pp. 3238–3247. External Links: Cited by: §2.2.
-  (2016-02) Fairy lights in femtoseconds: aerial and volumetric graphics rendered by focused femtosecond laser combined with computational holographic fields. ACM Trans. Graph. 35 (2), pp. 17:1–17:14. External Links: Cited by: §2.1.
-  (2015) HeatHapt thermal radiation-based haptic display. In Haptic Interaction: Perception, Devices and Applications, H. Kajimoto, H. Ando, and K. Kyung (Eds.), pp. 105–107. External Links: Cited by: §2.1.
-  (2013-07) AIREAL: interactive tactile experiences in free air. ACM Trans. Graph. 32 (4), pp. 134:1–134:10. External Links: Cited by: §2.1, §3.1, §3.3.
-  (2002) Design of force feedback utilizing air pressure toward untethered human interface. In CHI ’02 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’02, New York, NY, USA, pp. 808–809. External Links: Cited by: §2.1.
-  (2005-01) Air jet driven force feedback in virtual reality. IEEE Comput. Graph. Appl. 25 (1), pp. 44–47. External Links: Cited by: §2.1, §2.1.
-  (2016) Interactive cheek haptic display with air vortex rings for stress modification. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’16, New York, NY, USA, pp. 1766–1771. External Links: Cited by: §2.1.
-  (1997) On the evolution of laminar vortex rings. Experiments in Fluids 22 (6), pp. 447–457. External Links: Cited by: §1, §2.1, §3.1.
-  (2011) FingerFlux: near-surface haptic feedback on tabletops. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, New York, NY, USA, pp. 615–620. External Links: Cited by: §2.1.
-  (2016) Magnetic field control for haptic display: system design and simulation. IEEE Access 4, pp. 299–311. External Links: Cited by: §2.1.