I Introduction
The effective use of redundant audio and tactile signals is relevant, not just for new audio-haptic academic proposals, but for commercial mobile devices and video games too. All these environments can take the best advantage of their own audio and haptic interfaces by defining an effective audio-tactile stimuli, which considers our multimodal perception capabilities and limitations. So it can be plausible to render and optimal audio-tactile sensation, even with restricted computational resources and/or limited haptic actuators. Therefore, it is important to perform psychophysical experiments oriented to understand the human multimodal audio-tactile capabilities, in order to define an effective method to represent an audio signal into haptics.
Additionally, it is logical to consider that an optimal representation of any kind of audio signal into a haptic vibration must be intrinsically similar the audio signal itself; similar in terms of our intrinsic perceptual mechanisms. But to this date, our multimodal perception capabilities are not clearly defined or understood yet, so it is important to find which specific properties of the signals have more impact on their perceived similarity. Therefore, this study is focus on identify which amplitude characteristics between and audio and tactile signals have more impact on their perceived similarity.
Ii Related Research
The closest studies are focused on the attack asynchrony perception of audio-tactile signals. Two of these studies [1] [2] measured the general asynchrony audio-tactile perception to be around 25ms. Also Altinsoy [2] specifically mentioned that an asynchrony is easier to detect when the auditory stimulus precedes the tactile stimulus. Both studies used record-play audio-tactile stimuli with identical waveform signals, but in none of these two studies the stimuli waveform was modified.


Iii Methods and Materials
Iii-a Apparatus
The audio-tactile stimulus was defined with single stereo audio signal, the left channel was for audio and the right one for haptics. This signal was generated using a desktop PC (Intel i7-3770S with 8Gb of RAM) and a external DAC (xDuoo XD-01). The control program was developed using real time audio libraries (CCRMA Stk [3]). Also, the same program was used to automatically adjust the signals latency and capture the participants responses. The audio signal (left channel) was directly displayed to the user in stereo using in-ear headphones (Sennheiser MX 475), while the haptic signal (right channel) was amplified (LP2020A) and then displayed on a surface transducer (Adafruit 5W 4Ohm). So, the latency between both signals was controlled without any inherited apparatus delay.
Iii-B Stimuli
The audio-tactile stimuli was generated using and ADSR envelope, by these means the attack, decay and release shape of the signals can be modified with precision. Due to the results reported by Altinsoy [2], only the haptic signal was delayed in accordance to each experiment. Since the perception of loudness and pressure (amplitude) depend on frequency in both senses, the frequencies of the audio and haptic signals were set on a the minimum amplitude threshold frequency. The audio signal frequency was 4kHz [5] while the haptic signal frequency was 250Hz [4]. For the first three experiments the inflexion points of the attack, decay and release were modified to introduce a delay on the haptic signal; in accordance to their respective phase. For the attack and decay phases, the shift of the attack and decay inflexion points created haptic signals with longer attack and decay slopes. For the decay experiment, the sustain value was always the 50% of the max amplitude value. For the release phase and the amplitude decrease experiments the transition between sound and silence was almost immediate (1ms). (see Figure 2). Also, the audio-tactile signal length was 500ms for the attack, decay and release experiments, while for the forth experiment –amplitude decrease JND– the audio-tactile signal length was 1 second, in order to give enough time to the participants to notice the amplitude variations.
Iii-C Procedure
The participants were asked to identify the stimulus with no delay, then the latency of the haptic signal was then adjusted in accordance to their responses. The participants () were graduate students and university staff, with 25 to 42 years old. The experiments were performed using a combination of the constant stimuli and the two-down, one-up adaptive staircase algorithm, with 10 reversals. Each participant repeated the same experiment two times, starting form minimum and maximum start latency respectively. The staircases’ maximum step, minimum step and start values are indicated in Table I.
Max step | Min step | Up value | Down value | |
---|---|---|---|---|
Exp#1 (attack) | 50ms | 5ms | 150ms | 5ms |
Exp#2 (decay) | 10ms | 250ms | 900ms | 10ms |
Exp#3 (release) | 50ms | 5ms | 150ms | 5ms |
Exp#4 (amp) | 15% | 5% | 90% | 5% |
Iv Results and Discussion
The standard deviation from all the up and down reversals was computed, for the two experiments of each phase. Also the middle point of each stair reversal was averaged, then the middle points and the average of all the up and down standard deviations were used to generate a normal curve. From the normal curve the CDF (Cumulative Distribution Function) is calculated for each participant and finally the PSE (Point of Subjective Equality at 0.5) and JND (at 0.75) are obtained from the averaged CDF. Naturally, our results on attack asynchrony JND differ form the already reported results
[2] [1], because in our case the attack was not instantaneous or constant. In our experiments the attack slope it self was longer, because the haptic attack inflexion point was delayed. And the same was done for the decay experiment. The results from the decay asynchrony are very disperse. This indicates that the participants had difficulties to precisely identify amplitude changes on the decay. Maybe because the continuous and gradual change of the vibrotactile stimuli caused an adaptation on the participants’ tactile mechanoreceptors. This lets us consider that the participants could not perceive small amplitude differences on decay. Finally, we want to re-define the decay phase experiment, so we can have concrete evidence, to suggests that amplitude variations at decay cannot be discriminated. Also, on a final full paper, we want to suggest specific and easy guidelines for the effective amplitude perception design of audio-tactile signals. So this guidelines can be applied on future multimodal interactive environments.References
- [1] B. D. Adelstein, D. R. Begault, M. R. Anderson, and E. M. Wenzel, “Sensitivity to haptic-audio asynchrony,” in Proceedings of the 5th international conference on Multimodal interfaces. ACM, 2003, pp. 73–76.
- [2] M. E. Altinsoy, “Perceptual aspects of auditory-tactile asynchrony,” in Proceedings of the Tenth International Congress on Sound and Vibration, 2003, pp. 3831–3838.
- [3] P. R. Cook and G. P. Scavone, “Ccrma - the synthesis toolkit in c++ (stk),” 1995. [Online]. Available: https://ccrma.stanford.edu/software/stk/
- [4] G. A. Gescheider, “Vibrotactile intensity discrimination measured by three methods,” The Journal of the Acoustical Society of America, vol. 87, no. 1, pp. 330–338, 1990.
- [5] Y. Suzuki and H. Takeshima, “Equal-loudness-level contours for pure tones,” The Journal of the Acoustical Society of America, vol. 116, no. 2, pp. 918–933, 2004.