Log In Sign Up

Sensitivity to Haptic-Audio Envelope Asynchrony

by   Alfonso Balandra, et al.

We want to understand the human capabilities to perceive amplitude similarities between a haptic and an audio signal. So, four psychophysical experiments were performed. Three of them measured the asynchrony JND (Just Noticeable Difference) at the signals' attack, release and decay, while the forth experiment measured the amplitude decrease on the middle of the signal. All the experiments used a combination of the constant stimulus and staircase methods to present two stimuli, while the participants' (N=12) task was to identify which of the two stimuli was synchronized. The audiotactile stimulus was defined using an stereo audio signal with an ADSR (Attack Decay Sustain Release) envelope. The partial results reveal JNDs for temporal asynchrony of: 54ms for attack, 265ms for decay and 57ms for release. Also the results reveal an amplitude decrease JND of 25%. Although for decay the results were to disperse, therefore we suspect that the participants were not able to the changes on the haptic signal.


page 1

page 2


Graphic relation between amplitude and sound intensity level

We present a simple experiment that allows us to demonstrate graphically...

Low-rankness of Complex-valued Spectrogram and Its Application to Phase-aware Audio Processing

Low-rankness of amplitude spectrograms has been effectively utilized in ...

A Berry-Esseen theorem for sample quantiles under association

In this paper, the uniformly asymptotic normality for sample quantiles o...

Gamma Boltzmann Machine for Simultaneously Modeling Linear- and Log-amplitude Spectra

In audio applications, one of the most important representations of audi...

The Effectiveness of Haptic PropertiesUnder Cognitive Load: An Exploratory Study

With the rise of wearables, haptic interfaces are increasingly favored t...

An Audio Envelope Generator Derived from Industrial Process Control

Audio envelopes serve a crucial role in ensuring the versatility of synt...

How well does surprisal explain N400 amplitude under different experimental conditions?

We investigate the extent to which word surprisal can be used to predict...

I Introduction

The effective use of redundant audio and tactile signals is relevant, not just for new audio-haptic academic proposals, but for commercial mobile devices and video games too. All these environments can take the best advantage of their own audio and haptic interfaces by defining an effective audio-tactile stimuli, which considers our multimodal perception capabilities and limitations. So it can be plausible to render and optimal audio-tactile sensation, even with restricted computational resources and/or limited haptic actuators. Therefore, it is important to perform psychophysical experiments oriented to understand the human multimodal audio-tactile capabilities, in order to define an effective method to represent an audio signal into haptics.

Additionally, it is logical to consider that an optimal representation of any kind of audio signal into a haptic vibration must be intrinsically similar the audio signal itself; similar in terms of our intrinsic perceptual mechanisms. But to this date, our multimodal perception capabilities are not clearly defined or understood yet, so it is important to find which specific properties of the signals have more impact on their perceived similarity. Therefore, this study is focus on identify which amplitude characteristics between and audio and tactile signals have more impact on their perceived similarity.

Ii Related Research

The closest studies are focused on the attack asynchrony perception of audio-tactile signals. Two of these studies [1] [2] measured the general asynchrony audio-tactile perception to be around 25ms. Also Altinsoy [2] specifically mentioned that an asynchrony is easier to detect when the auditory stimulus precedes the tactile stimulus. Both studies used record-play audio-tactile stimuli with identical waveform signals, but in none of these two studies the stimuli waveform was modified.

Fig. 1: The ADSR envelopes used on each experiment. A) Envelope for the attack async. experiment. B) Envelope for the decay async. experiment. C) Envelope for the release async. experiment, with an instantaneous volume drop. D) Envelope for the amplitude decrease experiment, with an instantaneous volume drop.
Fig. 2: The PSE and JND of each experiment, plotted over their respective CDF in color. Also, on the same plot, each participant’s CDF was plotted in gray. Only for the amplitude decrease experiment [D)] the x-axis indicated the drop percentage, from the signal maximum amplitude.

Iii Methods and Materials

Iii-a Apparatus

The audio-tactile stimulus was defined with single stereo audio signal, the left channel was for audio and the right one for haptics. This signal was generated using a desktop PC (Intel i7-3770S with 8Gb of RAM) and a external DAC (xDuoo XD-01). The control program was developed using real time audio libraries (CCRMA Stk [3]). Also, the same program was used to automatically adjust the signals latency and capture the participants responses. The audio signal (left channel) was directly displayed to the user in stereo using in-ear headphones (Sennheiser MX 475), while the haptic signal (right channel) was amplified (LP2020A) and then displayed on a surface transducer (Adafruit 5W 4Ohm). So, the latency between both signals was controlled without any inherited apparatus delay.

Iii-B Stimuli

The audio-tactile stimuli was generated using and ADSR envelope, by these means the attack, decay and release shape of the signals can be modified with precision. Due to the results reported by Altinsoy [2], only the haptic signal was delayed in accordance to each experiment. Since the perception of loudness and pressure (amplitude) depend on frequency in both senses, the frequencies of the audio and haptic signals were set on a the minimum amplitude threshold frequency. The audio signal frequency was 4kHz [5] while the haptic signal frequency was 250Hz [4]. For the first three experiments the inflexion points of the attack, decay and release were modified to introduce a delay on the haptic signal; in accordance to their respective phase. For the attack and decay phases, the shift of the attack and decay inflexion points created haptic signals with longer attack and decay slopes. For the decay experiment, the sustain value was always the 50% of the max amplitude value. For the release phase and the amplitude decrease experiments the transition between sound and silence was almost immediate (1ms). (see Figure 2). Also, the audio-tactile signal length was 500ms for the attack, decay and release experiments, while for the forth experiment –amplitude decrease JND– the audio-tactile signal length was 1 second, in order to give enough time to the participants to notice the amplitude variations.

Iii-C Procedure

The participants were asked to identify the stimulus with no delay, then the latency of the haptic signal was then adjusted in accordance to their responses. The participants () were graduate students and university staff, with 25 to 42 years old. The experiments were performed using a combination of the constant stimuli and the two-down, one-up adaptive staircase algorithm, with 10 reversals. Each participant repeated the same experiment two times, starting form minimum and maximum start latency respectively. The staircases’ maximum step, minimum step and start values are indicated in Table I.

Max step Min step Up value Down value
Exp#1 (attack) 50ms 5ms 150ms 5ms
Exp#2 (decay) 10ms 250ms 900ms 10ms
Exp#3 (release) 50ms 5ms 150ms 5ms
Exp#4 (amp) 15% 5% 90% 5%
TABLE I: Staircases’ initial conditions

Iv Results and Discussion

The standard deviation from all the up and down reversals was computed, for the two experiments of each phase. Also the middle point of each stair reversal was averaged, then the middle points and the average of all the up and down standard deviations were used to generate a normal curve. From the normal curve the CDF (Cumulative Distribution Function) is calculated for each participant and finally the PSE (Point of Subjective Equality at 0.5) and JND (at 0.75) are obtained from the averaged CDF. Naturally, our results on attack asynchrony JND differ form the already reported results

[2] [1], because in our case the attack was not instantaneous or constant. In our experiments the attack slope it self was longer, because the haptic attack inflexion point was delayed. And the same was done for the decay experiment. The results from the decay asynchrony are very disperse. This indicates that the participants had difficulties to precisely identify amplitude changes on the decay. Maybe because the continuous and gradual change of the vibrotactile stimuli caused an adaptation on the participants’ tactile mechanoreceptors. This lets us consider that the participants could not perceive small amplitude differences on decay. Finally, we want to re-define the decay phase experiment, so we can have concrete evidence, to suggests that amplitude variations at decay cannot be discriminated. Also, on a final full paper, we want to suggest specific and easy guidelines for the effective amplitude perception design of audio-tactile signals. So this guidelines can be applied on future multimodal interactive environments.


  • [1] B. D. Adelstein, D. R. Begault, M. R. Anderson, and E. M. Wenzel, “Sensitivity to haptic-audio asynchrony,” in Proceedings of the 5th international conference on Multimodal interfaces.   ACM, 2003, pp. 73–76.
  • [2] M. E. Altinsoy, “Perceptual aspects of auditory-tactile asynchrony,” in Proceedings of the Tenth International Congress on Sound and Vibration, 2003, pp. 3831–3838.
  • [3] P. R. Cook and G. P. Scavone, “Ccrma - the synthesis toolkit in c++ (stk),” 1995. [Online]. Available:
  • [4] G. A. Gescheider, “Vibrotactile intensity discrimination measured by three methods,” The Journal of the Acoustical Society of America, vol. 87, no. 1, pp. 330–338, 1990.
  • [5] Y. Suzuki and H. Takeshima, “Equal-loudness-level contours for pure tones,” The Journal of the Acoustical Society of America, vol. 116, no. 2, pp. 918–933, 2004.