A proto-object based audiovisual saliency map

1 Introduction

Scientists and engineers have traditionally separated the analysis of a multisensory scene into its constituent sensory domains. In this approach, for example, all auditory events are processed separately and independently of visual and/or somatosensory streams even though the same multisensory event might have created those constituent streams. It was previously necessary to compartmentalize the analysis because of the sheer enormity of information as well as the limitations of experimental techniques and computational resources. With recent advances, it is now possible to perform integrated analysis of sensory systems including interactions within and across sensory modalities. Such efforts are becoming increasingly common in cellular neurophysiology, imaging and psychophysics studies Stein and Stanford (2008); Stevenson and James (2009); Calvert et al. (2004); Spence et al. (2009); Alais and Burr (2004).

Recent evidence from neuroscience Ghazanfar and Schroeder (2006); Stein and Stanford (2008) suggests that the traditional view that the low level areas of cortex are strictly unisensory, processing sensory information independently, which is later on merged in higher level associative areas is increasingly becoming obsolete. This has been proved by many fMRI Von Kriegstein et al. (2005); Watkins et al. (2006), EEG van Wassenhove et al. (2005) and neuro-physiological experiments Ghazanfar et al. (2005); Wang et al. (2008) at various neural population scales. There is now enough evidence to suggest an interplay of connections between thalamus, primary sensory and higher level association areas which are responsible for audiovisual integration. The broader implications of these biological findings may be that learning, memory and intelligence are tightly associated with the multi-sensory nature of the world.

Hence, incorporating this knowledge in computational algorithms can lead to better scene understanding and object recognition for which there is a great need. Moreover, combining visual and auditory information to associate visual objects with their sounds can lead to better understanding of events. For example, discerning whether the bat hit the baseball during a swing of the bat, tracking objects under severe occlusions, poor lighting conditions

etc can be more accurately performed only when we take audio and visual counterparts together. The applications of such technologies are numerous and in varied fields.

In summary, a better understanding of interaction, information integration, and complementarity of information across senses may help us build many intelligent algorithms for scene analysis, object detection and recognition, human activity and gait detection, elder/child care and monitoring, surveillance, robotic navigation, biometrics etc,

with better performance, stability and robustness to noise. In one application, for example, fusing auditory (voice) and visual (face) features improved the performance of speaker identification and face recognition systems

Çetingül et al. (2006); Tamura et al. (2005). Hence, our objective in this study is to develop a scene analysis algorithm using multisensory information, specifically vision and audio. We develop a purely bottom-up, proto-object based audiovisual saliency map (AVSM) for the analysis of dynamic natural scenes.

Building on the work of Russell et al. (2014), we add visual motion (Section 3.1.1) as another independent feature type along with color, intensity and orientation, all of which undergo a grouping process (Section 3.4) to form proto-objects of each feature type. In the auditory domain, we consider the location and intensity of sound as the only proto-objects as these are found to be most influential in drawing the spatial attention of an observer in many psycho-physics studies. Various methods of combination of the auditory and visual proto-object features are considered (Section 3.6). We demonstrate the efficacy of the AVSM in predicting salient locations in the audiovisual environments by testing it on real world AV data collected from a specialized hardware (Section 4) that can collect $360^{0}$ audio and video that are temporally synchronized and spatially co-registered. The AVSM captures nearly all visual, auditory and audio-visually salient events, just as any human observer would notice in that environment.

2 Related Work

The study of multi-sensory integration Alais et al. (2010); Calvert et al. (2004); Ghazanfar and Schroeder (2006), specifically audio-visual integration Meredith and Stein (1986); Spence et al. (2009) has been an active area of research in neuroscience, psychology and cognitive science. In the computer science and engineering fields, there is an increased interest in the recent times Evangelopoulos et al. (2008); Song (2013); Ramenahalli et al. (2013). For a detailed review of neuroscience and psychophysics research related to audio-visual interaction, please refer to Alais et al. (2010); Calvert et al. (2004). Here, we restrict our review to models of perception in audio-visual environments and some application oriented research using audio and video information.

In one of the earliest works Grossberg et al. (1997)

, a one-dimensional computational neural model of saccadic eye movement control by Superior Colliculus (SC) is investigated. The model can generate three different types of saccades: visual, multimodal and planned. It takes into account different coordinate transformations between retinotopic and head-centered coordinate systems, and the model is able to elicit multimodal enhancement and depression that is typically observed in SC neurons

Meredith and Stein (1996); Meredith et al. (1987). However, the main focus is on Superior Colliculus function rather than studying audio-visual interaction from a salience perspective. A detailed model of the SC is presented in Casey et al. (2012) with the aim of localizing audio-visual stimuli in real time. The model consists of 12,240 topographically organized neurons, which are hierarchically arranged into 9 feature maps. The receptive field of these neurons, which are fully connected to their input, are obtained through competitive learning. Intra-aural level differences are used to model auditory localization, while simple spatial and temporal differencing is used to model visual activity. A spiking neuron model Huo and Murray (2009) of audio-visual integration in barn owl uses Spike Timing Dependent Plasticity (STDP) to modulate activity dependent axon development, which is responsible for aligning visual and auditory localization maps. A neuromorphic implementation of the same using digital and analog mixed Very Large Scale Integration (mixed VLSI) can be found in Huo et al. (2012).

In another neural model Anastasio et al. (2000); Patton et al. (2002)

the visual and auditory neural inputs to the deep SC neuron are modeled as Poisson random variables. Their hypothesis is that the response of SC neurons is proportional to the presence of an audio-visual object/event in that spatial location which is conveyed to topographically arranged deep SC neurons via auditory and visual modalities. The model is able to elicit all properties of the SC neurons. An information theoretic explanation of super-additivity and other phenomena is given in a

Patton et al. (2002). They also show that addition of a cue from another sensory modality increases the certainty of a target’s location only if the input from initial modality/ies cannot reduce the uncertainty about target. Similar models are proposed in Patton and Anastasio (2003) and in Colonius and Diederich (2004), where the problem is formulated based on Bayes likelihood ratio. An important work Ma et al. (2006)

based on Bayesian inference explains a variety of cue combination phenomena including audio-visual spatial location estimation. According to the model, neuronal populations encode stimulus information using probabilistic population codes (PPCs) which represent probability distributions of stimulus properties of any arbitrary distribution and shape. They argue that neural populations approximate the Bayes rule using simple linear combination of neuronal population activities.

In Wilson et al. (2002), audiovisual arrays for untethered spoken interfaces are developed. The arrays localize the direction and distance of an auditory source from the microphone array, visually localize the auditory source, and then direct the microphone beamformer to track the speaker audio-visually. The method is robust to varying illumination and reverberation, and the authors report increased speech recognition accuracy using the AV array compared to non-array based processing.

In Torres and Kalva (2014) the authors found that emotional saliency conveyed through audio, drags an observer’s attention to the corresponding visual object, hence people often fail to notice any visual artifacts present in the video, suggest to exploit this property in intelligent video compression. For the same goal authors of Lee et al. (2011) implement an efficient video coding algorithm based on the audio-visual focus of attention where sound source is identified from the correlation between audio and visual motion information. The same premise that audio-visual events draw an observer’s attention is the basis for their formulation. A similar approach is applied to High Definition video compression in Rerabek et al. (2014). In these studies, spatial direction of sound was not considered, instead stereo or mono audio track accompanying the video was used in all computational and experimental work.

In Ruesch et al. (2008), a multimodal bottom-up attentional system consisting of a combined audio-visual salience map and selective attention mechanism is implemented for the humanoid robot iCub. The visual salience map is computed from color, intensity, orientation and motion maps. The auditory salience map consists of the location of the sound source. Both are registered in ego-centric coordinates. The audio-visual salience map is constructed by performing a pointwise max operation on visual and auditory maps. In an extension to multi-camera setting Schauerte et al. (2009), the 2D saliency maps are projected into a 3D space using ray tracing and combined as a fuzzy aggregations of salience spaces. In Schauerte et al. (2011); Schauerte (2016), after computing the audio and visual saliency maps, each salient event/proto-object is parameterized by salience value, cluster center (mean location), and covariance matrix (uncertainty in estimating location). The maps are linearly combined based on Onat et al. (2007). Extensions of this approach can be found in Kühn et al. (2012a). A work related to Kühn et al. (2012a) is presented in Kühn et al. (2012b)

where weighted linear combination of proto-object representations obtained using mean-shift clustering is detailed. Even though the method uses linear combination, the authors do not use motion information in computing the visual saliency map. A Self Organizing Map (SOM) based model of audio-visual integration was presented in

Bauer et al. (2012) in which the transformations between sensory modalities, and the respective sensory reliabilities are learned in an unsupervisory manner. A system to detect and track a speaker using a multi-modal, audio-visual sensor set that fuses visual and auditory evidence about the presence of a speaker using Bayes network was presented in Viciana-Abad et al. (2014). In a series of papers Evangelopoulos et al. (2008, 2008); Rapantzikos et al. (2007) audio-visual saliency is computed as a linear mixture of visual and auditory saliency maps for the purpose of movie summarization and key frame detection. No spatial information about audio is considered. The algorithm performs well in summarizing the videos for informativeness and enjoyability for movie clips of various genres. An extension of these models incorporating text Saliency can be found in Evangelopoulos et al. (2013). By assuming a single moving sound source in the scene, audio was incorporated into the visual saliency map in Nakajima et al. (2014) where sound location was associated with the visual object by correlating sound properties with the motion signal. By computing Bayesian surprise as in Itti and Baldi (2008), the authors in Nakajima et al. (2015)

present a visual attention model driven by auditory cues, where surprising auditory events are used to select synchronized visual features and emphasize them in a audio-visual surprise map. A real-time multi-modal home entertainment system

Korchagin et al. (2011)

performing a Just-In-Time association of features related to a person from audio and video are fused based on the shortest distance between each of the faces (in video) and the audio direction vector. In an intuitive study

Hershey and Movellan (2000) speaker localization by measuring the audio-visual synchrony in terms of mutual information between auditory features and pixel intensity change is considered. In a single active speaker scenario, they obtain good preliminary results. No microphone arrays are used for the localization task. In Blauth et al. (2012) visually detected face location is used to improve the speaker localization using a microphone array. A fast audiovisual attention model for human detection and localization is proposed in Ratajczak et al. (2016).

The effect of sound on gaze behavior in videos was studied in Song (2013); Song et al. (2012) where a preliminary computational model to predict eye movements was proposed. They use motion information to detect sound source. High level features such as face are hand labeled. A comparison of eye movements during visual only and audio-visual conditions with their model shows that adding sound information improved predictive power of their model. The role of salience, faces and sound in directing the attention of human observers (measured by gaze tracking) was studied with psychophysics experiments and computational modeling in Coutrot and Guyader (2014a) and an audiovisual attention model for natural conversation scenes was proposed in Coutrot and Guyader (2014b), where they use a speaker diarization algorithm to compute saliency¹¹1 Speaker diarization deals with the segmentation of speech into non-overlapping homogeneous regions separated by silence and assigning each of the segmented speech bits to unique speakers. Even though multi-modal speaker diarization methods perform audio-visual integration, for example in Noulas et al. (2012)

they combine audio and visual information using an Expectation Maximization (EM) algorithm in a Dynamic Bayesian Network (DBN) framework, the application is specific to speech and cannot generalize to a saliency map, hence not reviewed. A review of these methods can be found in

Miro et al. (2012).. Even though their study is restricted to conversation of humans and not applicable any generic audio-visual scene, hence cannot be regarded as a generalized model of audio-visual saliency, some interesting results are shown. Using EM algorithm to determine the individual contributions of bottom-up salience, faces and sound in gaze prediction, they show adding original speech to video improves gaze predictability, whereas adding irrelevant speech or unrelated natural sounds has no effect. By using speaker diarization algorithm Coutrot and Guyader (2014b) when the weight for active speakers was increased, their audio-visual attention model significantly outperformed the visual saliency model with equal weights for all faces. An audio-visual saliency map is developed in Sidaty et al. (2014) where features such as color, intensity, orientation, faces, speech are linearly combined with unequal weights to give different types of saliency maps depending on the presence/absence of faces and/or speech. It is not clear as to whether location of the sound was used in their approach. Plus, they do not factor in motion, which is an important feature while designing a saliency map for moving pictures.

3 Description of the model

The computation of audiovisual saliency map is similar to the computation of proto-object based visual saliency map for static images explained in Russell et al. (2014), except for (i) the addition of two new feature channels, the visual motion channel and the auditory loudness and location channel; and (ii) different ways of combining the conspicuity maps to get the final saliency maps. Hence, whereever the computation is identical to Russell et al. (2014), we will only give a gist of that computation to avoid repetition and detailed explanation otherwise.

The AVSM is computed by grouping auditory and visual bottom-up features at various scales, then normalizing the grouped features within and across scales, followed by merging features across scales and linear combination of the resulting feature conspicuity maps (Figure 1). The features are derived from the color video and multi-channel audio input (Section 4) without any top-down attentional biases, hence the computation is purely bottom-up. And the mechanism of grouping “binds” features within a channel into candidate objects or “proto-objects”. Approximate size and location are the only properties of objects that the grouping mechanism estimates, hence they are termed “proto-objects”. Such proto-objects, many of them, form simultaneously and dissolve rapidly Rensink (2000) in a purely bottom-up manner. Top-down attention is required to hold them together into coherent objects.

Also, computation of AVSM is completely feed-forward. Many spatial scales are used to achieve scale invariance. First the independent feature maps are computed, features within each channel are grouped into proto-objects. Such proto-object feature pyramids at various scales are normalized within and across scales. Such feature pyramids are merged across scales followed by normalization across feature channels to give rise to conspicuity maps. The conspicuity maps are linearly combined to get the final AVSM. Each of these steps is explained in more detail below.

Figure 1: Computation of proto-object based AVSM. (a) Grouping mechanism: At each scale, feature maps are filtered with center-surround cells, normalized, followed by border ownership (BO) computation. Grouping cells, at each scale, receive a higher feed-forward input from BO cells if they are consistent with Gestalt properties of convexity, proximity and surroundedness. Hence, grouping gathers conspicuous BO activity of features at object centers. Each feature channel undergoes same computation at various scales (except Orientation, see explanation), only Intensity channel shown in Figure 1(a). (b) Audivisual saliency computation mechanism: Five feature types are considered: Color, Orientation, Intensity and Motion in the visual domain; spatial location and loudness estimate in the auditory domain. All features undergo grouping as explained in (a) to obtain proto-object pyramids. The proto-object pyramids are normalized, collapsed to get the feature specific conspicuity maps such that isolated strong activity is accentuated and distributed weak activity is suppressed. The conspicuity maps are then combined in different ways as explained in Section 3.6 to get different saliency maps. Figure. 5 of Russell et al. (2014) modified with permission.

3.1 Computation of feature channels

We consider color, intensity, orientation and motion as separate, independent feature channels in the visual domain. Loudness and spatial location as features in the auditory space. The audio-visual camera equipment used for data gathering (See Section 4) guarantees spatial and temporal concurrency of audio and video.

A single intensity channel, where intensity is computed as the average of Red, Green and Blue color channels is used. Four feature sub-channels for angle, $θ = {0, \frac{π}{4}, \frac{π}{2}, \frac{3 π}{4}}$ are used for Orientation channel. Four color opponency feature sub-channels: Red-Green ( $R G$ ), Green-Red ( $G R$ ), Blue-Yellow ( $B Y$ ) and Yellow-Blue ( $Y B$ ) are used for color channel. The computation of color, orientation and intensity feature channels is identical to Russell et al. (2014).

Visual motion, auditory loudness and location estimate are the newly added features, the computation of which is explained below.

3.1.1 Visual motion channel

Motion is computed using the optical flow algorithm described in Sun et al. (2010) and the corresponding code available at Sun et al. (2008). Consider two successive video frames, $I (x, y, t)$ and $I (x, y, t + 1)$ . If the underlying object has moved between $t$ and $t + 1$ , then the intensity at pixel location, $(x, y)$ at time, $t$ should be the same in a nearby pixel location at $(x + Δ x, y + Δ y)$ in the successive frame at $t + 1$ . Using this as one of the constraints, the flow is estimated which gives the horizontal and vertical velocity components, $u (x, y, t)$ and $v (x, y, t)$ respectively, at each pixel location $(x, y)$ at time $t$ . For a more detailed explanation, see Sun et al. (2010). Since we are interested in detecting salient events only, we do not take into account the exact motion at each location as given by $u (x, y, t)$ and $v (x, y, t)$ , instead, we look at how big the motion is at each location in the image. The magnitude of motion at each location is computed as,

M (x, y, t) = \sqrt{u (x, y, t)^{2} + v (x, y, t)^{2}}

(1)

The motion map, $M (x, y, t)$ gives the magnitude of motion at each location in the visual scene at different time instances, $t$ . Figure 2 shows two successive frames of the video in (A) and (B) respectively. The computed magnitude of motion²²2for an alternate method to compute motion and incorporate it into saliency map, see Molin et al. (2013) using the optic flow method in Sun et al. (2010) is shown in (C). The person in the video is the only moving source.

Figure 2: Computation of motion magnitude map. (A)A frame from the dataset, (B)the successive frame. The person in the two frames is the only moving object in the two image frames, moving from left to right in the image. (C)The motion magnitude map

3.1.2 Auditory loudness and location channel

The auditory input consists of a recording of the 3D sound field using 64 microphones arranged on a sphere (See Section 4 for details). We compute a single map for both loudness and location of sound sources, $A (x, y, t)$ . The value at a location in the map, $A (x, y, t)$ gives an estimate of loudness at that location at time, $t$ , hence we simultaneously get the presence and loudness of sound sources at every location in the entire environment. These two features are computed using beamforming technique as described in Donovan et al. (2007); Meyer and Elko (2002); O’Donovan et al. (2007). A more detailed account is given in Section 4. Two different frames of video and the corresponding auditory loudness and location maps superimposed on the visual images are shown in Figure 3(B) and (F) respectively. Warm colors indicate higher intensity of sound from that location in the video.

Figure 3: Computation of auditory location and loudness map: Two different video frames in (A) and (E). Corresponding audio samples from one of the audio channels are shown in (D) and (H), which display only 4410 samples, i.e. , audio of 0.1 s. Corresponding auditory loudness and location maps are in (C) and (G). Corresponding video overlays in (B) and (F) for illustration only, not used in any computation. Left column: sound of a clap. Right column: some part of the word “eight” uttered by the person

3.2 Feature pyramid decomposition

Feature pyramids are computed for each type. As the scale increases, the resolution of the feature map decreases. The feature maps of successively higher scales are computed by downsampling the feature map from the previous scale. The downsampling factor can be either $\sqrt{2}$ (half-octave) or 2 (full octave). The feature pyramids thus obtained are used to compute proto-objects by border ownership and grouping computation process explained the next two sections.

3.3 Border ownership pyramid computation

Computation of proto-objects by grouping mechanism can be divided into two sub-steps: (i) border ownership pyramid computation, and (ii) grouping pyramid computation.

The operations performed on any of the features, auditory or visual is the same. Edges of four orientations, $θ = {0, π / 4, π / 2, 3 π / 4}$ are computed using the Gabor filter bank. The V1 complex cell responses Russell et al. (2014) thus obtained are used to construct the edge pyramids. Border ownership response is computed by modulating the edge pyramid by the activity of center-surround feature differences on either side of the border. The rationale behind this is the observation made by Zhang and von der Heydt (2010), where they reported that the activity border ownership cells was enhanced when image fragments were placed on their preferred side, but suppressed for the non-preferred side.

Two types of center-surround (CS) feature pyramids are used. The center-surround light pyramid detects strong features surrounded by weak ones. Similarly, to detect weak features surrounded by a strong background, a center-surround dark pyramid is used. The rationale behind light and dark CS pyramids is that there can be bright objects on a dark background and vice-versa. The center-surround pyramids are constructed by convolving feature maps with Difference of Gaussian (DoG) filters.

The CS pyramid computation is performed in this manner for all feature types including motion and audio, except for the orientation channel. For the orientation feature channel, the DoG filters are replaced by the even symmetric Gabor filters which detect edges. This is because, for the orientation channel, feature contrasts are not typically symmetric as in the case of other channels, but oriented at a specific angle.

An important step in the border ownership computation is normalization of the center-surround feature pyramids. We follow the noramlization method used in Itti et al. (1998) which enhances isolated high activities and suppresses many closely clustered similar activities.

This normalization step enables comparison of light and dark CS pyramids. Because of the normalization, border ownership activity and grouping activity are proportionately modulated, deciding relative salience of proto-objects from the grouping activity.

The border ownership (BO) pyramids corresponding to light and dark CS pyramids are constructed by modulating the edge activity by the normalized CS pyramid activity. The light and dark BO pyramids are merged across scales and summed to get contrast polarity invariant BO pyramids. For each orientation, two BO pyramids with opposite BO preferences are be computed. From this, the winning BO pyramids are computed by a winner-take-all mechanism.

3.4 Grouping pyramid computation

The grouping computation shifts the BO activity from edge pixels to object centers. Grouping pyramids are computed by integrating the winning BO pyramid activity such that selectivity for Gestalt properties of convexity, proximity and surroundedness is enhanced. This is done by using Grouping cells in this computation, which have an annular receptive field. The shape of G cells gives rise to selectivity for convex, surrounded objects. At this stage we have the grouping or proto-object pyramids which are normalized and combined across scales to compute feature conspicuity maps, and then the saliency map.

3.5 Normalization and across-scale combination of grouping pyramids

The computation of grouping pyramids as explained in Section 3.4 is performed for each feature type. Let us represent the grouping pyramid for intensity feature channel by $G_{I}^{k} (x, y, t)$ , where $k$ denotes the scale of the proto-object map in the grouping pyramid. The color feature sub-channel grouping pyramids are represented as $G_{R G}^{k} (x, y, t)$ for Red-Green, $G_{G R}^{k} (x, y, t)$ for Green-Red, $G_{B Y}^{k} (x, y, t)$ for Blue-Yellow and $G_{Y B}^{k} (x, y, t)$ for Yellow-Blue color opponencies. The orientation grouping pyramids are denoted by $G_{O}^{k} (x, y, t, θ)$ where $θ$ denotes orientation, motion feature channel by $G_{M}^{k} (x, y, t)$ and auditory location and intensity feature channel by $G_{A}^{k} (x, y, t)$ . The corresponding conspicuity maps for , intensity $I (x, y, t)$ , color $C (x, y, t)$ , orientation $O (x, y, t)$ , motion, $M (x, y, t)$ and auditory location and loudness estimate, $A (x, y, t)$ are respectively obtained as,

I (x, y, t) = k = 10 ⨁ k = 1 N (G_{I}^{k} (x, y, t))

(2)

\begin{matrix} C (x, y, t) = k = 10 ⨁ k = 1 (N (G_{R G}^{k} (x, y, t)) + N (G_{G R}^{k} (x, y, t)) + N (G_{B Y}^{k} (x, y, t)) + N (G_{Y B}^{k} (x, y, t))) \end{matrix}

(3)

O (x, y, t) = k = 10 ⨁ k = 1 \sum θ \in {0, \frac{π}{4}, \frac{π}{2}, 3 \frac{π}{4}} N (G_{O}^{k} (x, y, t, θ))

(4)

M (x, y, t) = k = 10 ⨁ k = 1 N (G_{M}^{k} (x, y, t))

(5)

A (x, y, t) = k = 10 ⨁ k = 1 N (G_{A}^{k} (x, y, t))

(6)

where $N (.)$ is a normalization step as explained in Itti et al. (1998), which accentuates strong isolated activity and suppresses many weak activities, the symbol $⨁$ denotes “across-scale” addition of the proto-object maps, which is done by resampling (up- or down-sampling depending on the scale, $k$ ) maps at each level to a common scale (in this case, the common scale is $k = 8$ ) and then doing pixel-by-pixel addition. We use the same set of parameters as in Table 1 of Russell et al. (2014) for our computation as well.

The conspicuity maps, due to varied number of feature sub-channels have different ranges of activity, hence if we linearly combine without any rescaling to a common scale, those features with higher number of sub-channels may dominate. Hence, each feature conspicuity map is rescaled to the same range, $[0, \dots, 1]$ . The conspicuity maps are combined in different ways to get different types of saliency maps as explained in Section 3.6.

3.6 Combination of conspicuity maps

The visual saliency map is computed as,

\begin{matrix} V S M (x, y, t) = w_{I} R (I (x, y, t)) + w_{C} R (C (x, y, t)) + w_{O} R (O (x, y, t)) + w_{M} R (M (x, y, t)) \end{matrix}

(7)

where $V S M (x, y, t)$ is the visual saliency map, $R (.)$ is the rescaling operator that rescales each map to the same range, $[0, \dots, 1]$ and $w_{I}, w_{C}, w_{O}$ and $w_{M}$ are the individual weights for intensity, color, orientation and motion conspicuity maps, respectively. In our implementation, all weights are equal and each is set to 0.25, i.e. , $w_{I} = w_{C} = w_{O} = w_{M} = \frac{1}{4}$ .

Since audio is a single feature channel, the conspicuity map for auditory location and loudness is also the auditory saliency map, $A S M (x, y, t)$ .

We compute the audio-visual saliency map in three different ways to compare the most effective method to identify salient events (See Section 5 for related discussion).

In the first method a weighted combination of all feature maps is done to get the audio-visual saliency map as,

\begin{matrix} {A V S M}_{1} (x, y, t) = w_{I} R (I (x, y, t)) + w_{C} R (C (x, y, t)) + w_{O} R (O (x, y, t)) + w_{M} R (M (x, y, t)) + w_{A} R (A (x, y, t)) \end{matrix}

(8)

where different weights can be set for $w_{(.)}$ such that the sum of all weights equals 1. In our implementation, all weights are set equal, i.e. , $w_{I} = w_{C} = w_{O} = w_{M} = w_{A} = \frac{1}{5}$ .

In the second method, the visual saliency map is computed as in Equation 7 and then a simple average of the visual saliency map and the auditory conspicuity map (also auditory saliency map, $A S M (x, y, t)$ ) is computed to get the audio-visual saliency map as,

{A V S M}_{2} (x, y, t) = \frac{1}{2} (R (V S M (x, y, t)) + R (A (x, y, t)))

(9)

The distribution of weights in Equation 9 is different from that in Equation 8. In method 2, a “late combination” of the visual and auditory saliency maps is performed, which results in an increase in the weight of the auditory saliency map and a reduction in weights for the individual feature conspicuity maps of the visual domain.

In the last method, in addition to a linear combination of the visual and auditory saliency maps, a product term is added as,

\begin{matrix} {A V S M}_{3} (x, y, t) = (R (V S M (x, y, t)) + R (A (x, y, t)) + R (V S M (x, y, t)) \otimes R (A (x, y, t))) \end{matrix}

(10)

where the symbol, $\otimes$ denotes a point-by-point multiplication of pixel values of the corresponding saliency maps. The effect of the product term is to increase the saliency of those events that are salient in both visual and auditory domains, thereby to enhance the saliency of spatiotemporally concurrent audiovisual events. A comparison of the different saliency maps in detecting salient events is in Section 5.

4 Data and Methods

Audio-Visual data is collected using the VisiSonics RealSpace^TM audio-visual camera Donovan et al. (2007); O’Donovan et al. (2007). The AV camera consists of a spherical microphone array with 64 microphones arranged on a sphere of 8 inches diameter and 15 HD cameras arranged on the same sphere (Figure 4). Each video camera can record color (RGB) videos at a resolution of $1328 \times 1044$ pixels per frame and 10 frames per second. The audio channels record high fidelity audio at a sampling rate of 44.1 kHz per channel. The audio and video data are converted into a single USB 3.0 compliant stream which is accepted by a laptop computer with Graphical Processing Units. The individual videos are stitched together to produce a panoramic view of the scene in two different projection types: spherical and Mercator. The audio and video streams are synchronized by an internal Field Programmable Gate Array (FPGA) based processor. The equipment can be used to localize sounds and display them on the panoramic video in real time and also record AV data for later analysis. The length of each recording can be set at any value between 10 seconds and 390 seconds . The gain for each recording session can set to three predefined values: -20 dB, 0 dB and +20 dB. This is particularly helpful to record sounds with high fidelity in indoor, outdoor and noisy conditions.

Figure 4: The Audio-Visual Camera being used to collect data in a recording session outdoors

To compute the loudness and location estimate of sound sources in the scene, Spherical Harmonics Beamforming (SHB) technique is used. The 3D sound field sampled at discrete locations on a solid sphere is decomposed into spherical harmonics, whose angular part resolves the direction of the sound field. Spherical harmonics are the 2D counterpart of Fourier transforms (defined on a unit circle) defined on the surface of a sphere. A description of the mathematical details of the SHB method is beyond the scope of this article, interested readers can find details in

Donovan et al. (2007); O’Donovan et al. (2007); Meyer and Elko (2002).

The 64 audio channels recorded at 44100 Hz are divided into frames, each of 4410 samples. This gives us 10 audio frames per second, which is equal to the video frame rate. Spherical harmonic beamforming is done for the audio frames to locate sounds in the frequency range, $[300, 6500]$ Hz. For natural sounds, a wide frequency range, as chosen is sufficient to estimate location and loudness of most types of sounds including speech and music. The azimuth angle for SHB is chosen in the range, $[0, 2 π]$ with the angular resolution of $\frac{2 π}{128}$ radians, i.e. , ${2.8125}^{0}$ and the elevation angle in $0, π$ with an angular resolution of $\frac{π}{64}$ radians, i.e. , ${2.8125}^{0}$ radians. The output of SHB is the auditory location and loudness estimate map as shown in Figure 3.

We collected four audiovisual datasets using the AV camera equipment, where three datasets are 60 seconds in length and the other one is of 120 seconds duration, all indoors. The AV camera equipment and our algorithms can handle data from any type of audiovisual surroundings, indoor or outdoor. We made sure all combination of salient events in purely visual intensity, color, motion, audio and audiovisual domains were present. SHB was performed with parameters set as explained in the previous paragraph to get the sound location and loudness estimates. The videos are stitched together to produce panoramic image in the Mercator projection which is used in our saliency computation. The stitch depth for panoramic images was 14 feet, hence in objects that are too close to the AV camera appear to be blurred due to overlapping of images from different cameras on the sphere (Figure 5).

The scene consisted of a loudspeaker placed on a desk in one corner of the room (green box) playing documentaries, a person (author) either sitting or moving around the AV camera equipment clapping or uttering numbers out loud, an air conditioning vent (blue box) making some audible noise and other objects visible in the scene. The loudspeaker acts like a stationary sound source, which was switched on/off during the recording session to produce abrupt onsets/offsets. The person wearing a black/blue jacket moving around the recording equipment acts like a salient visual object with or without motion, when speaking acts as an audiovisual source. The person moves in and out of the room producing abrupt motion onsets/offsets. The air conditioning vent, which happens to be very close to the recording equipment is a source of noise which is audible to anyone present in the room, acts as another stationary audio source. The bright lights present in the room act as visually salient stationary objects. The set of sources together produce all possible combinations of visually salient events with or without motion, acoustically salient sources with or without motion and audiovisual salient events/objects with and without motion. The dataset can be viewed/listened to at the following url: https://preview.tinyurl.com/ybg4fch4. Results of audiovisual saliency, comparison with unisensory saliencies on this dataset are discussed in the next section.

Figure 5: The audiovisual data collection scene. The scene consists of a loudspeaker (green box), a person and an air conditioning vent (blue box)

5 Results and Discussion

First, we will examine which of the three audiovisual saliency computation methods described in Section 3.6, Eqs 8 - 10 performs well for different stimulus conditions. Then we will compare results from the best AVSM with the unisensory saliency maps followed by discussion of the results.

All saliency maps computed as explained in Section 3.6 will have salience value in $(0, 1)$ range. On such a saliency map, unisensory or audiovisual, anything above a threshold of 0.75 is determined as highly salient. This threshold is same for all saliency maps, Visual Saliency Map ( $V S M$ , variables $(x, y, t)$ dropped as unnecessary here), Auditory Saliency Map ( $A S M$ ) and the three different Audio-Visual Saliency Maps ( ${A V S M}_{i}, where i = 1, 2, 3$ ). Hence, this provides a common baseline to compare the workings of unisensory SMs with AVSM, and among different AVSMs.

To visualize the results we did the following: On the saliency map (can be $V S M$ , $A S M$ or ${A V S M}_{i}$ ), saliency value based isocontours for the threshold of 0.75 are drawn and superimposed on each of the input video frame. For example, see Figure 6, where ${A V S M}_{1}$ for frame # 77 of Dataset 2 is shown. Any thing that is inside the closed red contour of Figure 6(B) is highly salient and has a saliency value greater than 0.75. Outside the isocontour, the salience value is less than 0.75. Exactly, along the isocontour the salience value is 0.75 (precisely, $0.75 \pm 0.02$ ).

The results can be best interpreted by watching the input and different saliency map videos. But, since it is not possible to show all the frames and for the lack of a better way of presenting the results, we display the saliency maps for a few key frames only. The videos and individual frames of the saliency maps are available at the url: https://preview.tinyurl.com/ybg4fch4.

Figure 6: Visualization of results with isocontours. (A) Input frame # 77 of Dataset 2. (B) The red contour superimposed on the same input frame is the isocontour of salience values. All the values along the red line are equal to the threshold value. Anything inside the closed red isocontour has a salience value greater than 0.75

Figure 7 shows ${A V S M}_{1}$ , ${A V S M}_{2}$ and ${A V S M}_{3}$

for input image frame # 393 of Dataset 1. At that moment in the scene, the loudspeaker (at the center of the image frame) was playing a documentary and the person was moving forward. So, there is a salient stationary auditory event and a salient visual motion. From visual inspection of Figure

7, it is clear that

{A V S M}_{1}

{A V S M}_{2}

and

{A V S M}_{3}

give roughly the same results, and are able to detect salient events in both modalities. This is the behavior we see in all AVSMs (

{A V S M}_{i}

) for a majority of frames. But, in some cases, when the scene reduces to a static image, the behavior exhibited by each of the methods will be somewhat different.

Figure 7: Comparison of audiovisual saliency computation methods. (A) Input frame # 393 of Dataset 1. A small loudspeaker at the corner of a room, which is located almost at the center of the image frame, is playing a documentary, hence constitutes a salient stationary auditory event. The person is leaning forward gives rise to salient visual motion. (B) ${A V S M}_{1}$ computed using Eq 8 where all 5 feature channels are combined linearly with equal weights. (C) ${A V S M}_{2}$ computed using Eq 9 where $V S M$ and $A S M$ are averaged. (D) ${A V S M}_{3}$ computed using Eq 10. All methods give similar results with minor differences

Consider, for example, frame # 173 of Dataset 3, where the visual scene is equivalent to a static image with a weak auditory stimulus, which is the air conditioning vent noise (Figure 8). Here, according to ${A V S M}_{1}$ (Figure 8 (B)), the most salient location coincides with the strongest intensity based salient location at the bottom part of the image. This is because in ${A V S M}_{1}$ we averaged the conspicuity maps with equal weights. So, when salient audio or motion is not present, the AVSM automatically switches to being a static saliency map with Color, Intensity and Orientation as dominant features. But, in ${A V S M}_{2}$ it is computed as the average of visual and auditory saliency maps, hence it leads to redistribution of weights in such a way that each of the visual features contributes only one-eighth to the final saliency map and audio channel contributes one half. As a result, auditory reflections could get accentuated and show up as salient, which may not match with our judgment, as seen in Figure 8(C). In reality, such auditory reflections are imperceptible, hence may not draw our attention as indicated by ${A V S M}_{2}$ .

Figure 8: Audiovisual saliency in a static scene. (A) Input frame # 173 of Dataset 3. The scene is almost still reducing the visual input to a static image with a weak auditory noise emanating from the air conditioning vent. (B) The most salient location according to ${A V S M}_{1}$ coincides with the intensity based salient location at the bottom of the image. (C) ${A V S M}_{2}$ shows some locations that are not salient according to any feature. This may be happening due to exaggeration of auditory reflections which are detected as salient in ${A V S M}_{2}$ . (D) ${A V S M}_{3}$ shows similar results as ${A V S M}_{2}$

In ${A V S M}_{3}$ where a multiplicative term, $V S M (x, y, t) \times A S M (x, y, t)$ is added, accentuates the conjunction of visual and auditory salient events if they are spatio-temporally coincident. But, since auditory and visual saliencies already contribute equally instead of the five independent features making equal contributions, the effect of the multiplicative term is small, so we see that ${A V S M}_{3}$ has similar behavior as ${A V S M}_{2}$ . We did not investigate whether the conjunction of individual feature conspicuity maps, like $I (x, y, t) \times A S M (x, y, t)$ , $O (x, y, t) \times C (x, y, t)$ , etc can result in a better saliency map. But based on visual comparison we can conclude that ${A V S M}_{1}$ , where each feature channel contributes equally, irrespective of whether it is visual or auditory, is a better AVSM computation method compared to ${A V S M}_{2}$ and ${A V S M}_{3}$ .

So an important observation we can make at this point is that, even though vision and audition are two separate sensory modalities and we expect them to equally influence the bottom-up, stimulus driven attention, this may not be the case. Instead, we can conclude that each feature irrespective of the sensory modality makes the same contribution to the final saliency map from a bottom-up perspective.

Next, we will compare how ${A V S M}_{1}$ performs in comparison to unisensory saliency maps, namely $V S M$ and $A S M$ . Since ${A V S M}_{1}$ was found to be better, the other two AVSMs are not discussed.

Figure 9 shows $A S M$ , $V S M$ and ${A V S M}_{1}$ for frame # 50 of Dataset 4, where the person moving is the most salient event, which is correctly detected in $V S M$ and ${A V S M}_{1}$ , but not in $A S M$ . This is expected.

Figure 9: Comparison of AVSM with unisensory saliency maps (A) Input frame # 50 of Dataset 4. The most prominent event in the scene is the person moving. (B) The most salient location according to $A S M$ misses the most salient location, but shows a different location as salient (C) $V S M$ shows the prominent motion event as salient as expected (D) ${A V S M}_{1}$ also captures the salient motion event as the most salient

Next, in Figure 10 saliency maps for frame # 346 of Dataset 2 are shown, where audio from the loudspeaker is the most salient event, which is correctly detected as salient in $A S M$ and ${A V S M}_{1}$ , but missed in $V S M$ , which agrees with our judgment.

Figure 10: Comparison of AVSM with unisensory saliency maps (A) Input frame # 346 of Dataset 2. The most prominent event in the scene is the audio from the loudspeaker. (B) The audio event is salient in $A S M$ (C) $V S M$ in this case would be equivalent to a static saliency map, hence the auditory salient event is missed here(D) ${A V S M}_{1}$ also captures the audio as the most salient as expected

In frame # 393 of Dataset 1, there is strong motion of the person as well as sound from the loudspeaker. The unisensory and audiovisual saliency maps are shown in Figure 11. Again, the salients events detected by the respective saliency maps agree with our judgment.

Figure 11: Comparison of AVSM with unisensory saliency maps (A) Input frame # 393 of Dataset 1. (B) $A S M$ detects the salient auditory event, but misses the salient visual motion (C) $V S M$ correctly detects the salient motion event, but misses the salient audio. (D) ${A V S M}_{1}$ captures both valid salient events from two different sensory domains

From these results we can conclude, the unisensory saliency maps detect valid unisensory events which agree with human judgment. At the same time, the audiovisual saliency map detects salient events from both sensory modalities, which again agree with our judgment. So, we can say, the AVSM detects more number of valid salient events compared to unisensory saliency maps. The unisensory saliency maps miss the salient events from the other sensory modality. Hence, AVSMs in general, and ${A V S M}_{1}$ in particular, perform better than unisensory saliency maps in detecting valid salient events. As a result, AVSMs can be more useful in a variety of applications like surveillance, robotic navigation, etc. Overall, the proto-object based audiovisual saliency map reliably detects valid salient events for all combinations of auditory, visual and/or audiovisual events in a majority of the frames. The readers can verify themselves additionally by watching the videos or looking at individual video frames at: https://preview.tinyurl.com/ybg4fch4.

An important distinguishing factor of our AVSM computation comes from the use of proto-objects. With proto-object based computation, we see that salient locations roughly coincide with object centers giving an estimate of audiovisual “objectness” Alexe et al. (2012). So, this enables selection of image regions with possible objects based on saliency values. Moreover, since saliency gives a natural mechanism for ranking scene locations based on salience value, combined with “objectness” that comes from proto-objects, this can serve to select image regions for object recognition, activity recognition, etc

with other methods, such as deep Convolutional Neural Networks

Krizhevsky et al. (2012); Ren et al. (2015).

Second, due to linear combination of feature conspicuity maps, the model adapts itself to any scene type, static or dynamic scenes, with or without audio. Because of this, we get a robust estimate of bottom-up saliency in a majority of cases. Plus, the method works well for a variety of environments, indoor and outdoor.

Finally, the AVSM computed in this manner enables us to represent and compare saliencies of events from two different sensory modalities on a common scale. Other sensory modalities or feature channels can be similarly incorporated into the model.

One of the factors that we have not considered in our model is the temporal modulation of audiovisual saliency. We treat each 100 ms interval as a snapshot, independent of previous frames and compute unisensory and audiovisual saliencies for each 100 ms frame. Even though this “memoryless” computation detects valid salient events well, temporal aspects are found to strongly influence saliency, especially from the auditory domain Kaya and Elhilali (2012). Hence, factoring in the temporal dependence of saliency can further improve the model. For example, in the few cases where saliency maps appears to be noisy, we can improve the results with temporal smoothing of the saliency maps. Though, the proportion of such noisy frames is very small compared to valid detections.

Temporal dependence of attention is important from the perspective of perception as well. For example, a continuous motion or an auditory alarm can be salient at the beginning of the event due to abrupt onset, but if it continues to persist, we may switch our attention to some other event, even though it is prominent in the scene. The mechanism and time course of multisensory attentional modulation needs to be further investigated and incorporated into the model.

Another aspect, related to temporal modulation of audiovisual saliency that we have not considered is the Inhibition of Return (IOR) Posner and Cohen (1984); Itti et al. (1998). IOR refers to increased reaction time to attend to a previously cued spatial location compared to an uncued location. The exact nature of IOR in the case of audiovisual attention is an active topic of research Van der Stoep et al. (2017); Spence and Driver (1998). More recent experimental evidence Van der Stoep et al. (2017) suggests that IOR is not observed in audiovisual attention conditions. If this is the case, not having audiovisual IOR may not be a significantly limiting factor, but certainly worth investigating.

Lastly, a drawback of our work is that the results are not validated with human psychophysics experiments. Since, saliency models aim to predict human attention based on bottom-up features, validating the results with human psychophysics experiments is necessary. Such validation would strengthen the findings of our study even more. But from visual judgment of the results, the readers can verify that the model is capable of selecting valid, perceptually salient audiovisual events for further processing. Moreover, our goal is to build a useful computational tool for automated scene analysis and the results show that the model is capable of doing so. features, validating the results by presenting it human beings and measuring how they allocate attention in such environments, how much of their attentional behavior can be explained purely based on bottom-up features should be studied to validate and further strengthen the findings of our study.

6 Conclusion and Future Work

We have shown that a proto-object based audiovisual saliency map detects salient unisensory and multisensory events, which agree with human judgment. The AVSM detects a higher number of valid salient events compared to unisensory saliency maps demonstrating the superiority and usefulness of proto-object based multisensory saliency map. Among the different audiovisual saliency methods, we show that linear combination of individual feature channels with equal weights gives the best results. The AVSM computed this way performs better compared to others in detecting valid salient events for static as well as dynamic scenes, with or without salient auditory events in the scene. Also, it is less noisy and more robust compared to other combination methods where visual and auditory conspicuity maps, instead of individual feature channels, are equally weighed.

In future, incorporating the temporal modulation of saliency would be considered. We would also like to validate the AVSM with psychophysics experiments. Also, the role of Inhibition of Return in the case of audiovisual saliency map would also be investigated. In conclusion, a proto-object based audiovisual saliency map with linear and equally weighted feature channels detects a higher number of valid unisensory and multisensory events that agree with human judgment.

References

Stein and Stanford (2008) B. E. Stein, T. R. Stanford, Multisensory integration: current issues from the perspective of the single neuron, Nature Reviews Neuroscience 9 (2008) 255–266.
Stevenson and James (2009) R. A. Stevenson, T. W. James, Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition, Neuroimage 44 (2009) 1210–1223.
Calvert et al. (2004) G. A. Calvert, C. Spence, B. E. Stein, The handbook of multisensory processes, MIT press, 2004.
Spence et al. (2009) C. Spence, D. Senkowski, B. Röder, Crossmodal processing, Experimental Brain Research 198 (2009) 107–111.
Alais and Burr (2004) D. Alais, D. Burr, The ventriloquist effect results from near-optimal bimodal integration, Current biology 14 (2004) 257–262.
Ghazanfar and Schroeder (2006) A. A. Ghazanfar, C. E. Schroeder, Is neocortex essentially multisensory?, Trends in cognitive sciences 10 (2006) 278–285.
Von Kriegstein et al. (2005) K. Von Kriegstein, A. Kleinschmidt, P. Sterzer, A.-L. Giraud, Interaction of face and voice areas during speaker recognition, Journal of Cognitive Neuroscience 17 (2005) 367–376.
Watkins et al. (2006) S. Watkins, L. Shams, S. Tanaka, J.-D. Haynes, G. Rees, Sound alters activity in human V1 in association with illusory visual perception, Neuroimage 31 (2006) 1247–1256.
van Wassenhove et al. (2005) V. van Wassenhove, K. W. Grant, D. Poeppel, Visual speech speeds up the neural processing of auditory speech, Proceedings of the National Academy of Sciences of the United States of America 102 (2005) 1181–1186.
Ghazanfar et al. (2005) A. A. Ghazanfar, J. X. Maier, K. L. Hoffman, N. K. Logothetis, Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex, The Journal of Neuroscience 25 (2005) 5004–5012.
Wang et al. (2008) Y. Wang, S. Celebrini, Y. Trotter, P. Barone, Visuo-auditory interactions in the primary visual cortex of the behaving monkey: electrophysiological evidence, BMC neuroscience 9 (2008) 79.
Çetingül et al. (2006) H. Çetingül, E. Erzin, Y. Yemez, A. M. Tekalp, Multimodal speaker/speech recognition using lip motion, lip texture and audio, Signal processing 86 (2006) 3549–3558.
Tamura et al. (2005) S. Tamura, K. Iwano, S. Furui, Toward robust multimodal speech recognition, in: Symposium on Large Scale Knowledge Resources (LKR2005), pp. 163–166.
Russell et al. (2014) A. F. Russell, S. Mihalaş, R. von der Heydt, E. Niebur, R. Etienne-Cummings, A model of proto-object based saliency, Vision research 94 (2014) 1–15.
Alais et al. (2010) D. Alais, F. N. Newell, P. Mamassian, Multisensory processing in review: from physiology to behaviour, Seeing and perceiving 23 (2010) 3–38.
Meredith and Stein (1986) M. A. Meredith, B. E. Stein, Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration, Journal of neurophysiology 56 (1986) 640–662.
Evangelopoulos et al. (2008) G. Evangelopoulos, K. Rapantzikos, A. Potamianos, P. Maragos, A. Zlatintsi, Y. Avrithis, Movie summarization based on audiovisual saliency detection, in: 15th IEEE International Conference on Image Processing, IEEE, pp. 2528–2531.
Song (2013) G. Song, Effet du son dans les vidéos sur la direction du regard: contribution à la modélisation de la saillance audiovisuelle, Ph.D. thesis, Université de Grenoble, 2013.
Ramenahalli et al. (2013) S. Ramenahalli, D. R. Mendat, S. Dura-Bernal, E. Culurciello, E. Nieburt, A. Andreou, Audio-visual saliency map: overview, basic models and hardware implementation, in: Information Sciences and Systems (CISS), 2013 47th Annual Conference on, IEEE, pp. 1–6.
Grossberg et al. (1997) S. Grossberg, K. Roberts, M. Aguilar, D. Bullock, A neural model of multimodal adaptive saccadic eye movement control by superior colliculus, The Journal of neuroscience 17 (1997) 9706–9725.
Meredith and Stein (1996) M. A. Meredith, B. E. Stein, Spatial determinants of multisensory integration in cat superior colliculus neurons, Journal of Neurophysiology 75 (1996) 1843–1857.
Meredith et al. (1987) M. A. Meredith, J. W. Nemitz, B. E. Stein, Determinants of multisensory integration in superior colliculus neurons. I. temporal factors, The Journal of neuroscience 7 (1987) 3215–3229.
Casey et al. (2012) M. C. Casey, A. Pavlou, A. Timotheou, Audio-visual localization with hierarchical topographic maps: Modeling the superior colliculus, Neurocomputing 97 (2012) 344–356.
Huo and Murray (2009) J. Huo, A. Murray, The adaptation of visual and auditory integration in the barn owl superior colliculus with spike timing dependent plasticity, Neural Networks 22 (2009) 913–921.
Huo et al. (2012) J. Huo, A. Murray, D. Wei, Adaptive visual and auditory map alignment in barn owl superior colliculus and its neuromorphic implementation, Neural Networks and Learning Systems, IEEE Transactions on 23 (2012) 1486–1497.
Anastasio et al. (2000) T. J. Anastasio, P. E. Patton, K. Belkacem-Boussaid, Using bayes’ rule to model multisensory enhancement in the superior colliculus, Neural Computation 12 (2000) 1165–1187.
Patton et al. (2002) P. Patton, K. Belkacem-Boussaid, T. J. Anastasio, Multimodality in the superior colliculus: an information theoretic analysis, Cognitive Brain Research 14 (2002) 10–19.
Patton and Anastasio (2003) P. E. Patton, T. J. Anastasio, Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons, Neural computation 15 (2003) 783–810.
Colonius and Diederich (2004) H. Colonius, A. Diederich, Why aren’t all deep superior colliculus neurons multisensory? a bayes’ ratio analysis, Cognitive, Affective, & Behavioral Neuroscience 4 (2004) 344–353.
Ma et al. (2006) W. J. Ma, J. M. Beck, P. E. Latham, A. Pouget, Bayesian inference with probabilistic population codes, Nature neuroscience 9 (2006) 1432–1438.
Wilson et al. (2002) K. Wilson, V. Rangarajan, N. Checka, T. Darrell, Audiovisual arrays for untethered spoken interfaces, in: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, ICMI ’02, IEEE Computer Society, Washington, DC, USA, 2002, pp. 389–.
Torres and Kalva (2014) F. Torres, H. Kalva, Influence of audio triggered emotional attention on video perception, in: IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, pp. 901408–901408.
Lee et al. (2011) J.-S. Lee, F. De Simone, T. Ebrahimi, Efficient video coding based on audio-visual focus of attention, Journal of Visual Communication and Image Representation 22 (2011) 704–711.
Rerabek et al. (2014) M. Rerabek, H. Nemoto, J.-S. Lee, T. Ebrahimi, Audiovisual focus of attention and its application to ultra high definition video compression, in: IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, pp. 901407–901407.
Ruesch et al. (2008) J. Ruesch, M. Lopes, A. Bernardino, J. Hornstein, J. Santos-Victor, R. Pfeifer, Multimodal saliency-based bottom-up attention a framework for the humanoid robot icub, in: IEEE International Conference on Robotics and Automation, IEEE, pp. 962–967.
Schauerte et al. (2009) B. Schauerte, J. Richarz, T. Plötz, C. Thurau, G. A. Fink, Multi-modal and multi-camera attention in smart environments, in: Proceedings of the 2009 international conference on Multimodal interfaces, ACM, pp. 261–268.
Schauerte et al. (2011) B. Schauerte, B. Kuhn, K. Kroschel, R. Stiefelhagen, Multimodal saliency-based attention for object-based scene analysis, in: IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, pp. 1173–1179.
Schauerte (2016) B. Schauerte, Bottom-up audio-visual attention for scene exploration, in: Multimodal Computational Attention for Scene Understanding and Robotics, Springer, 2016, pp. 35–113.
Onat et al. (2007) S. Onat, K. Libertus, P. Konig, Integrating audiovisual information for the control of overt attention, in: Journal of Vision, 7(10):11, 2007.
Kühn et al. (2012a) B. Kühn, B. Schauerte, R. Stiefelhagen, K. Kroschel, A modular audio-visual scene analysis and attention system for humanoid robots, in: Proc. 43rd Int. Symp. Robotics (ISR).
Kühn et al. (2012b) B. Kühn, B. Schauerte, K. Kroschel, R. Stiefelhagen, Multimodal saliency-based attention: A lazy robot’s approach, in: IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, pp. 807–814.
Bauer et al. (2012) J. Bauer, C. Weber, S. Wermter, A som-based model for multi-sensory integration in the superior colliculus, in: The 2012 international joint conference on Neural Networks (IJCNN), IEEE, pp. 1–8.
Viciana-Abad et al. (2014) R. Viciana-Abad, R. Marfil, J. M. Perez-Lorenzo, J. P. Bandera, A. Romero-Garces, P. Reche-Lopez, Audio-visual perception system for a humanoid robotic head, Sensors 14 (2014) 9522–9545.
Evangelopoulos et al. (2008) G. Evangelopoulos, K. Rapantzikos, P. Maragos, Y. Avrithis, A. Potamianos, Audiovisual attention modeling and salient event detection, in: Multimodal Processing and Interaction, Springer, 2008, pp. 1–21.
Rapantzikos et al. (2007) K. Rapantzikos, G. Evangelopoulos, P. Maragos, Y. Avrithis, An audio-visual saliency model for movie summarization, in: Multimedia Signal Processing, 2007. MMSP 2007. IEEE 9th Workshop on, IEEE, pp. 320–323.
Evangelopoulos et al. (2013) G. Evangelopoulos, A. Zlatintsi, A. Potamianos, P. Maragos, K. Rapantzikos, G. Skoumas, Y. Avrithis, Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention, Multimedia, IEEE Transactions on 15 (2013) 1553–1568.
Nakajima et al. (2014) J. Nakajima, A. Sugimoto, K. Kawamoto, Incorporating audio signals into constructing a visual saliency map, in: Image and Video Technology, Springer, 2014, pp. 468–480.
Itti and Baldi (2008) L. Itti, P. Baldi, Bayesian surprise attracts human attention, Vision Research epub (2008).
Nakajima et al. (2015) J. Nakajima, A. Kimura, A. Sugimoto, K. Kashino, Visual attention driven by auditory cues, in: MultiMedia Modeling, Springer, pp. 74–86.
Korchagin et al. (2011) D. Korchagin, P. Motlicek, S. Duffner, H. Bourlard, Just-in-time multimodal association and fusion from home entertainment, in: 2011 IEEE International Conference on Multimedia and Expo (ICME), IEEE, pp. 1–5.
Hershey and Movellan (2000) J. R. Hershey, J. R. Movellan, Audio vision: Using audio-visual synchrony to locate sounds, in: Advances in Neural Information Processing Systems, pp. 813–819.
Blauth et al. (2012) D. A. Blauth, V. P. Minotto, C. R. Jung, B. Lee, T. Kalker, Voice activity detection and speaker localization using audiovisual cues, Pattern Recognition Letters 33 (2012) 373–380.
Ratajczak et al. (2016) R. Ratajczak, D. Pellerin, Q. Labourey, C. Garbay, A fast audiovisual attention model for human detection and localization on a companion robot, in: The First International Conference on Applications and Systems of Visual Paradigms (VISUAL 2016).
Song et al. (2012) G. Song, D. Pellerin, L. Granjon, How different kinds of sound in videos can influence gaze, in: 13th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), IEEE, pp. 1–4.
Coutrot and Guyader (2014a) A. Coutrot, N. Guyader, How saliency, faces, and sound influence gaze in dynamic social scenes, Journal of Vision 14 (2014a) 5.
Coutrot and Guyader (2014b) A. Coutrot, N. Guyader, An audiovisual attention model for natural conversation scenes, in: IEEE International Conference on Image Processing (ICIP), IEEE, pp. 1100–1104.
Noulas et al. (2012) A. Noulas, G. Englebienne, B. J. Kröse, Multimodal speaker diarization, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2012) 79–93.
Miro et al. (2012) X. A. Miro, S. Bozonnet, N. Evans, C. Fredouille, G. Friedland, O. Vinyals, Speaker diarization: A review of recent research, IEEE Transactions on Audio, Speech, and Language Processing 20 (2012) 356–370.
Sidaty et al. (2014) N. O. Sidaty, M.-C. Larabi, A. Saadane, Towards understanding and modeling audiovisual saliency based on talking faces, in: Tenth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS), IEEE, pp. 508–515.
Russell et al. (2014) A. F. Russell, S. Mihalas, R. von der Heydt, E. Niebur, R. Etienne-Cummings, A model of proto-object based saliency, Vision Research 94 (2014) 1–15.
Rensink (2000) R. A. Rensink, The dynamic representation of scenes, Visual cognition 7 (2000) 17–42.
Sun et al. (2010) D. Sun, S. Roth, M. J. Black, Secrets of optical flow estimation and their principles,
in: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 2432–2439.
Sun et al. (2008) D. Sun, S. Roth, M. Black, Optic flow estimation matlab code, http://cs.brown.edu/~dqsun/research/software.html, 2008.
Molin et al. (2013) J. L. Molin, A. F. Russell, S. Mihalas, E. Niebur, R. Etienne-Cummings, Proto-object based visual saliency model with a motion-sensitive channel, in: Biomedical Circuits and Systems Conference (BioCAS), 2013 IEEE, pp. 25–28.
Donovan et al. (2007) A. O. Donovan, R. Duraiswami, J. Neumann, Microphone arrays as generalized cameras for integrated audio visual processing, in: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp. 1–8.
Meyer and Elko (2002) J. Meyer, G. Elko, A highly scalable spherical microphone array based on an orthonormal decomposition of the soundfield, in: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), volume 2, IEEE, pp. II–1781.
O’Donovan et al. (2007) A. O’Donovan, R. Duraiswami, N. Gumerov, et al., Real time capture of audio images and their use with video, in: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, IEEE, pp. 10–13.
Zhang and von der Heydt (2010) N. Zhang, R. von der Heydt, Analysis of the context integration mechanisms underlying figure–ground organization in the visual cortex, The Journal of Neuroscience 30 (2010) 6482–6496.
Itti et al. (1998) L. Itti, C. Koch, E. Niebur, A model of saliency-based fast visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 20 (1998) 1254–1259.
Alexe et al. (2012) B. Alexe, T. Deselaers, V. Ferrari, Measuring the objectness of image windows, IEEE transactions on pattern analysis and machine intelligence 34 (2012) 2189–2202.
Krizhevsky et al. (2012) A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, pp. 1097–1105.
Ren et al. (2015) S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, pp. 91–99.
Kaya and Elhilali (2012) E. M. Kaya, M. Elhilali, A temporal saliency map for modeling auditory attention, in: Information Sciences and Systems (CISS), 2012 46th Annual Conference on, IEEE, pp. 1–6.
Posner and Cohen (1984) M. I. Posner, Y. Cohen, Components of visual orienting, in: H. Bouma, D. G. Bouwhuis (Eds.), Attention and Performance X, Hilldale, NJ, 1984, pp. 531–556.
Van der Stoep et al. (2017) N. Van der Stoep, S. Van der Stigchel, T. Nijboer, C. Spence, Visually induced inhibition of return affects the integration of auditory and visual information, Perception 46 (2017) 6–17.
Spence and Driver (1998) C. Spence, J. Driver, Auditory and audiovisual inhibition of return, Attention, Perception, & Psychophysics 60 (1998) 125–139.

A proto-object based audiovisual saliency map

Authors

Dynamical optical flow of saliency maps for predicting visual attention

An Integration of Bottom-up and Top-Down Salient Cues on RGB-D Data: Saliency from Objectness vs. Non-Objectness

Review of Visual Saliency Detection with Comprehensive Information

DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction

Audio-Visual Spatial Aligment Requirements of Central and Peripheral Object Events

Global and Local Sensitivity Guided Key Salient Object Re-augmentation for Video Saliency Detection

Exploiting Egocentric Object Prior for 3D Saliency Detection

1 Introduction

2 Related Work