Audio-visual Interaction



  Auditory attention task in virtual classroom scene Copyright: © IHTA Auditory attention task in virtual classroom scene

In our daily life, information is perceived via multiple senses, especially via eyes and ears. For example, a singing bird or a car passing by, will be heard and seen. In other words, auditory and visual perception will always interact in real-life situations. Especially in complex acoustic situations such as in urban environments, classrooms or in a cocktail-party situation this holds true.

To better understand the integration of different sensory modalities in these real-life situations and to further reach higher ecological validity, realistic and/or plausible listening conditions must be created. The goal is to bring field studies, which are hard to reproduce and time consuming, into the lab using virtual scenes in an acoustical and visual manner. The visual part can vary from static images to complete virtual reality-support with room- and head-mounted displays. In combination with the acoustical reproduction, the virtual sound source becomes visually and acoustically localizable, as given in a real-life situation. By bringing these two modalities together, an accurate simulation of the real world can be created and used to study combined auditory and visual perception in controlled environments.

The research field of audio-visual interaction is highly interdisciplinary. It touches biology, psychology and computer science, next to engineering and acoustics. Though every field has its own emphasis with specific research questions, all different fields must work closely together to combine knowledge and achieve top-notch insights in multi-sensory integration.

For the acoustic field, for example, it is interesting to examine the coherence between auditory and visual playback, the influence of room acoustics on audio-visual integration, as well as the plausibility of spatial sound reproduction in combination with visual objects. The effects of auditory and visual distractors on the multi-sensory speech perception and recognition are objects of research. Furthermore, audio-visual interaction is not only relevant for adults, but also for children. This research field therefore examines developmental changes in children’s cognitive behaviour in virtual scenes.

The aim of this research field “audio-visual interaction” is to gain a better understanding of the human’s audio-visual integration when listening in challenging situations. The benefits of examining the combination of auditory and visual perception lead to a better understanding of complex auditory cognitive processes such as the speech perception and the (auditory) selective attention.