Figure 1. Experimental setup. (A) The participants were equipped with the VR headset and could identify and report the source of sounds originating from five different locations (± 26.6°, ± 11.3°, 0°) along a white bar that was located 10 m in front of the participant and spanned 73.74° along the azimuth. (B) First person perspective within the VR environment during the auditory localization task. (The person in the picture is an author of the paper and gave consent to publish an identifiable image of him).
Figure 2. Results from all experiments. (A) Box-plots of the auditory remapping for all experiments. A significant improvement of the participants' auditory localization error wasand the localization test in a observed following the 60 s Audiovisual (AV) exposure. No such improvement was observed following the Auditory Only exposure. In the experiment on the effect of impact sounds, improved auditory source localization was observed following the synchronous audiovisual exposure phase with the additional impact related auditory cues (AV + Impact Sync). No significant remapping was observed following exposure to asynchronous but spatially aligned audiovisual stimuli (AV + Impact Async), or when the training was done in one sound and the localization was tested using a different sound (V + Impact Sync). (B) Mean pre- and post- adaptation localization errors for all participants, with each participant's data represented by pair of dots connected by a line. Asterisks indicate significant difference between pre-exposure and post-exposure phases (*p < 0.05) and “n.s.” indicates that there was no significant difference between pre- and post-exposure phases (p > 0.05).
The Plasticity of SPatial Sound Perception in Virtual Reality
In light of the difficulty in rendering accurate spatialized audio for users in virtual reality (by using individualized head related transform functions), in this study we investigated whether vision can assist in improving the perception of sound via brain plasticity. We found that exposure to visual stimuli and auditory stimuli that were related in time and space in the virtual environment improved the users' localization of sounds. Read the full paper here. See a demo of our paradigm here.
Paper Title: Generic HRTFs may be good enough in Virtual Reality. Improving Source Localization Through Cross-Modal Plasticity
Paper Abstract: Auditory spatial localization in humans is performed using a combination of interaural time differences, interaural level differences, as well as spectral cues provided by the geometry of the ear. To render spatialized sounds within a virtual reality (VR) headset, either individualized or generic Head Related Transfer Functions (HRTFs) are usually employed. The former require arduous calibrations, but enable accurate auditory source localization, which may lead to a heightened sense of presence within VR. The latter obviate the need for individualized calibrations, but result in less accurate auditory source localization. Previous research on auditory source localization in the real world suggests that our representation of acoustic space is highly plastic. In light of these findings, we investigated whether auditory source localization could be improved for users of generic HRTFs via cross-modal learning. The results show that pairing a dynamic auditory stimulus, with a spatio-temporally aligned visual counterpart, enabled users of generic HRTFs to improve subsequent auditory source localization. Exposure to the auditory stimulus alone or to asynchronous audiovisual stimuli did not improve auditory source localization. These findings have important implications for human perception as well as the development of VR systems as they indicate that generic HRTFs might be enough to enable good auditory source localization in VR.