publications

2025

1. AES

   Real-time Voice Conversion with Online Speaker Adaptation for Creative Audio Processing

   Anders R. Bargum, Stefania Serafin, and Cumhur Erkut

   In Proc. 2025 AES International Conference on Artificial Intelligence and Machine Learning for Audio, Late Breaking, 2025

   Video Website
2. IEEE VR

   Exploring the Impact of AI-Generated Speech on Avatar Perception and Realism in Virtual Reality Environments

   Anders R. Bargum, Emil Sønderskov Hansen, Cumhur Erkut, and 1 more author

   In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2025

   Abs HTML

   The representation of avatars as virtual human characters plays a pivotal role in shaping key aspects of the user experience in virtual reality (VR), including immersion, social presence, and overall enjoyment. While prior research has predominantly focused on the impact of visual and behavioral realism on the perception of such avatars, the influence of synthetic speech on avatar realism, emotional communication, and social presence remains insufficiently explored, both in offline applications and for real-time scenarios. This study systematically compares various machine learning-based speech synthesis and voice conversion algorithms with human speech across neutral and emotional stimuli to evaluate how different AI-based synthetic speech methods affect user perceptions of avatars in VR. Building upon existing literature, the investigation seeks to examine the impact of nuanced variations in synthetic auditory stimuli, with consistent timbre and speaker identity, while exploring the use of a potentially real-time voice conversion algorithm in an ecologically valid VR setting. Our findings provide valuable insights into the interplay between AI, auditory realism and avatar presence, with broader implications for live social VR platforms, therapeutic interventions, and immersive environments.

3. FADL

   Avatar Terapi: Når AI-og VR-teknologi giver stemme til det uhåndgribelige

   Anders R. Bargum, Simon Lajboschitz, Farah Shiraz, and 1 more author

   In Kunstig Intelligens i Sundhed, FADL’s Forlag A/S, 2025

   Abs HTML

   Kunstig intelligens (AI) er omkring os, i lommen på os, styrer store dele af vores opmærksomhed og liv. Det ændrer vores arbejdsmarked, vores adfærd, vores kommunikation og vores sundhed. Men hvor meget kan AI rent faktisk hjælpe os i sundhedsvæsenet? Hvordan kan teknologien konkret hjælpe patienter, sundhedspersonale og samfundet? Og hvordan gør den det allerede? Kunstig intelligens i sundhed giver et dybdegående indblik i brugen af AI i det danske sundhedsvæsen, med et virvar af lokale aktører, et snørklet offentligt sundhedsvæsen, store markedskræfter, og voldsom interesse fra patienter, udviklere og politikere. Ved at beskrive de mest prominente danske AI-løsninger giver forfatterne et billede af, hvordan en AI-løsning bliver til, hvilke udfordringer der kan være på vejen, og kommer med bud på sund og bæredygtig skalering og, hvordan vi opnår den bedste ressourcebesparelse med AI i sundhedsvæsenet.

4. APSIPA

   Unified Timbre Transfer: A Compact Model for Real-Time Multi-Instrument Sound Morphing

   Anders R. Bargum, Naotake Masuda, Bogdan Teleaga, and 2 more authors

   In Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2025

   Abs Video

   Recent advances in transformer-and diffusion-based deep-generative models have significantly impacted the field of music and audio synthesis. However, controllable and real-time interactive models, such as those used for timbre transfer in music production, remain largely dominated by auto-encoders and generative adversarial networks. In pursuit of efficient and flexible timbre morphing and multi-instrument timbre trans- fer, we propose a simplified modeling approach, utilizing an upsampled two-dimensional timbre space in conjunction with engineered and instrument-dependent excitation signals. Different from many other works, our model enables any-to-many timbre transfer with added control over timbre, pitch, and loudness. We additionally allow for seamless interpolation between instruments, eliminating the need for separate model training. Our evaluation shows performance comparable to specialized models, making it highly relevant for the broader creative audio community.

2024

1. Frontiers

   Reimagining Speech: A Scoping Review of Deep Learning-based Methods for Non-parallel Voice Conversion

   Anders R. Bargum, Stefania Serafin, and Cumhur Erkut

   Frontiers in signal processing, 2024

   Abs HTML

   Research on deep learning-powered voice conversion (VC) in speech-to-speech scenarios are gaining increasing popularity. Although many of the works in the field of voice conversion share a common global pipeline, there is considerable diversity in the underlying structures, methods, and neural sub-blocks used across research efforts. Thus, obtaining a comprehensive understanding of the reasons behind the choice of the different methods included when training voice conversion models can be challenging, and the actual hurdles in the proposed solutions are often unclear. To shed light on these aspects, this paper presents a scoping review that explores the use of deep learning in speech analysis, synthesis, and disentangled speech representation learning within modern voice conversion systems. We screened 628 publications from more than 38 venues between 2017 and 2023, followed by an in-depth review of a final database of 130 eligible studies. Based on the review, we summarise the most frequently used approaches to voice conversion based on deep learning and highlight common pitfalls. We condense the knowledge gathered to identify main challenges, supply solutions grounded in the analysis and provide recommendations for future research directions.

2. DAFx

   RAVE for Speech: Efficient Voice Conversion at High Sampling Rates

   Anders R. Bargum, Simon Lajboschitz, and Cumhur Erkut

   In Proceedings of the 27th International Conference on Digital Audio Effects (DAFx24), 2024

   Abs HTML

   Voice conversion has gained increasing popularity within the field of audio manipulation and speech synthesis. Often, the main objective is to transfer the input identity to that of a target speaker without changing its linguistic content. While current work provides high-fidelity solutions they rarely focus on model simplicity, high-sampling rate environments or stream-ability. By incorporating speech representation learning into a generative timbre transfer model, traditionally created for musical purposes, we investigate the realm of voice conversion generated directly in the time domain at high sampling rates. More specifically, we guide the latent space of a baseline model towards linguistically relevant representations and condition it on external speaker information. Through objective and subjective assessments, we demonstrate that the proposed solution can attain levels of naturalness, quality, and intelligibility comparable to those of a state-of-the-art solution for seen speakers, while significantly decreasing inference time. However, despite the presence of target speaker characteristics in the converted output, the actual similarity to unseen speakers remains a challenge.

2023

1. Springer

   Spatial Audio Mixing in Virtual Reality

   Anders R. Bargum, Oddur Ingi Kristjánsson, Péter Babó, and 3 more authors

   Sonic Interactions in Virtual Environments, 2023

   Abs HTML

   The development of Virtual Reality (VR) systems and multimodal simulations presents possibilities in spatial-music mixing, be it in virtual spaces, for ensembles and orchestral compositions or for surround sound in film and music. Traditionally, user interfaces for mixing music have employed the channel-strip metaphor for controlling volume, panning and other audio effects that are aspects that also have grown into the culture of mixing music spatially. Simulated rooms and two-dimensional panning systems are simply implemented on computer screens to facilitate the placement of sound sources within space. In this chapter, we present design aspects for mixing in VR, investigating already existing virtual music mixing products and creating a framework from which a virtual spatial-music mixing tool can be implemented. Finally, the tool will be tested against a similar computer version to examine whether or not the sensory benefits and palpable spatial proportions of a VE can improve the process of mixing 3D sound.

2. DAFx

   Differentiable All-pass Filters for Phase Response Estimation and Automatic Signal Alignment

   Anders R. Bargum, Stefania Serafin, Cumhur Erkut, and 1 more author

   In Proceedings of the 26th International Conference on Digital Audio Effects (DAFx24), 2023

   Abs HTML

   Virtual analog (VA) audio effects are increasingly based on neural networks and deep learning frameworks. Due to the underlying black-box methodology, a successful model will learn to approximate the data it is presented, including potential errors such as latency and audio dropouts as well as non-linear characteristics and frequency-dependent phase shifts produced by the hardware. The latter is of particular interest as the learned phase-response might cause unwanted audible artifacts when the effect is used for creative processing techniques such as dry-wet mixing or parallel compression. To overcome these artifacts we propose differentiable signal processing tools and deep optimization structures for automatically tuning all-pass filters to predict the phase response of different VA simulations, and align processed signals that are out of phase. The approaches are assessed using objective metrics while listening tests evaluate their ability to enhance the quality of parallel path processing techniques. Ultimately, an overparameterized, BiasNet-based, all-pass model is proposed for the optimization problem under consideration, resulting in models that can estimate all-pass filter coefficients to align a dry signal with its affected, wet, equivalent.

2021

1. IEEE VR

   Virtual Reconstruction of the Ambisonic Concert Hall of the Royal Danish Academy of Music

   Anders R. Bargum, Devansh Kandpal, Oddur Ingi Kristjansson, and 3 more authors

   In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2021

   Abs HTML

   In this paper we present a simulation and validation of the Ambisonic concert hall at the Royal Danish Academy of Music (RDAM), which was inaugurated on September 19th, 2019 in Copenhagen.Validation with experts showed that the simulated ambisonic room was perceived to match the acoustic- and spatial attributes of its real counterpart.