Whether it’s taking a call on a busy street, listening to music on a commute, or using open-ear devices on the move, one challenge remains constant: maintaining clarity in noise.
Some solutions attempt to reduce background sound. However, due to technical constraints in open-ear designs, they often fall short of fully restoring lost content. Mimi Noise Adapt was developed to address this gap. Rather than focussing on removing environmental sounds, it focuses on the target signal itself, restoring the components that are masked in noisy environments. This results in clear speech and rich audio experiences.
We spoke with Vinzenz Schönfelder, Research Lead, to explore the thinking behind Noise Adapt, the limitations of traditional approaches, and what these new innovations could mean for the future of audio experiences.
1. The problem we’ve been misunderstanding
Was there a specific moment or insight where you realized this was the problem to solve?
In fact, we first discussed the original idea for this product feature many years ago, when the first true wireless products came to the market and consumers started to wear them more regularly during everyday activities. We realised that Mimi's hearing loss restoration algorithms could also be applied in cases of "situational hearing loss", i.e., when a listener was unable to perceive the signal as intended because of their current acoustic environment. In certain aspects, the challenges they face are similar to the consequences of clinical hearing loss: low intensity signals remain below the threshold of perception, while high intensity signals remain clearly audible. Furthermore, both effects are typically more prominent in some frequency regions than others. It only seemed natural to try and apply Mimi's solution for hearing loss to listening-in-noise scenarios.
Why do you think this specific listening problem wasn’t being solved by existing approaches?
In most TWS devices, the challenges of listening-in-noise are being solved by managing the environmental noise, using ANC in combination with passive attenuation. More recently, however, new form factors started to enter the market – particularly open wireless headphones, as well as hearing glasses, with little or no acoustic isolation of the ear canal, which severely constrains the ability to control environmental noise. The only "solution" that is currently available to users is increasing the overall device volume, which may still not fully restore audibility while unnecessarily increasing sound exposure.
Why does audio clarity still break down in real-world environments when using volume control to overcome background noise?
On the one hand, depending on the form factor, there are technical limitations for the audio signal level that an open device can produce before technical sound reproduction artefacts occur.
Second, from a perceptual perspective, increasing volume does not necessarily improve clarity. Auditory filters that help our brain distinguish and separate sounds at nearby frequencies are becoming less sharp at higher signal levels. Thus, counterintuitively, increasing the sound volume will eventually decrease signal clarity. Applying gain is only necessary and beneficial when it actually contributes to unmasking the target signal.
With Mimi Noise Adapt, the amount of signal amplification is applied in a level- and frequency-specific manner that is continuously adjusted based on the characteristics of the noise as well as target audio signal.
What’s the biggest misconception about how we perceive sound in noisy environments?
In many cases, a significant proportion of the signal remains audible despite the presence of background noise. There is no need to amplify all components of the signal, as you would do by increasing device volume. It's only parts of the audio, mostly low and medium intensity components in specific frequency regions, whose audibility is truly impaired and where an intervention is required.
2. Why existing approaches fall short
Why do you think traditional approaches focused on increasing volume or broad adjustments instead of addressing masking directly?
Raising the volume is an intuitive and technically simple approach. Manipulating the signal on a more fine-grained level requires significantly more complex signal processing algorithms, which in turn substantially increases the risk of creating undesirable processing artefacts. This may have prevented the adoption of more refined solutions similar to Mimi Noise Adapt.
How does this loss of signal show up in everyday situations like commuting or taking calls?
The most obvious negative effect in everyday use will be problems in understanding speech, whether in calls or when listening to podcasts, for example. But also when listening to music, some artistically valuable components may be missing, limiting the overall experience.
3. The breakthrough behind Noise Adapt
From a signal processing perspective, what makes restoring clarity in real-world noise so difficult?
Any processing approach that differently treats individual components of an audio signal tends to generate undesired side-effects in the form of acoustic artefacts. Even using a simple static compressor or a frequency-specific EQ presents such a risk. On the other hand, noise is highly variable which calls for a dynamic approach to optimally restore signal audibility. Therefore, providing a clearly audible benefit without generating unpleasant acoustic distortions is not a trivial feat.
At a high level, how does Noise Adapt approach this problem differently from ANC or a traditional EQ?
Relative to an ANC, we accept that background noise can only be controlled up to a certain level. We actively embrace the fact that masking can not be avoided and therefore focus on treating the target signal to overcome the effects of the background noise.
Relative to an EQ, we recognise that noise is dynamic, and that the actual masking effect depends both on the noise and the current spectral levels of the signal. In this way, we are able to treat the signal in a more targeted way while restoring audibility.
How do you improve clarity without compromising the naturalness of the audio?
This is where we benefit from Mimi's background in hearing loss restoration. Our DSP architecture mimics the effects of a healthy auditory system. Apart from being able to naturally counter the effects of cochlear hearing loss, it also allows us to treat the signal in a frequency- and level-dependent manner while avoiding the occurrence of unpleasant acoustic behaviours or artifacts of standard dynamic range compression architectures. To borrow a term from audio engineering: We are striving to make our processing completely "transparent" – remaining unnoticed by the listener – all the while it is working hard to maximise signal audibility and clarity.
4. From research to real-world impact
What should teams consider when integrating Noise Adapt into their audio stack?
The processing used for Noise Adapt is based on the research and technological foundations of our sound personalization technology, which we have heavily optimised to reduce computational requirements. Mimi's processing has already been ported to various SOC platforms and we intend to offer Noise Adapt similarly widely. In terms of physical design, the microphone position should be chosen so as to avoid feedback from headphone speakers to the microphones that capture the environmental noise.
In which scenarios does Noise Adapt make the most noticeable difference?
The main user benefit will appear in devices that are built for everyday use and where controlling the background noise is not possible for technical reasons or not desired, such as open-ear form factors (OWS devices), hearing glasses or smart glasses, where speakers are usually installed along the frame.
What feedback have you received from early partners or testing so far?
Partners who have tried our early software prototypes are enthusiastic about the technology. Once you've tried it out, you will experience a clearly noticeable benefit, particularly in situations that pose a significant acoustic challenge for open-ear devices in everyday use cases. The benefits experienced have ranged from improved speech intelligibility to increased music enjoyment.
5. The future of adaptive audio
How do you see adaptive audio evolving over the next few years, especially for open-ear and wearable devices?
At the moment, even experienced users are sometimes overwhelmed by all the features and settings provided in modern headphones. Similar to the way our mobile phones have developed into personal computing devices, headphones become increasingly powerful and complex. However, this also means controlling and managing another digital device, requiring user attention and effort. At Mimi, we work towards solutions that reduce that additional effort by building intelligent control mechanisms that solve real-life problems without requiring manual user intervention.
Do you see a shift from reacting to noise to understanding its impact as a long-term direction for the industry?
Adapting audio playback to the individual situation of the user can be considered a much broader concept than the specific use case of Noise Adapt. When music (or even high quality speech material) is mixed and mastered for distribution, studio engineers often have the perfect listening situation in mind: the listener will be consuming the audio recording in a quiet room with expensive HiFi equipment, with full focus and attention, no disturbance and perfect ears.
In reality, though, this listening situation will be the exception rather than the rule. Instead, the equipment, situation and state of the listener will vary dramatically. This observation strongly calls for an approach where the signal is individually "re-mastered" during reproduction to match the individual listening situation, without distorting the experience in any undesirable way.
In terms of technology, we have now reached a point where consumer audio devices are becoming seriously capable of fulfilling that vision.
So far, Mimi offers solutions for two aspects in that domain:
- Personal hearing ability: mitigated by our Sound Personalization technology.
- Limitations in audibility due to everyday background noise: which we now tackle with Noise Adapt.
However, these are but two out of many more aspects that influence our everyday listening experiences. We expect that the benefit of individualised sound reproduction in consumer audio will be recognised and embraced much more widely than is currently the case.