Today, as binaural spatial audio has taken hold with offers from Apple and Sony and even the ill-fated Bose Frames experiment, we can see there remains an appetite for more immersive audio experiences. When we started Kinicho five years ago, it was with a vision built on the inexorable rise of headphone-based listening that binaural spatial audio would become a format to complement stereo but could also replace it as the de facto audio standard of the 21st century.

The inspiration for our approach started over a decade ago. We had an active production interest in both Surround Sound for Cinema and for Music in general. As producers, we had a need for headphone-based monitoring tools that would help in the studio but also saw how the same approach could improve externalization/virtualization for the end-consumer. But no matter what solutions we tried - and we even tried to come up with our own - we were disillusioned with the quality that the Head-Related Transfer Function (HRTF) approach could deliver.

“Why did all binaural audio have artifacts and sound so brittle?” we would ask ourselves. This was a question that lingered with us for five years.

In that time we did a lot of thinking about spatial audio and a lot of tinkering with different techniques to try and improve the final signal quality. We had some OK results, but never satisfied that quest for quality. Then, in 2016, we started to look at spatial audio for Virtual Reality (VR). In VR, audio-quality was important but less so because of how it worked with the other stimuli and it was still in its early stages. Instead, speed and latency were the biggest issue. There was detectable divergence between head movement and audio changing in response to the motion. When we started to look at latency issues in more depth, we came to a more general conclusion about binaural spatial audio: it wasn’t one thing that caused any of the problems we were encountering all this time, it was a combination of all the engineering components in the HRTF process.

These problems start with the logic of the emulation approach – the use of Impulse Responses (IRs) to emulate how sound is heard at the ears - and it runs through to the final signal processing of the source audio when it is convolved with the IRs. These IRs only contain directional information. This really challenges the notion of this being 3D or spatial audio in this context; distance effects must be included in the sound design prior to spatialization. Similarly, the capture of the IRs and the interpolation model, which determines the size of the sample buffer required, affects the angular resolution that is possible in the system.

In turn, this limits the spatial definition so that in most commercial applications the spread of any sound is between 45° and 90° with audible transitions around 5°-15°. This lack of definition creates a requirement for the audio to be convolved with multiple IRs, this causes artifacts and comb filtering that degrade the quality of the audio.

Finally, but not least, the end-to-end audio process is computationally intensive and is consequently the primary cause of latency in the system. In head-tracked applications, latency is a big issue when the motion-to-audio latency is over 30ms.

While this is only a high-level summary of the problems with HRTF-based binauralization, it highlights how the components and limitations interact with one and another to cause the problems we describe!

After our research efforts, we concluded that the HRTF process is not fixable. While components of the process can be improved with incremental innovation, at the end of the day, the emulation of the system as a "‘bubble" of virtual speakers around the listener is not a practical imitation of human hearing. Instead of being constrained by the engineering limits of the HRTF-based approach, we speculated that a more radical technique diverging from the state of the art would ideally be constrained by the human auditory capability. To explore this proposition, we went back to first principles to consider what is happening in the physical world as listening spatially is our natural disposition.

Our investigation led us to more fully understand the problems caused by HRTF and identify a solution, a high-performance physics engine for spatial audio that we call the Sonic Reality Engine®. The engine is based on a very efficient algorithm that allows us to simulate soundwave propagation in a virtual volumetric environment to produce audio that is more akin to natural listening.

This simulation algorithm allows us to account for how sound gets from an emitting object to ear, accommodating obstructions and reflective surfaces that might shape the listening experience. For example, the first obstruction any soundwave encounters is the medium it is traveling through, ordinarily this is air but we have also simulated water. Reflections processed by the engine are volumetric because simulation is volumetric.

And because everyone has a unique set of ears, especially the width between the two ears, we discretely also treat the virtual ears volumetrically so the listener can get a more precise simulation of the effects described by Rayleigh's Duplex Theory. The theory provides an explanation for the ability of humans to resolve sound spatially, comparing the differences in what each ear senses.

This ear-centric approach allows users to get personalization without the need for expensive and difficult-to-access personalized HRTFs – it’s a simple, easy-to-use single parameter where the changes can be heard in in the audio in response to a change in the parameter.

The ultimate purpose of this simulation is to produce a set of coefficients that can parametrize our analog-modeled, mastering-grade filters to deliver a HiFi-quality response to the listener in a single pass.

The engineering of the simulation and DSP components are super-optimized, making the entire process very efficient. Spatializing with this approach uses a much smaller number of processing instructions than with HRTF-based systems – around 5%. While the main benefit of this efficiency is significantly lower power consumption in mobile applications, our algorithm also ensures spatial audio can run on a wider range of targets than ever before.

The same algorithm can be deployed to fixed-point embedded systems including DSP codecs, SoCs, and ASICs, as well as floating-point applications in software for other devices from smartphones, on the desktop and even in the cloud with our serverless infrastructure. Because of this efficiency, the Sonic Reality Engine is the first spatial audio processor that can run everywhere and deliver the same flawless result.

As a user experience, this efficiency also means zero-latency, which for head-tracked applications means real-time responsiveness; for gamers or in industrial applications this can massively improve the user’s reaction times. The standard threshold for detectability is 30ms. When the latency is below this threshold, the experience is perceived in real-time. With our Sonic Reality Engine approach, motion-to-audio latency of less than 5ms is possible where there is no communication barrier between the Inertial Measurement Unit (IMU) and the audio processor.

Typically, this would be where the audio is being spatialized on an embedded system when the processor and the IMU are on the same integrated circuity or in a tethered scenario when data from the IMU is delivered over USB or other network to parametrize the simulation and the binaural audio returned to the ear over cable.

For TWS/Hearable applications, stereo can be spatialized on the hearable device after wireless transmission has concluded. The virtualized/externalized spatialization can happen in real-time, which is a better experience for the listener whether they’re consuming Film, TV, Music, or other content.

For information-based services, such as personal assistants (for example, Siri, Alexa, and Bixby), or audio navigation and so on, two channels of audio can be used economically to create the perception of many audio objects and give the user a richer experience that responds to head motion in real-time. Kinicho has a proof-of-concept application of this in voice-search; the user asks for coffee shops nearby and the audio feedback lists the results using spatial audio to guide the user’s attention in the right direction to support them in making choices and getting to the right destination.

Latency is primarily an issue for TWS applications where there is a need to process multi-channel audio. In this scenario, the multiple channels cannot be delivered over-the-air because of bandwidth limitations and the audio processing must still be done on another device, such as a smartphone. However, this adds a lot of extra latency to the process. For example, we measured the latency of Apple’s Spatial Audio at over 200ms, well over the threshold for detectability. This is because three processes are picking-up latency. In the first instance, the IMU (inertial measurement unit) data is transmitted from the AirPods Pro back to the iPhone; then, the audio is spatialized into a binaural signal picking up more latency; and finally, the audio is streamed back to the AirPods Pro adding even more latency.

The efficiency of the simulation algorithm affords us high spatial definition. Efficiency allows us to use very small sample buffers and when combined with 7th Order Spherical Harmonics and a novel approach to convergence, we’re able to reduce the spread to a few degrees with audible transitions approaching 1°. And because the simulation engine is truly volumetric, distance is functionally part of the spatialization process. Ultra-high spatial definition comes as standard for even the most compact of applications.

Beyond real-time applications, audio quality is a big issue for the user experience. This is best exemplified in music applications. If the user is switching between stereo and multi-channel they will likely notice the degraded quality in the latter. In our experience, there isn’t an HRTF processor that doesn’t add artifacts and make the audio sound brittle or colored in a negative way. There’s a strong likelihood that faced with this choice listeners will opt to retain the HiFi-quality of stereo - especially now that lossless streaming is a reality - and reject the diminished experience with HRTF-based spatialization of Spatial Audio conten.. This would be a disappointment for the whole industry now that spatial audio in headphones is having its big moment. We’re hoping the timbral integrity of the Sonic Reality Engine® will establish a HiFi standard for spatial audio.

This is standard that content creators and producers need to power better monitoring tools to use when they’re working in headphones for surround and spatial audio with formats like Dolby Atmos, MPEG-H, and Ambisonics. This is especially important when they are working on content that will be delivered to the end-user as a binaural spatial audio. There’s some incredible work being delivered in Dolby Atmos equipped studios – and it probably sounds incredible in the listening room. But do producers really understand what will sound like for the end user when they’re listening on headphones?

While there has traditionally been an acceptance that consumers will accommodate a poorer experience on costs grounds, it has been conventionally the case that audio professionals work on the best pro-equipment during the production and mastering stages so they can create the best listening experiences. This is where Kinicho started, we wanted a solution that offers the best timbral integrity possible, a solution that matches the ambition for high production values. That’s now possible for other creators and producers to get this.

Over the next 12 months we’re hoping to see some exciting applications of the Sonic Reality Engine® as we see the IP used in hearables, powering immersive experiences in XR and changing the way we monitor in the studio. In the longer term, we’re hoping to see productions move to volumetric object-based audio too. The step-change in performance that we're delivering is another stride on that journey that began in Paris 140 years ago to give consumers more life-like listening experiences.

audioXpress | Voice Coil | LIS 

audioXpress features great articles, projects, tips, and techniques for the best in quality audio. It connects manufacturers and distributors with audio engineers and enthusiasts eager for innovative solutions in sound, acoustic, and electronics.

Voice Coil, the periodical for the loudspeaker industry, delivers product reviews, company profiles, industry news, and design tips straight to professional audio engineers and manufacturers who have the authority to make powerful purchasing decisions.

The Loudspeaker Industry Sourcebook is the most comprehensive collection of listings on loudspeaker material in the industry. Purchasers and decision makers refer to the guide for an entire year when making selections on drivers, finished systems, adhesives, domes, crossovers, voice coils, and everything in between.