The 40m screen at Orbi requires a 22-source sound system.

Looks like Bristol-based Films at 59 might have invented the art of mixing audio for VR over five years ago based on their 360° work at Orbi in Japan.

Back in 2013 a new visitor attraction created by SEGA and BBC Earth was officially opened to the public in Tokyo. It was seen as a world’s first in entertainment, by mixing SEGA’s technology with BBC Earth’s nature content to immerse visitors in the ‘wonders of nature’.

Their official release describes Orbi as “taking people on a multi-sensory journey, placing visitors in the heart of nature and inviting them to explore with sight, smell, touch and sound. Twelve different nature zones create experiences that would be impossible in the real world.”

There are nature zones that lead up to a huge main theatre where bespoke nature films created by BBC Earth Productions play on one of Japan’s largest screens. When we say large we mean spanning 40m across and eight metres high. The main theatre’s screen is curved to surround the audience, but more pointedly there is a 22-source sound system which beat Dolby ATMOS to this multi-dimensional punch. But what that sound system also did was to give Films at 59’s Chris Domaille the knowledge and experience to take sound mixing from Orbi into the mixing for their own VR projects.

Chris said, “For the last five years we’ve been delivering films on a 40m widescreen that has also got two rear screens. That has been an immersive experience, a bit like Dolby ATMOS before it was invented. We’ve overhead speakers, so we had to build a bespoke set-up for it. So really we’ve been doing this immersive, VR-type audio mixing for five years and have already ironed out the problems and decided on the solutions. None of which are really that revolutionary; it’s not so much a VR mix but an immersive one.”

Part of those solutions included using a gaming engine to build spatialisation into the mix. “What we subsequently did was to then build the theatre in the Unity game engine so we could work out how all the audio spatialisation was working. The theatre itself is a 22-channel audio system; they’ve got other systems there, one of them is a 36-channel audio system. They are 4D theatres so have smoke machines and scent machines and have transducers built into the floor so we can make the room vibrate slightly. All of a sudden we’ve just found ourselves converging onto this broadcast world, it’s like the broadcast world and the gaming world are starting to converge slightly.”

Google Daydream

Films at 59 started getting involved in a number of 360° shoots for YouTube. “We wanted to try and introduce ambisonics into the YouTube phenomenon. From April this year especially on the Spotlight series, part of the Google Cardboard series, they’ve allowed us to put ambisonics mixes in which means if you have a Google Cardboard or Google Daydream all the sound pans with your head movement. Before, if you watched 360° video the sound would be static so when you moved your head the sound stayed the same and that’s not what happens in the real world. So we’ve been delivering these ambisonic mixes.”

Chris did a job for a band called Flatbush Zombies who wanted a 360˚ film for their new video. “What we did was to take all of the individual feeds in the studio and then position them before tracking their movements. It’s fantastic as when you put on your headset we start out in mono, and the picture’s very small and 4:3, an old-fashioned sort of TV style. The picture then opens out so you’re in your spherical world and as you move your head all the sounds are panning just as they would in the real world. So you have this captivating feeling of being there because what you hear is what you’re looking at. Of course, that is what the game world has been able to do for a long time where if you get closer to a source it gets louder.

“I’m really excited about audio at the moment because where I’ve been a sound mixer delivering for a flat screen for years, now I’m looking at an environment where the mix is infinite and it depends on where I am in the environment as to what that sounds like.”

How To Spatialise

For the Google Spotlight videos on YouTube there are various tools that you can use to build your spatialisation. There’s a company called 3Dception (www.twobigears.com), recently bought by Facebook, who have released some software which allows you to build your project. There’s also software from a company called Audio Ease (www.audioease.com) which allows you to take in an ambisonic mix. Chris said, “Let’s say I was recording an orchestra, I would take an ambisonic microphone that would be the basis for my sound. In my world, because I’m so used to post-producing things, what I’d really like is a lot of microphones so if I’m in with a string quartet then I would take the feeds from all the microphones. If I’m in the middle of that quartet, then I can pan those microphones to those positions. So that means when I have a headset on which allows me to spatialise that audio as I look at the different instruments I get a slightly different mix. I hear more of what I’m looking at. Which is kind of what happens in the real world, if you hear a sound coming from the right, you look over to the right and so you hear it clearer as both your ears are focusing on it.”

Ambisonics is a 1970s technology but at the moment YouTube are only allowing first-order ambisonics on their 360° videos, so encoding into four channels, W, X, Y and Z. This is called ambisonic B-format. It might be second-order ambisonics in a year or so. “We’re looking at a technology which has only been out since April – actually it’s been around since the 1970s but it’s only really been implemented in this way, this year.”

ORBI

VR Audio On A 40-Metre Screen