Paul White
Welcome to the Sound On Sound recording and mixing channel podcast. I'm here with Nuno Fonseca of SoundParticles, and he's going to tell me how SoundParticles came about. Because from my perspective in the music industry, you just appeared from nowhere with these rather interesting 3D manipulation plugins.
Nuno Fonseca
So essentially it all started with a crazy idea like 15 years ago. I also love cinema and at some point I realized that the most interesting visual effects that I was seeing on movies, all of them use particle systems. That is a technique where they create thousands of points to create fire and smoke and desert storms and fairy dust. And I thought, wouldn't it be nice to do the same thing with sound and have thousands of small sounds around you that would create these epic soundscapes.
But at the time, it was only an idea. And then in 2012, I had finished my PhD on a different topic, and since no one was using particle systems for sound, so as a computer nerd that I am, because this face doesn't fool anyone, I decided to, okay, let's create my own software with particle systems and be able to generate like thousands of sounds around you.
And that would be interesting for soundscapes. Um, and that was the beginning. Two years later, I went to a LA to a conference, but before going to the conference, I decided to send five or six emails to some professionals say, okay, I'm doing this. I think this could be particularly interesting for big productions with a lot of explosions, that epic scenes.
And I'm going to be in town in two weeks. So if you'll, Want to know more, just let me know. And the first reply that I've got was from Skywalker, invited me to go to Skywalker ranch, do a presentation there. And then within six months, I end up doing presentations at Wonder Brother, Universal, Paramount, Sony, Fox, later Disney, Pixar, Blizzard, Netflix, Apple, and many other places.
And pretty much that was the beginning of Sound Particles. And we, okay. Starts releasing sound particle software. That idea was to use this kind of computer graphics visual effects kind of approach, but to sound. And then we started mainly with television and cinema. Our software is using things like Game of Thrones, uh, Frozen 2, Star Wars, Dune, uh, in video game companies like Blizzard, Epyx.
And then at some point we said, no, no, now we want to start also playing with music and bring our technology to music and we start releasing our plug ins in that area.
So the first things you brought into music were really spin offs from what you were doing for the film industry, which is, uh, positioning things in three dimensional space. Yeah, essentially we have a Several blocks that we could reuse, like panning in 3D in pretty much any format from stereo to 5.1.
There'll be atmos, ambisonics, up to six order and all of those things. And essentially we say, okay, now we have this blocks. What kind of things can we do to, to um, to use this also with music? Because for a long time we wanted to start to play things with 3D. And with music and one of the, the first plugins that we made specifically for music was energy banner.
The idea of having a banner, but controlled by the intensity of the sound. And actually it was an idea that I had as a, one day I was driving to the office after leaving the kids at school. And I was. Listen to some music. And I thought, okay, it would be nice if I, if the drum starts to move around when they have this, the, the, this beat and drums starting to move and say, okay, why can we, why not doing a, a banner that is controlled by the intensity of the sound?
So we, we create energy banner and, With this idea of a panner, that pretty much is almost like a compressor, but instead of using that to control the gain, uh, we use them to control the panning. And then we decided, okay, why, why can't we do the same thing, but instead of using intensity of sound, Why not use, uh, the frequency content.
So we create brightness banner that is based either on the brightness or on the pitch of the note. So when you have, for instance, an arpeggio, you start having the notes moving around depending on the pitch. Um, and that was the, our initial plugins in the music domain. But then you came out with control from the position of a mobile phone, which seems to be going back to your film roots again.
Yeah, we, of course, we are, uh, we have a big passion for special, uh, audio. Um, and we want everything to do with special audio. And one of the things that we always thought that was almost like a kind of ending. Cap is that we were not satisfied with any kind of way to control three depending because, okay, you can use knobs for stereo.
They are perfect for stereo. When you start working with surround knobs are a little more complicated, eventually joysticks, but if we, if you want to actually work in 3d and do some Crazy or complex movement using knobs or joysticks or touchscreens or a mouse any any of those situations actually Made us very happy.
So they say okay, how can we do this and the idea? Okay Why not use a phone and using the the motion sensors of a phone to simply point on the direction? We want to sound if you want the sound to be Coming from a particular direction, simply point the, the phone in that direction. So it's like a laser pointer that you can create interesting movements, or if you can go crazy and even use two smartphones, one in each hand, one for left, another for right, and you start doing these crazy things.
Strange choreographies, panning things around and yeah, even if you are using in stereo, our, uh, the idea behind our plugins is always to do, okay, even if you're working with stereo, you still can use this a lot and, uh, having, uh, a lot of fun and interesting things in stereo, but if you're using surround or even using immersive, it's even better.
And, uh, for instance, I was just talking with Alan Myerson. Uh, a few, two weeks ago at NAMM, uh, and he was telling me that, okay, he uses space controller every single day because for positioning sound, it's much more interesting. It's much more fun and it's much more creativity than actually using knobs or touch screens.
But of course we have a biased opinion. It is a very intuitive way of matching sound position to visuals. It's simply panning, it's in pointing and you get the sound there. And if you want to do. crazy things, or even if you simply want some movements like having things like small movements just to bring more life to your mixes.
I think it's, it's fires much more, much more of the creativity than using simply knobs or joysticks. So before we move on, I understand that you're also involved in a project with East West a while ago, which was something very different, the text to singing engine. How did that work? We started Very long time ago.
It was in 2001. Um, so East West had just released, um, uh, some library called Voices of the Apocalypse. It was on the time that Gigastudio appeared. It was the beginning of the, uh, Samplers by software and essentially it was a sample library with voices from choirs. You have all the vowels, all the consonants completely separated and pretty much allow the user to, Okay, I'm going to use the vowel A and you play A, A, A, A, A or E, E, E, E and pretty much allow you to And, and at that time, once again, I always like this crazy idea and try to think outside the box.
And I sent, uh, them an email saying, okay, have you ever considered having a program where you could actually write the text that you want the choir to sing? Uh, and that was the beginning. So I started creating, uh, this software, uh, people will be able to write the text like hallelujah and all of the things that you want the choir to sing.
And then essentially. When you play the MIDI note, you would fire the right samples to make sure that if, for instance, you write the word love, okay, and you start, okay, press the MIDI note and you get, okay, starts the L and then starts the O and then you stop the L and old, the old, and then on release, you would stop the O and start the V.
Um, so that allows you to actually, uh, be able to write the text before and then. As you play, you start to have the choir singing. It was a lot of fun, uh, especially because on the beginning, I find out that people were having a lot of fun, uh, having a symphonic choir singing dirty words, uh, but then the things evolved and actually that became, became word builder, which is today still used on East West products like Hollywood choirs.
And many other, uh, choir products from East West, uh, that allow people to write the text and then, uh, having the, the computer to singing that text depending on the MIDI notes that they are receiving. It's certainly a very sophisticated thing to develop that long ago. Yeah. And it was interesting because we start adding new things like, for instance, an English to phonetics dictionary.
To allow people to write text in English. And then the software automatically know how that word should be pronounced. And we, you also would be able to edit the, the, um, all of these things. Like if you want a consonant to stay longer or be more stronger or more weaker. Uh, uh, and you are, you were able to almost like have a piano role, but with piano.
With the, the, the several phonemes that you want to, to tweak to make even, uh, uh, have much more interesting. And it was, uh, quite rewarding to, to see that, uh, project, uh, being used by a lot of, uh, composers around the world. So yeah, it's. Something quite interesting that then leaves, left me to my, uh, PhD that was to take that even further with having the computer re synthesizing everything.
So imagine I'm singing something to the computer and then the computer will extract all the, the melody, the dynamics, the text that I'm being sent, and then simply. Resync that with a different, different, uh, sound library. So it was a very interesting project that I spent a few years before starting Sound Particles.
The other new product you've got is, uh, Density, which again, seems rather different from anything that, uh, I believe you've done before. And of course there are lots of double tracking programs around and detuning tricks, but you seem to have gone further. Yeah. Once again, the idea of people try to.
Increase the number of instruments and the number of voices. It's not new. We can see that even chorus. It's pretty much the first take on that approach when you try to create multiple instruments based on the single instrument. What we have done was simply, okay, let's try to do something almost like chorus, but with a much more modern approach using, uh, granular synthesis, also to be using 3D spatial, uh, audio, if you want to use immersive audio, um, and using particles.
So the idea of density is to simply generating several fragments based on the original audio input, and by having these additional fragments with, uh, Slightly pitch shifting versions of it, slightly delayed versions of it to be able to create, uh, more ensembles. And either if you have a solo voice instrument that you want to convert into ensemble, or if you already have an ensemble, you simply want to add much more, some additional layers, uh, to it, it allows very interesting results because When you think, when you talk about special audio, and especially now with Dolby Atmos for music and all of those things, I think that sometimes we are targeting it wrong, because we almost position these as a pending tool for mixers.
And for me, uh, special audio for music should be, uh, Creative tools. It should be things that the artists, the composers, the producers, the musicians work directly with special. It's not something that, okay, let's record an album down. In the end, a mixer will add some 3D features to it. Now, I think it's much more interesting if we start right from the start, right from the creatives to use, uh, tools to explore this special audio.
And that's the reason for us. Having all of our plugins like energy panel, space controller, now with density to have more ways to explore special audio, uh, for music, but in a creative way and by creatives, by artists, instead of simply look at it as a mixing tool for mixes later on during the final mix.
What format do you think is going to be the one that wins over the consumers, because everyone's still listening, or most people are still listening on stereo headphones. Does that mean that everything needs binaural, or is there something better? From the practical point of view, we know that most people will not be able to afford the 20 speakers for their living rooms.
Yes, a few, very few, may actually do that. Or, yeah, a few may have some interesting high quality soundbars. But we need to face the fact that most people will continue to use headphones to listen to music, to listen to, uh, their series on Netflix, on their iPad, or to play video games. So, I believe that the future, more and more people will continue to use headphones.
In the past, uh, over the last years, headphones continue to grow in terms of, uh, percentage of use. So I think that special audio must be done with headphones. Nonetheless, we still have a problem today. To, to fix because binaural sound is not yet on the, where they should be nowadays, when you select, if you want special audio or without special audio, pretty much we are selecting, if you want some special and more special related features, but eventually with some costs in terms of equalization and sound quality, uh, or if you want a better sound quality, but in stereo, but without a special audio.
So I actually believe that we have. Probably one, two years to fix, uh, to resolve these binaural and be able to get high quality binaural sound. There's actually one of the things that we are also doing at Sound Particles. We are trying to create our own binaural technology because I confess that I'm not fully satisfied with all the existing solutions out there.
Even the personalized ones still are not to the spot where they should be. So I think that we have Pretty much two years to fix this. If we are not able to get a very good binaural solution within two years, probably this hype about spatial for music could eventually decrease and be like a temporary fashion.
But I believe that if we are able to get these binaural things right, it's going to be definitely the future of music because you get this sense of 3D. And once again, sometimes people think that It's to put guitars moving around you and behind you. Now, even if you are doing, listening to classical music and everything is on the front and static, the simple fact that you add these reverbs and these reflections to additional directions besides left and right, it gives you a much more interesting experience and it's for that that we are also proud.
Working on to try to come up with some solution to make sure that people get a better solution for music with spatial audio. Well certainly for my own music I've been working in a combination of traditional stereo and binaural where the binaural process parts are probably not the key parts but they're the bits that make the mix interesting and that way the the lack of precision in the way that people perceive it is probably less important because we still have this problem that everyone has a different physical set of ears.
And no one binaural solution will fit them all. Yeah, and even we are doing a lot of things from computational simulations of ears, almost like the same way that we use acoustic software to, uh, see how it's going to be the acoustics of a, of a concert hall that is going to be built or something like that.
We are trying to use the same approach. process just to analyze. Okay, I have this scan three D of this person. Uh, how is actually the HRTF of that? But by now it's a very complicated problem. It's something like it's almost like a chain that breaks by the weakest links. And actually we need to fix several problems.
You need to get the perfect HRTF. We need to be able to also manage the equalization from it. Add font, uh, uh, eventually adding a track. Uh, so there is a several problems that if, even if you get one of them, very good, very well handled, uh, it's going to break by the weakest link, because if you have something that is not very good.
handled the binaural itself is going to, to break the experience. So, uh, yeah, I really think it's going to be the, the, the future. Currently, pretty much all, all the companies are, uh, are working on it. It's pretty much the Holy grail of the current times because everyone from Dolby to Apple. Apple, Facebook, Microsoft and many other companies, Genelec and others are trying to to handle this.
But it's also very, very complex problem to fix because you need to mix computer vision to be able to personalize the ears off someone because people will not be able to go to under quite chambers to have them, uh, their exactly have captured or They will not use the 3D scanners, professional 3D scanners.
So there is a lot of things that, uh, needs to be done. And, uh, from the engineering point of view, it's a very, very challenging and complex problem. Uh, probably the biggest problem that we, uh, I face on my, uh, all these years working in the audio industry, but we also want to give it a try, want to try and see if we can come up with an interesting solution that can help anyone enjoying even more music.
It is a difficult one, especially things like head tracking, because if you're playing a computer game, then maybe head tracking is useful. If you've got, especially if you have a very wide screen. But if you're listening to music, walking to work and you're looking around at the traffic, maybe it will be disorientating having the music shifting around all the time.
So it's something that might suit some situations better than others. Yeah, definitely. Uh, uh, although eventually the tracking could actually even have the opposite effect of actually, uh, instead of having, Everything mixed together because if I'm using binaural without a tracking, if I've moved my head, everything comes with me.
Uh, and if I use a track, everything stays locked and I simply move my head like I would do in the studio. So yeah, it's even a tracking is one of the things that we are also doing. Trying, uh, different things to see, uh, how it's going to be the end result, uh, of it with all of this, uh, technology. But, yeah, that's, it's interesting because it's like we are on the pioneering years of this special audio.
There is so many things open up, so many, uh, ways that we can use, uh, Uh, special audio in a technical, uh, way. Um, and even we are working on some projects that we are not, uh, we cannot announce it yet, but it's very interesting, creative ways for musicians and artists to start using with special audio, because like I mentioned, for me, it's about.
Having creative tools, uh, imagine, for instance, if stereo, it was only limited to have a panner and no plug in support, uh, stereo, everything was only mono, it would be much more dull, uh, world that we would live nowadays, but the fact that every plug in support stereo creates interesting things in stereo.
Okay, let's try to do the same thing and bring more, uh, special features into the audio software. So, I know you can't tell me about all the things you're working on in the future, but are there any, uh, directions that you're particularly interested in exploring? I've been playing around with quite a lot of these granular effect plugins, granular delays, which are very, very interesting.
But if you could use your 3D positioning to put the grains in different places and get a sort of cloud of effects that surround you, that could be a very interesting thing to listen to. Yeah, we are working on several fronts. Um, one of them is. We are now working for almost three years now on the third generation of sound particles.
And until now, uh, sound particle software was pretty much a sound design, um, software for these epic movies and epic sound fake soundscapes and sound effect. But essentially what we are, Doing with some particles three that we hope to release until the end of the year, more or less, um, that is to actually turn some particles into a digital audio workstation, which means that imagine doing having a digital audio workstation with all the things that you do in the other station, but actually doing all of that in 3D.
So imagine. Having MIDI tracks and virtual instruments and, uh, plugins and audio editing, all of those things. But actually, instead of doing something like in a digital station, when you simply mix sounds on top of sounds in buses, you say, okay, I have this bus, so let's put everything on top of each other's and then use spanners.
The idea is to, okay, let's put things in the 3D space, and then instead of using because even if you're working only with stereo, um, when you are working with digital audio extension, your stereo features is pretty much a banner that you control how much do you want to, to each place. Nonetheless, if you are actually recording something, your choices are much wider because you can use, okay, if you use a middle side or like a tree or, or TF or any other stereo recording technique you get.
Completely different kinds of stereo. So the idea would be able to come with some of those features into sound particle to say, okay, let's create something. This is my music. I want to place the guitar one meter, uh, slighter to the back. I want to bring that the piano, uh, one meter closer and slightly to that side and then create something.
And since the virtual, since the, the sound particles create this I have this idea of a virtual microphone that pretty much it's almost like a virtual camera. It makes the system agnostic, which means that I can create something and say, okay, if I want to export this and ambisonics said, okay, it's an ambisonics microphone and I have a ambisonics rendered.
But if someone asked me, but now I want a 7. 1. 2. No problem. We change the settings of the microphone and you get the exact same music now being played with the 7. 1. 2. So, that's one of the things that we want to be able to, to explore and allow people to use sound particles and this potential is with granular synthesis and 3D environment.
Uh, but allowing people from music that until now pretty much only people from sound effects were actually using it for sound design. But now having people from music say, okay, I What would be the sound of 100 flying violins or what would be the sound of 100 drummers spread over a large area and capture everything in 7.
1. 2. So start creating, uh, and thinking about new ways of exploring audio or eventually having something say, okay, I want playing some music, but it's not. I want each note to be a particle and floating instead of bending the entire instrument around. No, it's not to be completely independent and be able to float notes around.
Uh, so, or eventually using space controller directly and be able to bend things around. So the idea, it's one of the things that we're doing, try to bring some particles, but almost like a 3D digital audio workstation. At the same time, we'll continue. We are, uh, uh, the idea it's to create artistic tools for people to explore special audio in a more interesting way.
Um, until the end of the year, we're going to release one more plugin that is going to be, uh, uh, huge from our point of view. People from music will really, really love it. Um, and also besides that, we have released Explorer that pretty much is a, uh, a sound collection software, um, that we wanted once again to bring full immersive support because there is already many, uh, softwares out there to, to manage your sound collections.
Uh, and they are very good, but they have a significant handicap is that they are, uh. Mainly target for mono and stereo. And you want to create everything that where special audio is completely a first citizen on in that world and have all the features. For instance, if I have ambisonics recording, and if I am, if I have, if I'm on my 5.
1 studio, if I press play, I want the ambisonics to be automatically decoded and things like that. And in the future, we want to also bring some cloud features into it. And finally. The binaural and creating our own technology for binaural to see if we can improve, uh, the technology in this area. So that's pretty much, uh, all the, the, the things that we have been involved.
Of course, for us, what we have done so far, it's only the top of the iceberg. We have so many things that we want to do. For instance, some particles alone, we have probably a list of literally, uh, 250 features that we want to bring into the software. And sometimes I tell people that the worst part of my work is that I have to say no to wonderful ideas that people, our users or our collaborators bring because, okay, we are limited number of developers.
We cannot do everything at the same time. But yeah, I Essentially, it's been a, uh, quite a ride, uh, being able to, to work in this area, create this product, and seeing those products being used by these amazing sound artists, and it's quite rewarding. Well, thanks for that, Nuno. It sounds like you've got a very busy year ahead of you.
Very busy indeed.
Thank-you for listening, and be sure to check out the show notes page for this episode, where you'll find further information, along with web links and details of all the other episodes. Oh, and just before you go, let me point you to the soundonsound.com/podcasts website page, where you can explore what's playing on our other channels.
