Nick Rothwell
Hello and welcome to the Sound On Sound People & Music Industry podcast channel with me Nick Rothwell. In this episode I talk to Andrew McPherson, head of the Augmented Instruments Laboratory, the developers of the Bela open source audio maker platform and inventor of the Magnetic Resonator Piano, which is a piano augmented to generate infinite sustain, harmonics, pitch bend and new timbres totally acoustically.
So Andrew, thank-you for joining us here on the Sound On Sound podcast, where we're going to talk about, I guess, a bit of audio processing and some stuff about instrument design and whatever comes up in the next 20 minutes, half hour or so. So why don't we start by you just giving us a kind of a potted CV, you know, your education, your background and how you got to where you are now.
Andrew McPherson
Thanks very much for having me here. I am a professor in the Dyson School of Design Engineering at Imperial College London, where I lead a group called the Augmented Instruments Laboratory. By training, I'm a composer, an electronic engineer and a researcher in human computer interaction and I guess over the last 10, 15 years, I've been combining these interests in the sort of service of designing new digital musical instruments, things that have elements connected to long standing musical traditions, but also to new forms of musical practice. So I originally came from MIT, where I did a double major in music and electrical engineering. I then did a PhD in music composition at the University of Pennsylvania, a postdoc at Drexel University and for 10 years or so I was an academic at Queen Mary University of London in their Center for Digital Music and I moved to Imperial last year, at the beginning of 2023.
Nick Rothwell
Okay, so I've got to admit that prior to let's say a week ago, I didn't know much about your work with the exception of the Bela board, which I've come across a few times over the last few years. So I don't know whether that's a good place to start, or whether there's some kind of a work you've done before that. Like, I see TouchKeys goes back about 10 years. Would you say that's kind of the start of this particular body of work, or does it go back before that?
Andrew McPherson
Well, if you want to go back to the beginning, the first instrument is one that's still current in my work, which is the Magnetic Resonator Piano. This is an augmented acoustic instrument, so it is literally an acoustic grand piano, inside of which have been placed electromagnets that make the strings vibrate even when they're not struck by the hammers. So the idea with the MRP is that you can have all of the traditional piano techniques that you would normally have, but that you can also make notes sustain indefinitely by bowing them with the magnets and it also lets you shape the sound of each note. For example, you can play crescendos, you can play pitch bends, harmonics, new timbres. There's a whole additional vocabulary that comes out of the instrument, but because it is still an acoustic piano, the sound production is still acoustic. You don't need speakers, it’s just coming from the strings and the soundboard. So that was the first project for me. I built that at the end of my PhD in 2009 and here we are 15 years later, it’s still going. There's dozens of pieces that have been written for this instrument, hundreds of performances. I've started to make more copies of this instrument as people have commissioned them, so that there can be other instruments out there, so that when people write pieces or they want to perform with them, that it's possible for that kind of repertoire and community to emerge. So yeah, that was the first major project that I did in musical instrument design and it's kind of gone on from there.
Nick Rothwell
Yeah, so I was looking at that and there's some nice videos and recordings online and it sounds absolutely gorgeous and I guess what surprised me about it, maybe it shouldn't have been surprising, but the fact that it is still an acoustic instrument, so it's not like some of these guitarists infinite sustain effects which, you know, record the audio and then loop them, or, you know, do some kind of electronic processing, it is actually using the strings of the piano in the same way that the strings are used when you play an ordinary piano. So in that sense, it's not an electronic instrument at all. It is an acoustic augmentation of a piano via electronic means.
Andrew McPherson
Well, exactly. Yeah, exactly and I think there's a certain kind of authenticity to the sound actually being produced acoustically, you know that it is literally these strings in front of you that are vibrating to produce that sound, you could reach your hand in, you could stop the strings with your hand if you wanted to and it gives it a kind of tactile quality, a kind of three dimensional quality that I think you don't get with a lot of digital processing or any electronic processing and there's also a way that this kind of physical system, this resonant box that, you know, has been, whose design has evolved over, you know, so many centuries to become what it is, it surprises you in different ways. It kind of throws things back. You try to put signals into the strings with the electromagnets but what actually comes back isn't quite what you put in and that's the beauty of it. If it was just a, you know, a fancy speaker, then you know, you could do things a lot more cheaply and a lot more simply but it's this kind of co-operation and sometimes a sort of friction between the acoustic elements of the instrument and the new digital ones that really gives it the character that it has.
Nick Rothwell
I guess there's also some creative constraint here. If you're doing DSP here, you could go nuts and do all sorts of things but the fact that you're using the strings as sound generators means you can't do that, so you are still working with the kind of the essence of a piano in a way, and it doesn't kind of strip, you know, stray too far from that.
Andrew McPherson
It's absolutely true and I think that Constraints are incredibly important in the design and the use of any instrument. I mean, you know, the idea that we would have one generic instrument that could do anything. I mean, you know, aside from not really being possible, I think if you could do that, it'd actually be really boring and so yeah, the fact that the piano really forces you to do certain things and not other things, I think, you know, is really important and, you know, what I found is early on, you know, when I was exploring this, I thought, well, what happens if you put this kind of sound into the strings? What if you do that? Can you play a recording into the strings? What does that sound like? And what you quickly find is that there's a whole space of things that just, you know, they're just not that effective or they're just not that interesting, but actually the things that really work oftentimes are really kind of crude and really basic sounds. You put them into this electromagnet and then that goes through this acoustic system and what comes out is somehow, you know, has become sort of magic in some, in some strange way. And you really struggle to explain why that is, but you have to work with those constraints, and the constraints are really important for the design and the constraints are also really important for composing and performing with it.
Nick Rothwell
Okay, I'm kind of thinking, I did now whether anybody's done a prepared piano as part of the resonator piano. Would that make sense, or was that just a bit too crazy?
Andrew McPherson
I'm sure it could be done. I mean, you know, what I think of is, you know, when you think of the Cage prepared piano, actually what it is, is it's the percussion of the piano with the resonance changed. That's why, you know, you put rubber mutes or, you know, steel bolts or something in the strings and you still have all of the percussive action of the instrument, but with a different kind of sound, a different timbre. And this is sort of the conceptual opposite, in a way, it is the resonance, the natural resonance of the strings, but detached from the sound of the felt striking the strings. So, you know, in principle I guess you could do both, you could put other objects into the strings as well, change the resonance and then somehow excite those new resonances. It'd be a little bit tricky because I'm not, you're not actually setting up feedback loops in this instrument, so it's not really listening to where the strings are tuned. You have to kind of know where the strings are tuned and then you configure it so that it, you know, plays in an effective way. So if you prepare the piano, then there would be a whole extra kind of step of thinking okay, what does this sound like now and how can we get those new modes to really speak? It'd be interesting to try.
Nick Rothwell
But you are picking up harmonics, I noticed. So, how much work did it take to actually get this thing to work? It sounds like it's kind of very highly tuned to exactly where the harmonics of the strings are. I mean, is it down to kind of very minute, detailed placement of the electromagnets, or what?
Andrew McPherson
You know, I think it sort of depends on what you're looking for out of it. It's a lot more robust than I might have expected when I first set out. I had a whole bunch of things that I tried to do in the early days, you know, try to, you know, come up with some sort of phase lock loop that would, you know, precisely tune the magnet into where the string wanted to ring. And, you know, as I went along, I found out you don't really need a whole lot of that, to be honest. You just put in signals that are kind of more or less where the strings are tuned, or one of the harmonics and it just kind of sings and it just works. Now you can detune it a bit, in fact and that turns out to be really interesting for reasons I still can't entirely explain. The strings sort of fight back in a way, so it does bend the pitch, but also kind of warbles a bit and that, it gives it a certain sort of character. But on the other hand, if you're trying to do something that is like a, you know, a kind of digitally precise sort of acoustic synthesiser of some sort where you can guarantee that, you know, the 7th harmonic of this string is going to sound exactly as loud as the 9th harmonic of that other one, that's really variable, it depends on the piano, it depends on where the magnets are. So, you know, it's like going from, you know, from one instrument to another of any acoustic instrument. They all have their own characters, they're slightly different. You get to know the kind of things that are consistent and the things that are not and, you know, you just work with that as a musician.
Nick Rothwell
So I've got to mention, just for the sake of completeness, TouchKeys, given that Sound On Sound covered those 10 years ago now. So that's similar in the sense that you're sensing keys being played and sensing continuous gesture there. But that is what, it's a USB mini device that just generates, what, after touch and pressure and just data signals then feed into whatever synth, whatever sound source you're using, presumably?
Andrew McPherson
Yeah, so yeah, TouchKeys was where I went next after the MRP and there I was really concerned with how this notions of a sort of detailed and nuanced control could be applied to a MIDI keyboard. So the idea is that go on the surface of an existing keyboard. So you keep the familiar action, you keep all that familiar technique, but then you also have these extra degrees of freedom by moving the fingers around on the key surface. It's a USB device, it plugs into your computer, there's some software that runs on the computer. It aggregates the signals from the touch sensor and from the keyboard the sensors are installed on and then you get various mappings that you can choose that you can add things like vibrato, pitch bends, you know, swells to the synths that you have to control. So the acoustic element isn't in that but it's, you know, it's intended in a different musical context. And I guess for me, that's one of the things that is important in the work that I do in the lab is just that every technology is matched to a certain kind of musical context that makes sense. And that, you know, you really think about what are musicians trying to do in this area and how can we build something that, you know, kind of speaks a language that they already understand but also brings something new to the table because, you know, there's plenty of gear that you can buy already. We don't need more gear for gear's sake. Really, everything should, you know, have a reason for existence.
Nick Rothwell
And I guess the TouchKeys is kind of working in the same space as Roli with their Keywaves, their phone based instruments.
Andrew McPherson
Yeah we, actually the Seaboard and Touchkeys were released, I think within about a year of one another, actually and you know and those aren't the only ones, either. It's actually really interesting to see all the different controllers and new instruments that are in that space and, you know, I'm always interested to see the new things that come out, because I'm just a big believer in sort of nuance and detail, I think the things that set touch keys apart, you know, sort of in terms of the ideas that I had to begin with, the reason I did it the way I did was to try to stay as close as possible to a familiar keyboard technique, even if that meant sacrificing certain kinds of techniques that might be possible on other physical interfaces. And I guess that's just, you know, that's a, again, a recurring thread in my work, is the idea that instruments should connect to some kind of existing musical skills or to some sort of communities or aesthetics, but then, you know, also extend that in some way.
Nick Rothwell
And I guess also, to put it fairly bluntly, TouchKeys and Resonator Piano are kind of aimed at different parts of the market. You know, one part is those who have actual real pianos and the other part is those who have MIDI keyboards that might want to augment their digital, or their, yeah, their synthesized music composition skills and performance.
Andrew McPherson
Yeah, totally. Mm hmm. Yeah, I mean, absolutely. It's, you know, it's different communities, it's, you know, it's different markets, really, and it's different musical aesthetics as well.
Nick Rothwell
Okay, so coming back to the Resonator Piano for a moment I, one thing that I find interesting about all these kinds of instruments is it's one thing to build the instrument but what's also important is to kind of build the ecosystem and the community around it and it seems to me that with the resonator piano, you also are trying to build this in terms of finding artists who want to experiment with this and I guess helping them get material recorded, performed and building up a kind of a repertoire that kind of establishes the resonator piano as a kind of a real instrument with some kind of pedigree to it. Is that a fair comment?
Andrew McPherson
I think that's absolutely fair and it goes back to the very early days of the instrument. So when I first built this, I was doing a PhD in music composition and so in a lot of ways I was building for myself, not to play because I'm not a great pianist, but to compose for, but I think if I'd stopped there, it probably wouldn't have been that interesting. The thing that really changed everything was then starting to work with other composers, first a group of six composers in Philadelphia and kind of from there, you know, now a few dozen more and you know, artists from a number of different musical traditions. Everybody kind of finds something new in the instrument and everybody has their own take on it, which is super interesting. But then also you see that people learn from one another, so somebody who comes in now to use the instrument, they're not just influenced by whatever demo I might give or whatever videos I might have put online, but also the, you know, the other pieces that have been written, the other performances that have been done and I think that's really important that's, you know, that these communities can form and can grow in that way. But it certainly takes a lot of facilitation work to do that kind of thing because for a lot of years it was really just me carting around these really heavy cases and you know, installing them in different pianos, you know, with varying reactions from the local piano technicians sometimes, and there was a risk I think, as the project went on that my own, you know, my own availability would ultimately become the sort of ball and chain that prevented from kind of sustaining a community. Because if I personally have to be there every time any performance is done anywhere in the world, obviously that doesn't scale. And so that's why, after, you know, a lot of years of having just one or two of these instruments around, just in the last couple years I've started to look at building more and making it possible to actually produce these at, you know, at some kind of modest scale. And it's really in a way about kind of, you know, letting it sort of fly free a bit and you know, for people to be able to take ownership and do their own things with it.
Nick Rothwell
So do you have any kind of sense of what kind of body of work may be built by this? I mean, I'm hoping it's not just, you know, lots and lots of ambient sustained piano albums, but will kind of spread into, or kind of flower into a more varied set of musical genres.
Andrew McPherson
Oh, it absolutely is. And in fact, it already is, you know, it kind of naturally lends toward a certain kind of, a droney sound, it’s very compelling. But in some ways, some of the most interesting work has been stuff that either has combined this with traditional pianism, so that you can get quite, you know, a lot of activity out of the, you know, out of the traditional hammer mechanism, or which has kind of looked for the kind of, you know, almost the flaws in the instrument and kind of gotten in there and pulled that apart to discover some, you know, really quite unusual and sometimes harder-edged sounds that I, as a designer, never intended in the first place, but which now are essentially inseparable from the identity of the instrument. And I mean, there's loads of examples of that kind of thing, you know, throughout music history. You know I've heard said, although I haven't verified for myself, that the key click on the Hammond organ wasn't a sort of design feature, it was a, you know, it was a thing that existed, and that, you know, maybe Lee Hammond didn't like that so much but now that's, you know, that's a part of the character of the instrument. So that's what we take it to mean, so yeah, there's a lot of different things that have happened. I mean it's not all ambient music. I mean there's, you know, there's been more, you know, jazz and rock influenced music, there's been some film scores, there's been some operas. I mean, you know, it's quite varied.
Nick Rothwell
Okay so let's look at Bela, because as I say that's something I was most familiar with, you know, over the past few years. It's kind of living in a space with a few other similar devices, like Monome Norns, which is a Raspberry Pi with a Supercollider engine and Lua on it, Cycling 74 now have max generating code for the Pi3. So, you know, they're working in a similar way. I guess the way I see Bela here is it's more kind of, in a sense it's hardcore because it's low latency, designed for real high performance and I guess that's because there's a lot of serious acoustic musicians who are kind of working with it demand that. Is that a fair comment, do you think?
Andrew McPherson
Yeah, I mean, I think the story with Bella is twofold. One is that, you know, there should be no compromise in the performance of the system. That means extremely low latency, you know, down to one millisecond round trip, which you're not going to get even on a, you know, a high spec laptop. But it's not just that, it also should be something that is immediately usable by people with widely varying technical skill levels. So, it's not the kind of thing that you're going to have to be some kind of, you know, Linux gearhead to be able to understand how to write code for it. You can just, you know, you can run a PD patch on it, you can actually export, you know, Max, you know, Rainbow or Gen. You can export to Bela as well and that, that code will compile. You can use Supercollider, you can use, you know, lots of different languages and then we have lots of examples. There are tutorials, I mean, I've done like a 21 lecture YouTube series on C audio programming and so the idea is that, you know, you shouldn't have to choose between tools that are immediately usable and tools that are performant and I guess that's been the kind of mission from day one there.
Nick Rothwell
Okay, so what's the kind of lineage of that? At what point did you decide, you know, if I go and build a Low latency high performance box like this that it's gonna get picked up and who's gonna pick it up?
Andrew McPherson
You know what, it kind of just grew organically out of originally the needs that we had in the Augmented Instruments Lab, you know. I think if we had just sort of started out by saying, you know, what is a hypothetical specification for something we'd bring to market, it would have been really hard to pull off but what had happened is that I was working on a research project around 2013, 14. It was around constraints in musical instrument design and how the constraints encourage people to sort of appropriate and modify and push back at their instruments. And, you know, for varying reasons that I won't get into, we needed an embedded platform with a lot of those specs, with very low latency, with a kind of high bandwidth of input and output for analogue and digital signals and so, you know, I made the core code that later became Bela. And then what we found after that was that actually a lot of people in the lab needed something very similar for varying reasons and so other PhD students in the lab started picking it up for their work. A number of them became active contributors to the project, you know, adding lots of features, including the browser based IDE that is now kind of one of the main features of the platform. But this was still all internal at the time. And then we started kind of working with a slightly wider network of external collaborators still through mainly academic networks or independent artists. And then it was probably about two, three years after I started that we thought you know what, this is ready more for primetime and we ran a Kickstarter campaign in 2016 and, you know, that kind of took off from there. So yeah, I guess it was one of those things where having identified a set of features or a certain set of challenges to be overcome, we discovered then that actually lots and lots of people had very similar things that they were looking for and I think that's what has helped establish Bela out there in the community.
Nick Rothwell
Okay, so maybe the kind of the secret sauce, as it were, was that you started out in an environment of smart academic practitioners who could kind of take this thing and really kind of work with it and that kind of sustained it to a kind of critical mass.
Andrew McPherson
I mean, I guess one of the things that is nice about academics is that you can direct your activities toward things that I think are, you know, kind of difficult, tough nuts to crack sometimes and that you can you know, you can spend some time really working through those without worrying about whether it's gonna produce a marketable product, you know next month, but yeah, I didn't set out to do this as a commercial product really. But then you know, ideally anything in academics, as well as anywhere else is that, you know, it should have some sort of real impact. It should make a difference to how, you know, how people design things, how people think about things, how people make music and yeah, I guess in a way it was, we just happened to be in the right place at the right time in terms of needing certain things that turned out to be fairly common across the community.
Nick Rothwell
Okay, so to clarify you've got an Augmented Instruments Laboratory and also an Augmented Instruments Limited. So if you've got like a research arm and a commercial arm, is that the way it's working?
Andrew McPherson
I suppose you could think about it like that. So Augmented Instruments Limited the company, it's a UK company and that was set up to commercialise Bela. So you know, Bela is the kind of the original and kind of main product line of Augmented Instruments Limited, although now we've also gotten more into touch controllers and into modular synthesis. So for example, one of the things that the company is doing is a Eurorack module called Gliss, which is like a capacitive touch controller for your modular synth, so basically you can kind of draw waveforms on it or, or control CVs directly from your fingertip. So I mean, you know, that's all under the umbrella of the company, the Augmented Instruments Lab. Basically a team of people located at the moment across two universities across Imperial where I am now and Queen Mary in their Centre for Digital Music, where a number of the PhD students in the lab, you know are still completing their degrees and so yeah, I mean, it's not the kind of thing where all the research in the academic side automatically ends up over at the company I mean that you know, the name I guess was shared because we envisioned that the company might do more things than just Bela, as indeed it has, but there is, you know there's, we're certainly on friendly terms.
Nick Rothwell
And I'm going to do a quick name drop here of Laurel Pardue because she's a violinist and I actually shared a stage with her many years ago and she's using Bela with her constructed violin and she's now at Ableton, I believe. Or was last time I checked.
Andrew McPherson
Yeah, she still is and yeah, I saw her just recently and yeah, so, I mean, her work is amazing and she's an incredibly good musician as well. She did a PhD and postdoc in the lab and there, and then also at Aalborg University in Copenhagen, she developed this instrument, which is called the Svampolin. It's a kind of hybrid acoustic electronic acoustic violin which essentially decouples the string vibrations from the body of the instrument, but in a way that you can sense them then with pickups and then sort of inject sound into the body of the violin. And what that means is that between the sensors and the actuator, you can stick some kind of digital sound processing and you know, what kind of thing do you need for that? Well, you need multi channel, you need low latency and you know, naturally Bela is, you know, what we chose to work with for that, yeah, but it's a fun project and you know, one that you know, I think we'll see continue to evolve.
Nick Rothwell
I guess that's a good example of why you need something like Bela, to have lots of sensory inputs and outputs and very low latency, because if you didn't have that performance then that kind of work just would not work, be convincing or not work at all.
Andrew McPherson
Yeah you know, I think one of the things that is really been impressed upon me is that there's a big gap between kind of being able to describe something in words, like I want to measure this and then I want this to happen and like really being able to do that to a level of detail and nuance that really makes it just kind of intuitively work as a performer, as a musician. And, you know, there's loads of stuff out there, there's loads of tools out there that you can, you know, you can read some sensors, you can send a message through a USB cable to the computer and then it can synthesise some sound in response. And that's fine, you know, that works. That, you know, describes a valid musical system. But there's a big difference between that as a kind of, you know, low sample rate, kind of sluggish, jittery sort of interaction and something that is just like absolutely there and immediate to the point that you'd almost believe that there was no digital processing at all. And, you know, it's not that Bela is the only way to do that of course, you can build a lot of platforms if you engineer them that way to give you that kind of performance. But I think it's just about, for me, it's about thinking about those kinds of value sets early on and figuring out what do you actually need your technical tools to do for you.
Nick Rothwell
Yeah I guess, I mean, if I was to kind of do my, you know, old studio engineer grumble here. If you look at things like latency on DAWs, it's actually quite high and somehow, you know, we're looking at things that say, you know, latency, 20 milliseconds and you think, nah, you can feel that. What'll be lost here?
Andrew McPherson
Absolutely, you can feel these things and you know what's interesting, we studied this too. We did a study with a percussion, you know, kind of a simple digital percussion instrument. This was Robert Jack who led this work and he's now working for Bela. But we were looking at do people notice latency and how much latency does it take for people to notice? And you know, it had been proposed by David Wessel like, you know, decades ago, that the answer was something around 10 milliseconds and our study more or less confirmed that. But what was really interesting was there was a sense where people's perception of the quality of the instrument degraded with latency, even before anybody really kind of noticed that latency was what was happening. And so it's like at some level there's something just kind of intuitive going on here, you know, is this thing like kind of right there under my fingertips or do I feel kind of, you know, detached from it a little bit?
Nick Rothwell
So let's wrap up. Let me first throw a slightly speculative question at you. If we were five years on from now having this kind of conversation, or talking about these things, where do you think we'd be? So if you kind of think forward, what's our next stages and what do you think is on the horizon?
Andrew McPherson
Well I guess, I mean, you know, future prediction is always hard. The obvious thing that, you know, that everybody I guess would say and I would say too, you know, well AI is obviously, it's not that it's on the horizon, I mean it's that it's here now and generative models that are producing sound and music are a thing that is unsettling a lot of established practices. I probably don't have a lot of, you know, greatly original things to say about that except to say that basically this idea that new tools come along and destabilise old working patterns is something that's basically as old as tools. And so, you know, surely creative musicians will, you know, find some ways to work with these things at the same time as everybody's going to grapple with all the economic forces that they've always grappled with, which tends to commodify artistic work. And you know, it's this perpetual push and pull. What I'm hoping for I suppose, closer to the work that I'm doing, is that the digital and the kind of physical worlds don't need to remain as sort of separate universes and in some cases, even as separate aesthetics. I think that one thing that the advancement in computing power can do for us is not just more computations per second, but it's actually better IO. It's that you have, you know, a richer link, better sensors and actuators, more channels, more resolution, what have you and if you use that sensibly, then I think there's ways that we could design instruments where they have physical components, they have digital components, but they don't need to be kind of walled apart from one another. And to somebody just making music with them, then you know, I think it should be an enriching experience where you don't kind of have to choose which of those sides of the divide you want to sit on. So I think five years is actually a reasonable time horizon to see some things start to change there. But you know 10, 15, 20 years, who knows?
Nick Rothwell
Indeed. Well, thanks so much for talking to us and maybe in five years time we'll come back and do it again and see how close we were in our predictions. Yeah super, really interesting. Thank-you very much.
Andrew McPherson
Absolutely. Thank you so much for your time.
Nick Rothwell
Thank you for listening and be sure to check out the show notes page for this episode, where you'll find further information, along with web links and details of all the other episodes. Oh and just before you go, let me point you to the soundonsound.com/podcasts website page, where you can explore what's playing on other channels. This has been a Project Cassiel Production by me, Nick Rothwell, for Sound On Sound. Bye bye.
