התמלול המלא של Hyperledger: How The Sausage Gets Made. Hart Montgomery, CTO of Hyperledger Foundation - E106

[00:06:00] something, in the Hyperledger code base. Yes. It's not uncommon at all.  Eric: Yeah. Welcome to decentralization, right? I think it's a great idea to start off with the big categories, the buckets if you would of what Hyperledger does, what pro what, what technologies fall into each before we jump into some of the hot new things.  Hart: Absolutely. How I like to describe Hyperledger is the Linux Foundation's umbrella project for blockchain and blockchain related work. The Linux Foundation fundamentally solves the problem of. You have a code base that lots of different companies or individuals wanna work on together, but there's no single point of trust, right? There's no one company that can be trusted to hold the code base, right? And, if you think about this sort of makes a lot of sense. Why you would need a [00:07:00] third party, neutral organization to handle this. If I b m contributed their blockchain code in a way that they just open sourced it, right? They just pushed it out on GitHub, sort . If you're a competitor of IBM, would you really want to build on that code? Would you wanna work on that code? If you know someone they could take it away at a Heartbeat, right? Or they could, push the project in a direction that deliberately broke your applications. And, now that they would but that's a possibility that you'd have to worry about, right? If, and if you want people to collaborate and work together, then you need to push it to a sort of neutral, third party organization. And this was essentially the rationale for Hyperledger Fabric and why it was the first project in Hyperledger and even more why Hyperledger got started. But this sort of, decentralized [00:08:00] development I is the reason that the Linux Foundation exists, and it's a really nice tie into blockchain, right? Because the whole purpose of blockchain is decentralization. And I point out that, if you don't have decentralized development, is that really what you want in blockchain, given the whole point is decentralization. But back to your question, the Hyperledger Foundation is about, it's the open umbrella for blockchain in the Linux Foundation. And we have a number of different projects. So, I lump things into four categories. And not all projects fit exactly into these categories but it's an approximate thing. We have distributed ledgers, right? So, we have fabric, which probably everyone knows. We have bass. Which, is an Ethereum execution client, and there's some permissioned modules that you can use to run besu in a permission setting. We [00:09:00] have obviously aroha and Sawtooth and then we have we have all identity stacks. Distributed I or decentralized identity, self-sovereign identity is a huge application. I like to think of it as the backbone for web three. Because everything if you want to do things in a decentralized way, you need to have a consistent identity. You can Use across different systems but you wanna be able to use it in a privacy preserving way. So we have a stack of now Indy, which is actually a ledger, but it's a ledger focused on an identity. So I'll put it in the identity stack. We have Aries SSA and a non-res each, which sort of address a different part of the identity stack, and I can go into that in more detail later. We have a host of new [00:10:00] projects that I would say focus on blockchain interoperability and integration. Everyone sees, we're going to be living in a world of many networks, right? There's not gonna be any one blockchain that. Is the best for every single application out there. And as we replace traditional systems with blockchains. These traditional systems, these old databases have to talk to each other, right? So obviously these blockchains are gonna have to talk to each other. We're, we've already seen a lot of interest in you. Cross chain transactions, moving assets across chains even. And we just think that's gonna get bigger and bigger. And we have a number of projects and people working on that aspect of interoperability and integration. And we also have people working on the style of project that is, [00:11:00] write code once and use it anywhere, right? It's, it'd be frustrating if you wrote blockchain code once and then you had to duplicate it for every single ledger that you wanted to work. So, the idea is that, projects like Firefly it's like a middleware container. For blockchain. So  Eric: we'll get into Firefly because that's one of the hot new things. So, one thing I find interesting is that, because Fabric was one of the it I don't know if I'd say it's the most popular, but it's certainly one of the most well-known Hyperledger projects. And it's certainly is one of the most utilized. But there's an impression that where you associate Hyperledger with sort of a one transaction orderer that you would get under fabric and some of the later projects, maybe people aren't connecting the dots quite as much because as you move into sawtooth and Besu, they do support both [00:12:00] permissioned and un permissioned, the, and each node potentially being an orderer. So it, it is a fairly comprehensive stack as it relates to permissioning and ordering of transactions and different mechanisms for achieving it. . Like Besu is an open source Ethereum client. So that's pretty that's pretty exciting when you think about it. And you think about all the different developments. We're gonna talk about Firefly next, just how Hyperledger is building out all these different projects to be more compatible with what a lot of people in the digital asset space look to for, a tokenized market, but You don't have the distraction of it. Before we touch on that let's talk a little bit about when you say we have people working on these things, why don't you to develop that a little bit and how projects are added to Hyperledger. Hart: Yeah, that's a great question or comment. And when [00:13:00] I say we have people working on, I mean we have open source

contributors Donating their time and working on, open source code and donating might be a little bit too it might be too generous. A lot of these people are paid, full-time by companies to work on open source projects. And if you think about it, from a company perspective if you are building a business that's based on an open source Project, it certainly makes sense to, to have some people working on that project and developing it and making sure it, it stays, in good health. And we have some very interesting contributors for, from some very interesting places. Perhaps one of my favorite facts is we have a number of employees from the government of British Columbia who are full-time [00:14:00] developers on our identity stack. And I think that I've always thought that's really cool. And those folks are really fantastic and great to work with. But to address your question we have a whole process on how code and projects come into Hyperledger. It's relatively similar across different parts of the Linux Foundation. There's obviously some variation between projects but we have a number of different designations for projects. We have labs which is the lowest bar. When we have projects, which we call, ooh, we have two tiers of projects. We have projects in incubation and graduated projects, and those sort of reflect the maturity of the community around the project. To get a project into labs. You just need to get a couple of people that you know, we have as what we call lab stewards to sign off on your project. To decide it's relevant to blockchain. But [00:15:00] labs are about experimentation. We don't really hold people to a, a super high standard for that. It's a place to showcase your code, really, it's Hey, we did this. And that's what we want it to be once we get to projects it's more about, do we see this, being a viable open source project in the long term. And a project in incubation has to be approved by the Technical oversight committee. Where they, there's a document and a process and it's basically just checking off the boxes. Do you have what it takes in place to eventually succeed as an open source project? And obviously not every project succeeds. We wouldn't be doing a good job if every project succeeded. That would imply we're much too restrictive about what we're, what we're encouraging people to try.[00:16:00]  It's generally a process of just making sure that people. Are in the right position to succeed. And then a graduated project is another approval step from our technical oversight committee. And that sort of indicates that, the project is in a good spot. It's following best security practices, it's doing good release management, all that stuff where, we think it's basically a, okay, you can trust product., you can use this in production, sort of an indicator. So that's how our it's what we call the project life cycle. And generally, we do see a lot of projects that go from labs to projects and incubation to graduated projects, and we've also seen some projects wind down. When o other stuff comes around or people wanna move on to different things and [00:17:00] decide, the project didn't achieve its goals. It's all about, maintaining healthy code and. Signaling to people, like being totally transparent, this is what state the code is in, this is what state the project is in. Eric: So that's really interesting because I was prepping for this call and I wanted to, I was digging through the Hyperledger site and I encourage, obviously anybody who's interested to do cause there's a lot of information on it. But I was seeing this distinction and at first I was just oh, why is Firefly. , if they got the big announcement and this other one doesn't, then I was like, hold it. It's, they're, they use different words, , to describe where they are in their evolution. It does tomorrow's firefly is probably already in a lab. And there's yeah, there's tons of, there's tons of projects in the lab and they're working their way up. So when they are released as a [00:18:00] full-blown project, that's a great signal.  Hart: Yeah. And labs are also great for collaboration. Like a lot of times we see people working on labs that are very similar. And so we'll say, Hey, have you all thought about, working with this other team? Because you all are doing, very similar things. And we have a number of different collaboration efforts were. Projects have merged or moved together over time, like cdi which is the interoperability project. Has a funny genesis. And I was involved in that back to my time at Fujitsu. So basically, we were at a Hyperledger conference, I believe in Tokyo. I believe it was the Tokyo Member Summit. And I was talking to some of the folks from Accenture, like Mike Klein and Tracy Kurt, who are fantastic. And we basically [00:19:00] realized we were planning on building almost the same thing for interoperability and open sourcing. And then, we had an a shucks moment maybe we should just work on this together in open source. And so we, published our codas labs, merged it together became cactus. And later on, lo and behold, i b M is working on something similar. They called Weaver and so now they're, merging that code base into to Cacti. It's a really, the labs are a great setup for collaboration as well. Eric: Excellent. And you get big teams all at once, right?  Hart: There's so many and duplicating effort, right? If you're, if two people want to build the same thing and they're okay working together, they should work together. And that's a big part of our job as Hyperledger staff is to is to make sure that, people know about other projects that [00:20:00] are doing similar things. And, giving them the option to work together if they so desire. We don't, make people work together. And there are some cases where people would prefer not to work together. And that's okay. But we don't want people not to work together because they don't know about something. Eric: It's like

the job of good managers to like what? Remove barriers and facilitate synergy. Could you break down, you talked about a number of identity solutions. Could you , could you break them down a little bit into their particular use cases?  Hart: Sure. So do you want me to go into the stack or the i the identity stack or identity use cases? Eric: we could try both .  Hart: Okay, sure. So the primary Hyperledger [00:21:00] identity stack right now is four projects. It's Indy. Aries, ersa and ACRs. So, each of these projects serve different roles. So, India is a ledger that is specifically focused on identity applications. So it does not have a lot of the smart contract functionality that you might want in a general purpose ledger. It is, optimized basically exclusively for a handful of identity applications. . So, Ursa is the cryptography layer. So Ursa is where people put the, cryptographic protocols that are used in this stack. And there are a number of other projects. They use Ursa as well, like Aroha uses Ursa. But at least in the identity stack it, it functions as the cryptography layer. Aries is technically an agent but if you're not familiar with that term, you can think of it as a wallet. So, it's the [00:22:00] wallet. And then a non-res is the layer of anonymous credentials, protocols which obviously use the cryptography layer. Anonymous credential protocols are much more than just cryptography. So there's a lot of code in that project as well. And the ideas. It's as modular as possible. So anonymous the non-res are anonymous credentials is actually a very new project. And it was created so that the identity stack could be run on, multiple different platforms. Whereas previously, most people ran this identity stack on Indy. Now the idea is that you could run a non-res and Aries on, whatever blockchain or system you desire.  Eric: Excellent. Alright, so I guess maybe we talked a little bit about [00:23:00] non-res, which I guess is new, right? It's one of the newer projects. Let's talk a little bit about Firefly and solang. Sure.  Hart: So, I like to think of Firefly and Solang as projects that sort of solve the, as I said before, the right code wants. Run it anywhere. Problem. And these projects go about it in different ways, right? So Firefly creates what they call the super node which I, this isn't exactly accurate, but I think it's a reasonable abstraction, right? It's the old saying that no models are accurate, but some are useful. I like to think of Firefly as like a middleware or a container for blockchain where the idea is you can write, smart contracts and everything you want for a particular blockchain application. You can write it once and you can run it anywhere using the Firefly stack, right? Any blockchain that the Firefly, stack supports [00:24:00] which is quite a lot and more than I can keep track of at this point. You. they support and Solan right is a solidity compiler for Solana and substrate, right? So the idea is that you

can, take your solidity code and compile it into, smart contract code that can be useful on these other platforms. And, that might be really useful if you are. , if you're a Solidity shop or you have Solidity developers and you wanna run on these other platforms.  Eric: Excellent. And then what was the process for Solang being accepted as the latest project? ,  Hart: They were a lab for a while. They had a lot of momentum. They had recently gotten more developers working on it. And so one of the, one of the criteria that we look at, [00:25:00] or more precisely our technical oversight committee looks at when, deciding whether to promote a lab into a project or not, is whether there's a diversity of developers and companies doing the development effort. So what we don't want and not only Hyperledger, but any open source, open development project is we don't want the project to fail if one person goes away or one company goes away, we want it to be diverse and decentralized enough where it can survive. If one person or one company goes away and Solan had gotten, some more interest, other developers were working on it, and so it, it seemed to be in a place. Where it was mature enough to move to a project in incubation status, and obviously, it goes without saying that everybody thought that the problem that they were trying to solve was [00:26:00] really interesting. So, they were clearly. In an area that interested our community.  Eric: Excellent. In the past podcast, we've covered this notion of choosing the right architecture. So I'm gonna talk about what some of our prior podcasts or podcast guests like Matt Zand or Raphael Belchior, and Daniella laid out as being some of the considerations and I'm hoping you'll shape and modify it. So, we, we are, we're collaborating, right? collaborating with prior episodes. Zan started out with Meza, started out saying, is blockchain even the right solution? That's a great first question. And then suggested using a data flow to determine whether it is and asking the question whether it's public, private, or hybrid. Hart: Yeah. So blockchain is about decentralization, right? And so the metric I [00:27:00] use and I encourage people to use is, A blockchain is a distributed database with decentralized trust where you don't have to, trust any one single entity. And do you need a blockchain boils down to, to do you need this primitive, is there one single actor who is willing and able to act as the host and be the source of truth for this? This information store and do other people trust this actor? And if there is an actor and people do trust them then you don't need a blockchain, right? You can just have this entity manage everything. But if there is no one single entity that everyone trusts and is willing and able to manage things and I say willing and able because there are a lot of use cases where there may be a single entity who people trust, but they're not, they're not able to [00:28:00] manage a whole You. information stream or database. I guess you see this a lot in like regulated financial industries where maybe the regulator people do trust and are willing to share data with, but the regulator is not capable of hosting a system containing all of the data. But once you've decided that you do need decentralization, and again, you don't necessarily need it for every application, then it comes down to how much decentralization you need. So, I, I don't think of, blockchain or distributed ledgers as being decentralized or not. There's all continuum of decentralization. It can depend on a lot of things from like governance to your consensus algorithm, right? , on one end, right? At least in theory not always in practice, but at least in theory, fully public blockchains, right? That [00:29:00] use, proof of work or a public proof of state protocol would appear to be the most decentralized, right? And on the far other end you have traditional databases, right? But there are a lot of choices you can make in between, right? You could use, Some sort of distributed ledger with a Byzantine fault tolerant consensus algorithm. And a robust governance mechanism, they would send somewhere in the middle probably of, for decentralization, right? You could weaken your decentralization guarantees if you wanted to use, say, crash fault tolerance, right? And generally though as you weaken your decentralization guarantees you also get better performance, right? So you want to go to the point of decentralization you need, and the but not really any further. And I guess this goes back to the famous blockchain tri, right? Which sort of says [00:30:00] you can have, you can have some of, security, decentralization or performance. But not all of them at the same time. And I like to think that no one should compromise on security. So you should be specifically trading off between decentralization and  Eric: performance. And speaking of the trilemma, when you look across all the different frameworks, do you think of that tradeoff between decentralization and performance across the different frameworks, Emphasizing the decentralization. Others emphasizing more the performance, maybe some emphasizing security to even the detriment of both performance and decentralization.  Hart: Not only between platforms, but also within platforms. Like Besu supports multiple consensus algorithms, right? And the [00:31:00] effect of decentralization you get will depend on which consensus algorithm you use. Same for just, at least in a permission network, how many nodes you want to run. If you run, 10 nodes across 10 different organizations, you're getting more decentralization than if you just run four nodes. And a lot of it depends on the configuration of the network, right? And a lot of external factors, , for even this is even a question for public blockchains, right? You can get into questions of, what degree of centralization do you think mining pools, force, what about things like m

e v and flash bots and all of this other stuff. It becomes a really complicated question once you start diving. ,  Eric: do you see the various encryption technologies, and I'll [00:32:00] put, Z K P and homomorphic encryption whether partially or fully, in, in that same bucket, do you see those types of technologies driving the need for More blockchain technologies, more public blockchain technologies because we had the capability to encrypt them, and it allows us to collect data from more of a more diffuse set of sources. Hart: Yeah. As someone with a cryptographic background, I'm a big fan of all of the Z kps and associated technologies being used on blockchain. I think they're extremely useful for a number of different applications. I don't necessarily think that they will solely cause people to say, move from permission to public blockchain or things like that. There are a lot of things you have to think about when [00:33:00] using Z kps and Z kps are great, but they're not, just because you use Z KPS doesn't mean you, get perfect privacy, right? There's still things like traffic analysis attacks there's still a whole number of things you need to consider. When you are, thinking about privacy. And so, one of the things that, that I that always bugs me is maybe you've seen the movie Dodge Ball. But I sometimes call, you know what people do in blockchain is the patches of Houlihan strategy for privacy or security where they just take a bunch of tools and they just throw tools at the problem until they think they've salt it. But you know what you really should be doing is you should be thinking about you. What do you want? What privacy guarantees [00:34:00] do you want? What security guarantees do you want? And then, appropriately pick the cryptographic protocol or protocols that give you those guarantees. And, and o obviously Z kps are a great tool for this. But you do have to think carefully about the guarantees you get.  Eric: So while we're talking about security,, security obviously in the public blockchain space has been much more of a concern over the last year. That the hacks have gotten bigger and it seems both bridges seem to be a particular target as well as protocol hacks. How has hyper ledgers community responded to this?  Hart: We've tried to have, rigorous security protocols in place. Putting proper procedures in, in place I think is most important. I don't view security as it's not a [00:35:00] bit you flip on or off. It, it's a process, and just making sure that you have the right people in place and you have the right processes in place. It is, it's about the best you can do. It's very tough to say you're not going to have security vulnerabilities. You're not gonna have security issues, everyone does. If you remember Hart bleed, right? Even what is ostensibly supposed to be the most secure piece of code in existence, had critical bugs. So, it's all about putting processes in place to mitigate the severity and the impact of bugs and do all that you can in, in that regard. We haven't seen as many, critical issues in hyper letters as necessarily some of the public chains, the [00:36:00] public chains are really juicy targets, it's. If you find a zero day in fabric, it's not as clear how you would exploit it as if you found a zero day in, it's one of these bridges. The bridge you can immediately take out, vast amounts of money. So  Eric: With fa fabric you would never get that didn't, that's the tradeoff.  Hart: How do, how, what would so it's a question of, are the attackers probably aren't as motivated. Just because the reward isn't as high.  Eric: But in the context of Hyperledger, would you say that there's more of a sharing of security? So for example, you have enterprises, larger enterprises, which sort of very developed security stacks doesn't necessarily mean that it's more secured necessarily. But nonetheless they have [00:37:00] developed code audit. Penetration testing, assessment, soc whatever, vulnerability scanning. But vulnerability scanning, probably less important. Penetration testing may be more important. Code review may be more important to, to what extent? Does that make its way in the process? To what extent does the foundation leverage that? So for example, if a big partner runs a penetration test, finds a critical software bug, of course they're gonna, you're gonna presume that they're gonna let you know, or they're gonna fix it and not let you know. Who knows? But what does, or how does Hyperledger Foundation try to ensure that it's collecting that kind of. On any bugs or potential vulnerabilities in, in the code base itself.  Hart: We have a whole security disclosure process. We also [00:38:00] do benefit from, having a lot of security expertise at all of the companies that, that contribute. Not well, obviously, some of the companies that contribute we have a bug bounty program. We do regular security audits. An again, it's just about having the process and procedures in place. And we can't force people obviously, to contribute bugs they find or to notify us. But, we do our best to encourage them to do and generally we try to make it a, a low effort process. The incentives are there.  Eric: You ever have a hackathon?  Hart: Yeah, we do. We've had a number of hackathons and we're ramping this back. Now that, people are willing to meet in public again post covid. So we definitely have those and we're now trying to, to work more closely [00:39:00] together with open s f on some of the security stuff. So I don't know if you're familiar with the Open ssf. It's an open-source security foundation. It's a Linux Foundation project. Brian Bailor is who used to be in charge of Hyperledger is in charge of that from the LF perspective. And the idea was it got started due to all of these, like software supply chain bugs, like log for J where lots and lots of people were using this, this not so well maintained program. And it, it caused a huge problem,

right? Yep.  Eric: Actually, do you want to just review that real quick so you know, for the listeners, because  Hart: Yeah, sure. I'll try to talk about this in a general perspective Most software today uses open source components. And when you're actually [00:40:00] writing like a big software program, it's build it's like putting together a sausage you're gonna pull from all of these different libraries and, and then you're gonna package it together into a single program. Of course, you. Traditionally, right? You don't know what's in the sausage of what you're consuming. And someone might build a software package that they claim is secure, but really under the hood they're using several, less secure or less studied. Software packages. And, if there's a vulnerability in one of these dependencies it can propagate up to the top level project. And that's a big problem. And that's what happened with the log for J Bog at a very high level. And one of the. The goals of the open s f is to help stop this kind of bug and to make people aware [00:41:00] of, the software supply chain. So, you know, if there's a bug and a downstream project, you can get notified and, and update accordingly. And it's also, focused on project health metrics so that, you can see if your dependencies are following best security practices. All of this stuff. It's a really interesting project. And, you can, we could easily do an entire podcast talking about, one of the like 10 tracks they're doing at the open s f , yeah, it's a great, you can talk to Brian about it or David Wheeler. All those guys do excellent work. But yeah, it's a really good effort. And we're trying to tie a Hyperledger closer to a lot of the stuff they're [00:42:00] doing. so that, helps us and helps them. At least helps them, but  Eric: it's a constant race for security on as it relates to public or private ledgers. So what do you see, if you look out to the next couple of years, three to five years for Hyperledger, how do you see it developing? Whether it's, versus other systems or directions that it heads in. What really makes you excited?  when you look ahead? Hart: Oh man, if I could give you a great answer to this, I would. I would be a stockbroker or a venture capitalist. I think we're really seeing, we're gonna see more adoption of decentralization. I think we're gonna see a lot of the traditional finance companies and banks that have been, they're very conservative organizations by nature. [00:43:00] And they've been slow to adopt. But I think we're gonna really see people start to adopt blockchain technology for some of these use cases. We've seen a ton of interest in Central Bank digital currencies or cbdc. This has really started to take off in the past year. We're seeing more and more of this and I think we're really going to see an adoption in finance that, everyone has predicted, it it's taken time. I think we're gonna see a lot more hybrid systems. We're gonna see a huge focus on interoperability, right? Everyone's accepted there's not going to be a blockchain to rule them all. And all of the blockchains in distributed ledgers are going to need to interact with all of the other blockchains in distributed ledgers and even legacy systems, right? You're going to want your blockchain to, [00:44:00] to be able to communicate and interact with centralized systems as well. I would say these are two trends. I hope that people renew their focus on privacy and confidentiality and other things. For blockchain, I think digital identity is really starting to take off. We've seen a number of governments start to adopt self-sovereign identity and similar principles. This is less popular in the US than it is in Canada and Europe.. So a lot of people in the US are not as up to date on the trend. I think digital identity is the backbone for everything in web three. So I really expect to see that take off.  Eric: Excellent. All right. It was it was great to have you on the podcast. If anybody wants to learn more about what you do, where should [00:45:00] they go?  Hart: Yeah, we have a ton of resources. On our website, on the Wiki. On our GitHub so it's all available online. We're an extremely transparent and an open organization. You can find almost anything you want. And we encourage people to just, join up. We like having lurks. If there's a meeting you want to join or an email list you want to join, just do it. You know it, it's all available. It's all open in public. So, we encourage people to come out and get started and kick the tires on stuff.  Eric: Excellent. Thanks so much for coming on the podcast.  Hart: Thank you very much for having me.[00:46:00]