Fuld transskription af Episode 132: Open Source Software for Manufacturing with UMH's Alex Krüger

steel cutting, blasting, then we connected all the machines with sensors, et cetera, and provisioned also hardware because we needed edge devices to fetch all the data and forward the data to the cloud. And this was on where we put the United Manufacturing app in place and then new problems came up. For example, now I need not one edge device, but 20, but like installing 20 edge devices for two guys is like a lot of work. And we just built a tool to automatically deploy all our services on one tool, make it versionable because we were the guys getting caught when something does not work. Yeah. And we want to minimize that amount of calls that we got. So we put in like alerting infrastructure tests, et cetera, against that infrastructure. And this is how we then evolved from like an architecture integration to more product because we need to standardize all our integration efforts. Natan: You were on a integration. Set of requirements and journey. Yeah. And a product came out. And I think what's cool about this story is that you were like two young guys out of university and you had to do all this work and you had a lot of choice, right? Yeah. I guess McKinsey and the other early customers were like, okay, we just need this to work. They, they didn't really care exactly what you choose to do it. Yeah. So. Tell me a little bit about those early choices, because I think those will inform the next set of big ideas that we should tackle here around, for example, open source for manufacturing. So what choices did you make? What tools did you pick with Jeremy to build the first, uh, UMH implementations? Alex: To quote Jeremy, or J, he was massively frustrated with the tools that you could buy. It was back then. You could buy OPC UA converters, S7 converters, edge devices from reputable brands in Europe and in the US, but the tools were at some time inferior and there was so much good tools already out there. So talking open source, there was node red, Natan: but the price was okay, right? Alex: The price was never the issue, Natan: I shared the same sentiment on, you know, those converters and all these, but on top of all of that, not only they're inferior and complex and honestly, in times bad software, they're also so expensive and like this premium, like, for example, I can't stand it. You need like a, an embedded computer, which is just a computer. But if the word industrial is before the PC, suddenly the price is like 20 percent or 30 percent higher. And it's nuts, you know, sometimes you need IP65, but come on, you know. Alex: Yeah, it's just random certifications stamp onto it and then price double, triple. And the problem also, especially on the software side was we needed to move fast because projects making the high pressure and then now we need to buy something and try to buy something in an industrial context. Weeks, months before you have the product in hand, and then try to find good documentation because you have a problem. Oh, now you need to text somebody. He comes back to you three business days later and sends you a random PDF, but on open source. So now it shows NodeRed for example, for connectivity. Back then you have YouTube videos. You have like communities actively discussing and also some random guy in Brazil, for example, outside of all business hours inside sales is just there to help. It's like so more traction on those tools, so better experience. And there was like no second thought about going closed source on the commodity tools. Natan: Yeah. So obviously. We share the same point of view and I believe passion towards like what the open source movement needs to do for technology in general, and certainly for operations and manufacturing. But I'm kind of curious about your perspective, but with Tulip and, you know, we're thinking about this from the lens of the continuous improvement person on the line of personal excellence, but also the control engineer and the IT OT architect, you know, there's many titles in sort of where we roam. You go there, often they know about the open source tools, but, you know, if you try and compare how they're used and adopted within operations manufacturing compared to any other engineering field, you know, that makes hardware, software, what have you, there's no comparison. It's like 0. 1 percent to like 99%, I think, or I don't know. Why is that? And how are we changing it? Alex: I think it's just on a speed thingy. So I think OT, so if you compare how to choose and design systems in either IT or OT and IT is all about user experiments, moving fast, scalable data infrastructure, you need to handle millions of users or trillions of data points per second, whatever. And OT is actually only about. So having something that works and is reliable and you can just install and let it run for 20 years. So there is no need for moving fast. Natan: But hold on, is it actually true nowadays? Alex: In the infrastructure perspective, I would, I would add. Natan: But I think generally speaking, sure, when you build up a value stream that needs to run 24 seven for the next 20 years, so to speak, though, I don't, I don't actually know a lot of them. I saw some statistic, I forget the source, that the mean time of a value stream is like five to seven years or something like that. Of course, there's some industries that go for longer, but because there's reconfiguration, new product, blah, blah, blah. I don't know. I feel like that's a little bit of what people say about infrastructure because that it kind of gives you a pass not to do continuous improvement on the infrastructure level. And I just don't believe it because technology cycles, even on automation setups are, they're still affected. Alex: I think they're moving, but if you just look at the, I think application is moving faster. So MES is now being microservice based architectures away from monolithic. But if you look at, for example, controllers or like SCADA systems. DCS

systems, everything is still based on Windows XP, for example, because it was certified in 2013 or whatever to run in this scenario. Natan: That's concerning. Alex: It is. It is. And this is also why they lock it away. It's like, Oh, we have definitely a lot of vulnerabilities in our infrastructure and we cannot afford to let the internet in. And this is an also what hinders innovation. IT is blocking security arguments. Natan: How are we fixing that? What is UMH approach and how does it relate to this idea behind the unified namespace architecture that, you know, I think everyone is still learning a lot on it in our circles, which we roam and there are different takes on what it is and what it's not. And so I'm kind of curious to get your take on two things. Like one, how do you see unified namespace? Well, first define it in your own words. Then why is it different than the traditional way of thinking about ISA 95, Purdue models, and is this hype or is this actually good benefits for manufacturers? Is there a real value in this approach? Alex: In my words, quite technically, it's an event driven architecture. And this is from Z. It's not a hype, it's already best practice in IT data streaming. So if you now design an R& D organization, sales, logistics, whatever, you would have like this Kafka broker in the middle and stream data through it. And the unified namespace is a new term of making this applicable for engineering. This is also where Walker, I'm not sure if he coined it, but he made the concept more accessible for people who have no idea what event driven architecture is because it's IT, it's theoretic, and now it's super tangible also for engineers to absorb. Natan: Walker by far educated more people on the internet on what unified namespace is and Promoted the core principles and has a lot of great contributions to people understanding how to put it practically in use. And I, I kind of agree. Yeah. Event driven architectures have been around for decades, but talk to me about how it's different than ISA 95 in your view. Alex: So also technical ISA 95 is from ETL. Like extract, transform, load in the IT terms to get something from one layer, transform, and perhaps destroy some information and push it to the next one. So architecturally super different, but I think far more important is, it's a different mindset on how you access data. Data is with a unified namespace, commoditized. It's easily accessible for users, for use cases. And from there on, you also A little bit tech of the traditional vertically integrated systems in manufacturing. So you buy from this manufacturer, the PLC, so you need to buy the SCADA system from the same one. You also add now the historian functionality from this one, et cetera, et cetera. And with the unified namespace, you decouple applications. Because data is one, but application is theoretically replaceable. And for there, you can just pick and place components around this unified namespace. Natan: Right. I think that you're hitting on the key point because all that sort of modularity, clear interfaces, event driven, and may I add. PubSub. You want to explain PubSub for our less technical audience in layman terms for a second, when why is that important to have a PubSub architecture? Alex: I'm not sure if I can do it in simple terms. Natan: It's okay, we'll include some references, but we are two nerds, we are two engineers, and augmented ops can also get nerdy. Alex: So Message Broker pops up if you like running Reuters, like the message agency, for example, and you now get reports from all people now reporting on Africa, on South America and in Asia. And you now have people in Europe interested in Africa. So you, and you have people also in North America interested in Africa. Somebody can publish. Messages on Africa and everybody in Europe and who's before interested in such information from Reuters can subscribe to those streams and they automatically get redirected the message from where it comes from. This is the same for PHC's information or other information that Different applications can want to have. Natan: Again, this is where we've been pushing the open source use, whether it's OPC UA or, you know, MQTT is built into Tulip. OPC UA servers are built into Tulip. We have Node RED is built into our edge device. All of that, what I'm seeing customers are doing and what it helps them think about their architectures as open, evolving environments where they can mix and match different systems. You know, everybody's kind of have different ways to talk about this, but like the welding of the IT and OT, because, you know, you said before this is changing the historian perspective because you're going to have to go through, okay, you got, you got the data from this PLC, but for one company, uh, they need to put it in Snowflake and another one need to just put it in some local copy of a SQL server because maybe it's aerospace defense and that flexibility. That's what customers are buying and it's also in a way they're buying future proofing because you know, IT people have been doing this for decades. Of course, stuff improved and so on, but I don't know, the origins of MQTT is not like the past three years, you know, it's been around for a while. So where is this going? Like tell me, like, okay, you're, so you're in the forefront of this and, you know, obviously followed all the great templates and, you know, you have a great knowledge base and community that have been, um, Nerding out on great stuff if you go to umhdocs as a resources. So from where you're seeing it, what's the future of open source and operations? What's, what's the stuff everybody should be tuning into? Alex: First it's knowledge. This is also where you and we are like quite heavily pushing on because before OT can apply a lot of those technologies, I think we need to create this common ground on names, technology, architectural types. As soon as we have

that, the unified namespace could be like this coupling in between OT and IT and also unblock these situations where IT is forbidding innovation. So this was also a perceived reality. I see when I talk to a lot of OT people, IT forbids me to publish data there. They forbid me to install edge devices here and there. And as soon as they speak the same language as IT understands the principles and have the unified namespace as a connecting piece to, for example, push all their SCADA or like all the PLC data into Tulip to build a better application and make it even then secure, et cetera. So the IT is happy. I think this could fuel a lot of innovation and make it way cheaper to innovate also. Natan: You know, we like to give our listeners some practical stuff. If you have to list like the five most important open source things to look at that operations people should know and understand. So we, we named one like Node RED. What else? Let's get really specific. What else would you recommend? Alex: Oh, I think I would actually also to really dive into like the fundamentals. How does networking work? Like, okay, this sounds also stupid and easy. You have cables and IP addresses and ports. Yeah. But what are the transport mechanisms in there? What like the advantages of UDP, TCP, um, the trace so that you can speak the same language. The same goes also for databases. I see a lot of confusion on databases because The difference between OLAP and OLTP, some databases are there for a routine transaction, like an SQL database. I'm a shop and now I sell two bananas. So please delete two bananas from my inventory versus like a snowflake where you push all your data and then once a month run a three hour query against it. And if you don't have this right. You can also get to good discussions, the same goes for how to connect building blocks together. So this was also back then a super hard confusion on ThingWorx because ThingWorx or don't want to rant about too hard, but they published them as this building block in the middle where you connect everything to. But architecturally It's like an database ish approach, but you cannot connect real time use cases with database. So what's the advantage of communicating point to point, database, message broker? I think getting those fundamentals right, and then go into SQL database, go to Node RED, go to MQTT, go to Kafka. And there's so much great content out there that you could just digest. And also perhaps build some cool dashboards in grafana. Natan: Grafana. It's getting better all the time, I feel like. Alex: Yeah, definitely. I think so. They even add more visual stuff that's important to manufacturing companies. I think they added also this canvas feature. I think it's built actually for IT observability, but you could just also draw like your lines there and put like temperature throughput OEE on there. And it's so ridiculously stable. It's built for enterprise IT environments. And you can just get it for free in the sense of great tooling. Natan: That's a good wrap on that. You know, we went in and out of depth on the open source stuff and tech, but really, if you zoom out, so much scale and complexity is handled by this open source infrastructure, whether it's like the Linux operating system, the Grafana, the cPanels. The Kubernetes, of course, and all this sort of stuff that honestly, we're taking more and more for granted the past few years, because, you know, when Kubernetes is introduced, I want to say it's coming on a decade plus, right from the beginning, but it's, it's not that old, you know, but in technology terms, it's like established and a huge community and great tooling. And that's not going away because it supports the internet. It supports any data center that runs today and to think that that's not going to run operations is just nuts. It's just ludicrous. But you know what's interesting in the few minutes that we have left, then we do it like a quick fire round because we have to wrap up pretty soon. So, okay, we agree on the open source infrastructure, all that kind of stuff, but now how do you think it applies to the higher level application? So I have two. Questions, two products. How do you think all this is changing historians in the traditional sense, because historians is kind of like, I'm trying to figure out what the hell is a historian anymore. So that's question one. And then I have a provocative one, which is, if you take UMH plus two, are we getting To the promised land, if it's at all a promised land of headless MESs, you know, not just composable, like composable next gen MES. And you said MES is becoming more microservice, which I'm not sure I fully agree. I mean, I'll believe it when I really see it, but you know what I mean? So let's start with a historian. So where are historians going? Alex: I think they will be here for another 20 years because they're too deeply embedded, especially in the GXP field. Because certifications, requirements make sure that they are there because they are fully validated, etc. Natan: But wait, why can't they use UMH and Snowflake? Isn't that, uh, you can't validate that? What's the problem? Alex: Because if they use Snowflake or like an open source tool like UMH, I think it will take a few years to penetrate in the complete market. And like those systems, especially in the pharma context will still run then once they're there at least 15, 20 years. So I think in that period, they're still relevant, but I think they don't solve a problem anymore. They come from a time where like. Data was expensive, storage was expensive and database are not that good in compression. So they developed their own algorithms like swing door, et cetera, and sampled down data. But now eight terabyte hard drive costs, I don't know, 40 dollars. This is ridiculously inexpensive. And then also storing in S3 cold storage, it's like 0, dot, 9 zeros and then one cent per gigabyte per month. So it's, they

don't solve a problem anymore. Because it's already been solved by progress in IT. Natan: So first of all, I kind of agree and disagree because I think 15, 20 years is a long time, even for GXP. I mean, our journey in GXP, people told us, Oh, you know, you'll never be able to run a pharmaceutical manufacturing in the cloud. And, uh, it's hard to validate and all this kind of continuous delivery and FDA don't go together. And I was like, well, not, not really. You know, we've been. We're audited. We're running, you know, thousands of station in production in GXP environments and it's all running from the cloud and it's all, you know, has SLAs and zero RPO redundancy and that's only possible from the cloud and so on. And, you know, we, we don't think about us as a historian, but we definitely collect a lot of data at the source and contextualize it and we obviously ship it to many other types of systems. So I'm seeing a little bit of a different trend and I think a lot can happen in 15 years, but that's one reason I disagree. The second reason I disagree. Is I don't think everybody thinks about the needs and requirements of a historian with a GXP level. I think a lot of when we say historian is like, well, maybe this part we do agree like giving my data anywhere. If I need it, I'm okay to wait like even two minutes or 10 minutes fetching it from cold storage, which Cost me nothing. And I actually don't want it in a data center somewhere in an air conditioned room in the factory because only Johnny has the key. And when Johnny is sick, I don't have my data, you know, that's my point of view on the story. And I think that space is we should really watch it carefully. And there's a lot happening there. That is super, super cool. Yeah. Alex: I hope I'm wrong. So I hope it goes faster. Natan: I think you're wrong because you have a big business terraforming historians if you keep going. Um, okay. And now to the last and not less exciting question. What is your take on headless MES? Alex: So we also get requests from large industrial companies who want to slice down the MES because it's like this ridiculously large monolithic application with no clear definition. Natan: Are you saying they're not satisfied with their MES? Alex: I'm saying that I think it's, it's not maintainable, it's not usable anymore. It's like, I think it will be sliced down. So it will be stripped down to core functionalities, planning, scheduling, orders, pushing to the shop floor. And then I think it will be then microservice based. I would agree. It's not, not fully there. And I actually also don't didn't see it at scale, but I think this is a way to go because now everybody's rebuilding the MES and really want to make sure this functionality, for example, interaction with the worker of all, let's use Tulip therefore, because it has a perfect gateway to interact with the. Persons on, on site, then they need another tool, perhaps for AI optimized scheduling, whatever, but they want to have this flexibility. And then I see you in S in the middle, S connecting all the dots there. Natan: On data operations and backbone to have an architecture that is scalable and using future proof open source tools. So Alex, this has been quite a ride. Thanks. I think we should do it again. We'll find more cool topics. So we're going to include some of the references for a lot of the technical mambo jumbo that, uh, two nerd engineers are going on here today. So with that, we say goodbye and we will see you again soon. Thank you so much for joining. Appreciate it. And, uh, you know, like in the open source fashion, you know, you should like, uh, post a lot. And so we can fork your stuff and, uh, contribute it back to the community and reuse it and build a better operation stack for all our stakeholders. So important. Thanks a lot, Alex. Voiceover: Thank you for listening to this episode of the Augmented Ops podcast from Tulip Interfaces. We hope you found this week's episode informative and inspiring. You can find the show on LinkedIn and YouTube. Or at tulip. co slash podcast. If you enjoyed this episode, please leave us a rating or review on iTunes or wherever you listen to your podcasts until next time.