"Advanced" Analytics and "Predictive" Maintenance

Optigo Networks: promise of advanced analytics and predictive maintenance is that the data was there all along. We just weren't listening to it. But how true is that really? Can the systems most buildings have today predict what's about to break? In this episode, I'm joined by Justin Carter of Enfra and Rudy Bohins from Alberio Energy to dig into what predictive maintenance actually means, what data you need to make it work, and whether the OT networks running our buildings can even handle these analytic layers being sold on top of them. I'm Ryan LaFlam. And this is how to handshake, episode six. Hello, welcome to the episode. thank you. ⁓ I'm once again, it's me, Ryan, and ⁓ sitting in the co-host seat this week is ⁓ our head of growth, John Atala. ⁓ hello, John. Hello, filling in for Ping, who is currently in California. And ⁓ joining ⁓ us over here as well is ⁓ Rudy, Rudy Bohintz from Alberio Energy. I'll just give you a moment to let people know what you do there. Sure. Nah, I appreciate you having me on. at Alberio I lead our managed services center, which handles our intelligence services and smart building services. ⁓ so Calm is our alarming platform, we do simple building analytics and some cloud hosting, all on top of our building automation divisions, ⁓ data center divisions as well. And ⁓ I think we have Justin. Hey. Perfect timing. Perfect timing. We're ⁓ we're just jumping into ⁓ introductions here, Justin. So I will send it right back to you. Justin, please introduce yourself. ⁓ who are you and ⁓ and what do you do? Yeah, so I'm Justin Carter. I work for ⁓ Infra. I'm one of their ⁓ controls technologists. ⁓ I'm based out of our Fateville Arkansas location. Perfect. And we're here today to talk about predictive maintenance, preventative maintenance, predictive maintenance. ⁓ you know, they have a lot of different terms, a lot of different marketing terms. I just want to jump right into it and ⁓ I'm gonna start off with the first question here. and you know, what do we mean by predictive maintenance? You know, how is that different if it's different at all from what people are calling proactive maintenance? Because sometimes these can be pretty interchangeable. So ⁓ I'd love to start with you, Rudy, and then ⁓ I'll over to you, Justin and and John. Sure. ⁓ we've heard a lot of them be interchangeable. Some of it's marketing fluff, I'll call it as well. ⁓ there's a lot of that buzzwordy type stuff out there. so I think it's really the big thing is where the ⁓ the actions meet the road, right? So I I always look at predictive as using your historical data to predict the future, right? I've had, you know. this many air handlers in my building and three of them failed after five thousand run hours, whatever it may be. So when I get to my next air handler, chances are I'm gonna get there. So I wanna do maintenance before that happens. ⁓ you know, a little bit of I think where I draw the line on the nuance is I'm using history data to predict the future. Proactive is more of, you know, less using that history data and kind of more on the ⁓ like an oil change in your car. Yeah. I change my oil every 5,000 miles because somebody told me to once. And that's what you do. ⁓ that's proactive. So, you know, not really based on any data that is constantly changing, but you know, based on kind of runtime and and things like that. Justin, does that is that ⁓ along your lines of thought? Yeah, definitely. ⁓ to take a little step further, I I would say if you were talking about the two words, the predictive maintenance would be. I'm a big organization. I've spent two million dollars to deploy this predicting algorithm, AI, whatever you want to call it. And I'm not changing my culture though. And I it tells me the chill water valve actuator is having an issue, but I'm not calling someone to come fix it. I'm not using my internal staff to fix it. So now you're predictive, but you're not proactive. I think maybe that's a little bit what I've actually seen in the real world. ⁓ it it's ⁓ it's a shame to watch some customers or hear someone go up and say, Man, we did a program and we just never did anything with it, you know, and that's I think the proactive part is changing your culture and ⁓ the predictive's great, but it's nothing without the proactiveness. ⁓ so I I love that. I I just to expand on that and maybe be a stickler for words. I mean, to me, proactive underlying that is sort of intentions, right? Like I want to be better about how I'm doing the same job that I've been doing for the last, you know, however many years. I want to do it better. Therefore I'll use the predictive, you know, analytics or the tools that are my at my disposal to allow me to do that. But you're absolutely right. Predictive analytics without Proactivity without wanting to get ahead of things and do better maintenance, run better programs is is pretty useless. And that's a really great segue into our our next question. And I think you mentioned it is predictive ⁓ analytics or advanced analytics. Again, vendor marketing, you know, du jour, you'll hear that a lot. What does advanced analytics look like as opposed to say regular data and regular traffic? And and how is that different from say trend graphs or dashboards that operators have been using for years? So ⁓ you know, if we approach a site and they they say they have analytics or they feel like they have advanced analytics and I go on there and it's something say put in by the incumbent controls contractor and it's histories and trends and nice dashboards and you get a little bit of that, but then if you go to market and you have the actual advanced analytics, you know, ⁓ community. Typically what you're going to see in their products is this is the issue. This is how much it's costing you. Here's a priority list. This one's compliance based. This one's cost based. So say if you have a hospital, it's like, this one may not be costing you money, but your OR is out, you know, out of compliance. It's above 60% relative humidity. You know, and that's that's probably what when you say the word advanced analytics, that's where my head goes. every client's a little different. Class A office building's not going to care about certain things like that. But ⁓ that's that's kind of sometimes I see people say they're doing advanced analytics and I'm like, ⁓ that's just a history or trend chart. Yeah, I think to your point, Justin, the biggest one is what are you doing with that data? You know, you could have all the advanced analytics behind it. I still think that having an operator be able to see what's going on and easily understand it is kind of, you know, what makes it advanced. And all of our jobs kind of on the ⁓ The analytics side, the control provider side is to make it as simple as possible to take action on that data. So you can run whatever AI models and you know stuff in the background to make it advanced, but if it's not directive for that client, you know, that's to Justin's point, it's different for hospital versus commercial office building. They're not all the same. So, you know, if you don't have it customized to them and their needs, then it you can do whatever you want and it's not going to make a bit of difference. You know, and that's where that that changing your workflow kinda becomes really important that you can take the advanced analytics and have it look at all sorts of different key performance indicators and you know metrics there, but boil it down into something simple, something that you can take action on that you can easily see, whether it's a trend or a dashboard, and then prove that you fix something. And that that's really what makes this stuff kind of more advanced, I think. Right. What are we usually looking for in advanced advanced analytics ⁓ as as we throw out in the questions here. What kinds of problems are these tools, these advanced analytics do you feel are really good at catching? And what are the things that they're still missing? You know, this could be your wish list. What are they good at? And where are the problems or where they still need room for improvement? Yeah, so I mean you're catching some simpler stuff for sure in the advanced analytics. You should be catching things like say ⁓ damper on something goes open and the airflow doesn't go up, you know, or it goes down and it's like, okay, actuators flipped on polarity or the PID loops backwards. You know, it did the opposite of what I expected. Some of that simpler stuff, that's that's been happening, I'm sure, for over a decade. So I've seen some of that that works pretty well nowadays. We do some advanced chiller analytics where we have found things that the chiller companies weren't finding, you know, with their techs on site. ⁓ we're able to look at pressures in the chiller and Does this make sense for that given refrigerant? That was some of more of the advanced stuff, I feel like, that even sometimes we're not not great at it and we have partners and we all look at the data and it's it it takes a minute. You know, that part might not be there sometimes. ⁓ but that's ⁓ there definitely is a threshold. I don't know exactly where it is, but simple air side stuff and water side stuff, I think the markets got pretty well. Yeah, I think it's where you start, you know, comparing Equipment performance versus rated performance and and published performance. I think where it really gets I'll call it advanced is taking all that data out of a building and comparing them one to another by equipment type. Like Justin said, the you know, equipment running after hours on holidays, scheduling damper stuff, you know, that's all been kinda baked for a while in the industry. I think that's kinda your retro commissioning scope, right? Like That everybody kind of has that part baked down. And then it's a matter of looking at performance versus other typical things of that nature. So I've got 15 pumps in a system. Why is this one pump operating differently than this other pump? And why does that only happen at certain flow rates? That's where I think you're starting to get into some of the more advanced things, but that requires somebody to generally look at it and kind of have an understanding of the building and the history there as well. I was gonna throw this at you, John, because I it seems to me that everybody is is is kind of hinting at the idea of what makes things a little bit more advanced is bringing in a larger context around it. Like you said, the situation or in comparison to other performances, real world performances versus stated performances. And I know, John, we have a very different sort of experience from a software side. And ⁓ I'm just curious what you would or what you know we consider in the hands on for Optical V and the advanced side of our analytics as opposed to say the core. Yeah, I mean what I what I was gonna say is that, you know, our analytics are going to be very different from what you might get out of a tool like Wireshark where it's just blah kind of all kind of there on a page for you. I mean, we're really talking about going from looking at, you know, hundreds of thousands of packets in a dashboard to getting needle in the haystack type information, you know, at your fingertips. So it's it's pretty game changing from that, you know, respect. But there are things that it may not do quite as well as you'd like, particularly for MSDP, right? When you get down to the physical kind of, you know, wiring level and electrical signals, that's unfortunately just something that you can't get in the zeros and ones of the packets. So that's the only limitation of the the technology today. But you know, hopefully that's going away with with T1L and with, you know, backnet IP ⁓ migrations and whatnot. Predictive models seem to to be pretty heavily involved in the obviously predictive and the proactive side of of learning and they need data to learn from. ⁓ so, you know, when we look at building systems specifically, what types of historical data would you say are are, you know, from the network and the controllers are actually the useful bits of data that would be for for a predictive model. ⁓ for people at home who may be trying to ⁓ set these up at their own organizations, how far back should they be holding on or pulling in these sorts of historical data to start building ⁓ something that's will be valuable for looking ahead. The historical data, and we've had some customers that have had old legacy systems, old legacy servers that have had data for 15 years just because they had it. ⁓ They didn't really have a use case for it, didn't know what they were going to do with it until later. So I think if you already have data, you know, we're primarily talking about building automation systems, just keep your trends on everything. ⁓ because you kind of never know. And I think You know, the other thing is the industry's changing so much that more and more data is becoming available that you kind of don't know what you're going to do with some of the data. So you may as well keep it as almost an insurance policy and and that way you can learn from your past as well. Now that being said, most of the data we deal with is pretty dirty. You know, there's not many naming conventions. I've seen, you know, space temperatures have five different names depending on, you know, who decided to program it at that time or what manufacturer it was. So like it's all relatively dirty. I don't think you need to spend time to clean up all of that data to make it useful. Like take it one step at a time, figure out what you're gonna do with it and then invest your time to clean that data before you try and boil the ocean. Right. Because there's there's a ton of information out there and you can get really hung up trying to, you know, do the direct data modeling on, you know, this small set of data points that has almost no return on investment. So I I think that keep all the data that you can, cost effectively and you know, everything like that, but then figure out what you want to do with it, then focus on getting that data cleaned up from a data modeling standpoint, and then kind of show the results there and then kind of chip away at it is is what we've seen as being successful. I'm gonna spin that as a question to Justin, ⁓ which is You know, as it relates to network data specifically, is there any value in historical data whatsoever, or is it more of a what have you done for me lately kind of scenario where, you know, really we just care that the network is in optimal shape today? And what happened yesterday is, you know, maybe a matter of fact to look back on when we need to, but it's not really valuable for, you know, today's operating ⁓ conditions. Yeah. So if you're talking about, say, H V C energy data You know, consistency is more key than links sometimes. I mean, you do need kind of seasonal, like if you got summer, winter, and kind of shoulder months, you can do some good analytics. Network's a little different. ⁓ cause the networks probably don't change much based on seasonal unless you've got some equipment outside, maybe or something. But yeah, network is typically more of do you got current projects going on? Is someone touching the network? Did you add 18 backnet devices? You know, what is that? Why is it all duplicated? The it it's definitely two different things. ⁓ I I would say it's probably more of a what have you done for me lately and what have you been doing recently to your network? Yeah. I I wouldn't need a whole lot of data probably from the network to say, ⁓ the last three weeks you've been acting like this and I see all these devices and then all of a sudden something's different. ⁓ where H V C data ⁓ it it can it can be definitely a different range ⁓ because there's a lot of different things going on and in that world. I wanna circle back around to something you mentioned, ⁓ Rudy, ⁓ about ⁓ the data being dirty. ⁓ dirty data. We we hear about that a long a lot. One of the sort of key things about making an analytics program work is having ⁓ you know, underlying data that is in good shape, right? You know, we if we say point names, complete trends, time stamps. all of those things. ⁓ so yeah, like how much cleaning and and tagging does a typical building need before it's sort of ready perhaps to be layered on with additional ⁓ additional analytics or advanced ⁓ predictive software and ⁓ you know for for people again who are probably organizing either at an SI or or at a facilities team, where would that usually fall in the organization to who gets the ⁓ who gets the ticket? I suppose is the question. I from what I've seen, you know, I kind of sit in managed services, we do both, the analytics side, but we also do a lot of the BAS support too, right? So we kind of get both ends of that spectrum where sometimes we're the SI and we're going in and doing some of this tagging, and then otherwise we're the, you know, data analytics guy. We've always approached it from the data analytics side is that we're gonna clean the data within our analytics platform. because we kind of know that what happens in the facilities isn't gonna change. They you know, they still want their zone temp to be called zone temp, whether we want it space ten doesn't matter because they need to use, you know, they need to use it on a daily basis. So we always found it more economical to do it in our analytics platform because that's where we have more control and that's the least impactful to the building operators and the facilities team that's using it. ⁓ That being said, I mean it's you know, it it it can be quite a lot. ⁓ we try and tag by equipment type and data type and everything like that and go through and do a a standard there. But we do also focus on, you know, what are we looking at, what are we trying to get out of this program? And then we have a couple of, you know, internal commissioning things that we look at for data integrity. So, you know, I've got a VAV box, I'm expecting 15 points out of it. How many do I actually have? How many fit this, you know. Spec from you know a tagging model, and then how many ⁓ history points do I have? Is that what I'm expecting to have? Do I have big gaps in data? Do I have sensors that aren't changing? You know, and that that's kind of the other problem is you've got operators involved, right? So they go in and do a maintenance, they have a hot and cold call, and they're overriding something, and sometimes those overrides stay, and then sometimes they remember to remove those overrides when that event is over. So from trying to make a you know, a regular trend curve of this is what I expected it to look like, but all of a sudden I flatlined for three days. Like that makes it real tough to do any sort of ⁓ historical analytics or plug it into an AI engine or anything like that. So we try and look at all that stuff as part of like a data integrity ⁓ effort when we when we onboard a project and you know go through and then we try and factor that into the analytics rules. And Rudy, you just touched on something that was interesting which ⁓ you know with regard to data integrity and ontology and what we found a lot at Optigo is that there's very much that sort of focus ⁓ from the SI or from the building owner or for whoever is you know deploying these technologies. But we always like to you know kind of relate that back to the network comms issues that we see. So for example, ⁓ if you had a you know building automation technician that you know mistakenly was having a bad day Or was forgetful and programmed, you know, the same BACnet device instance ID into two VAV controllers. And now you're pulling that data into your, you know, into your your your software or whatever, it's probably pulling the wrong data half the time. And now those systems are making certain assumptions or running different, you know, FDD rules or whatever the case may be on completely flawed data to begin with. So I think it just kind of, you know. serves as a good reminder for folks who are deploying these types of products that ⁓ you y you don't want to have one without the other. You really want to have, you know, proper network performance and device addressing and all that kind of core stuff ⁓ before or at the same time as you're deploying these F D D tools because otherwise who knows whether you're operating under good assumptions to begin with, right? Yeah. I know when we're recording these podcasts, if if I'm not careful to give everything a proper name when it when it you know, a few weeks down the road, if I need to come back and make edits, if everything is called take one, then you know, it's it's sort of the same thing. And and Justin, I saw you nodding your head through that. I'm assuming ⁓ this is ⁓ the a shared experience for you on your end as well. Yeah, so we we call the offering like monitoring based commissioning and we have a team that does like the data part. And so there's kind of I I'm thinking maybe three verticalities here. One is I got a connection and it's stable. And that means no duplicate backnet IDs, blah, blah, blah. Right. Second team comes in and ⁓ kind of does that tagging. And so like you can use Haystack or whatever any other product we use uses that. And then the last team, the one that actually gets to have the fun and do the analytics, the prerequisites for for them to even see it in the in that product that we use, it's gotta have a stable connection first. So, you know, we're using Optigo to make sure the backnet scores are good and and other things before we even give it to the site build team. And then if the site's not built right and you haven't really checked like air handler two serves these 15 VAVs, if that's not done, it doesn't make it to the team that's trying to do the analytics, get an issues log in front of an owner. And so I, you know, I don't think I'm giving away any secrets there that this is what the analytics world has to deal with. And if you're not at least doing those kind of three verticalities that I'm laying out, you're having a tough time for sure. There you go. There's some ⁓ I hope everyone wrote down those three tips. And you're exactly exactly right. Kind of understanding ⁓ understanding your network is is the first step. You know, we we've used the the visual back net in the past too, and we won't touch a job until it gets to a certain score because otherwise you can't trust anything and then you're throwing good money at bat. Right. You know, I I we've gone into analytics customers and they say, I know I have a hole in the wall. I don't need to pay you to tell me that. It's like no, you don't. So fix your fix your foundation and then you can really do the the fun stuff with it. So Justin's entirely right. Pete, stop me if if it sounds wrong, but coming from a from ⁓ a previous role on the other side in the IT world, ⁓ one of the things you you hear a lot about, especially in the last few years, is is edge analytics, right? The idea of doing the data crunching out where the data's coming in rather than trying to to shovel it all into one central computer to to try and figure that out. ⁓ and you know, you don't hear about it a lot in in OT networking and in smart building. At least I haven't to this point. and it's probably not surprising, you know, that a lot of the OT networks that that we deal with are running a lot of older or low powered, you know, hardware that really may not have the horsepower for any meaningful compute at the edge from your own experiences being out in the field, how real is that constraint? And you know, what does a path forward look like to perhaps start bringing in, you know, the efficiencies that that edge ⁓ edge compute can do? Is it a rip and replace? Is it piecemeal? Is it gradual upgrades? ⁓ or is it just pushing the analytics somewhere else entirely? So to go ahead, Rudy. Sure. Yeah. I mean, we haven't seen a whole lot of analytics on the edge yet. ⁓ I think the closest we get is still at the supervisory level within a building. ⁓ you know, we will do it up in the cloud as well, but you know, we're not seeing analytics on a chiller or a VAV box, you know, or or any of those controllers. ⁓ where we are seeing the the edge, I guess, is trends. ⁓ you know, rather than always looking at, you know, the backnet curve out. we are sometimes looking at the trend value because then we get that trend collection on the edge. So if we lose a network connection or IT shuts down a port or something like that, you still have that data collecting at the edge. And then when that connection is re-established, it transfers up to the supervisor, to the analytics engine there. ⁓ So we'll do localized trends there, we'll do extended trending or long-term trends. either in the supervisor or in a SQL database, and then we'll pull those into the analytics engine. I think in the future you'll start seeing more and more compute power at the edge. And I think you'll see different OEMs, you know, whether it's a chiller manufacturer or air handlers, VRF, whatever, they're going to start packaging their analytics locally there. And then I think the whole role of an analytics provider changes to if they know their chillers best, Let them analyze that. Right. We can factor that in and then we can apply what they're finding to the rest of the building because it's all kind of cascades down too. So that's I think that's kind of where the edge analytics ties into your supervisor and then you gotta have somebody make sense of it all. Yeah, you know, to be honest, ⁓ you know, I don't I don't have much experience with ⁓ you know, analytics at the edge. Of of course everything that we get is, you know, is captured from from somewhere on the network. But ⁓ I think Rudy gave a really excellent answer. ⁓ it's interesting where this is gonna go. I will say that it's it's maybe semi related. I was at the Niagara conference a couple months ago and ⁓ was speaking to a gentleman who was talking about, you know, network performance and ⁓ you know, a need to monitor network performance or do anything about it. And ⁓ I recall asking him, Hey, you know, as it relates to Optigo anyway, you know why why would you, you know, not want to investigate or or look at a tool like this for your company? And the answer that I got back was pretty surprising. It was something to the effect of, well, we only use IP networks now. We only build out IP networks. So we, you know, we're not using MSTP anymore. So we wouldn't have any of the slowness. We wouldn't have any of the, you know, bottlenecks or problems that you, you know, your your product ⁓ you know, can help us fix or whatever. And ⁓ I thought that that was I I hope that's not indicative of the larger industry ⁓ as far as how they're thinking about this stuff because ⁓ you know backnet IP does not mean lack of broadcast or lack of congestion in a network. ⁓ you know, it just means that it can go faster and hit harder when it does happen. So I thought that that was an interesting anecdote from ⁓ from folks out there who were thinking about, you know, networking and edge and all that stuff. I think as you add more analytics and people are using the data more. You know, they're they're trying to get building automation data, electrical meter data into the rest of their, you know, building platforms and Power BIs and Grafana. I think networking performance is going to become more important because you're going have more systems and more people getting into this data. And, you know, a lot of what we've seen has been either a serial connection into the building automation system or backnet polling. And whether it's IP or MS T P that'll slow everything down. So, you know, I I think that as analytics changes, it goes to the edge and back up like different ways to connect, we've got to catch up with the rest of the IT world. And network performance is gonna become more critical because you layer on these things and starts to bog things down. Building operators aren't gonna understand why. They're just gonna get, you know, graphics that spin and spin and spin and data that's old and stale because it can't update. So That's where your network performance, you know, becomes more important, not less. As the OTIT convergence continues, ⁓ OT is being dragged into the larger IT world. And what comes with that is all of that extra overhead, that much more data, that much more ⁓ bandwidth needs. And ⁓ I think it was in our last interview who said simply easier these days to just say, you know, if it's on a if it's on the network, it's all IT. ⁓ and we need to start sort of looking at it like that and start raising the expectations of the OT network's performance to that of IT if we're going to expect preventative maintenance, predictive analytics, all of these advanced algorithms running, all of that comes with a cost, right? I know we've restructured my team to, you know, as part of our managed services center that We recognize we've got system integrators across the country. Mm-hmm. We're dealing more and more with IT teams and we don't generally speak that language. So we've taken the approach of we've hired IT people, brought them up to speed on BACnet and Modbus and O T networks. And now we we do, you know, a lot of heavy network stuff through my team because of that type of, you know, interconnection there. So Have you have you noticed that too, Justin, that there's simply more and more of the the IT world creeping in? Absolutely. ⁓ can you all hear me? Okay. Good. So ⁓ apologies. ⁓ IT issue, right? Today. I'm sitting here joining on the phone because my IT is very secure. Apologies. I gotta Well well, you have your test port. You've got your test lab behind you, your OT test lab behind you, and we're we're troubleshooting IT issues right now. Yeah. Yeah. So sorry. Sorry about that. So yeah, I know ⁓ we ⁓ the data delivery team that I've been working with that does that connection. Th ⁓ you know, when we sign the big contracts or say if the big energy is a service type of contract and we and we get in front of some IT department or even if it's just a quick retro commissioning, a little monitor based commissioning, we actually just got a guy that joined the team. I think he's been doing IT stuff for twenty or thirty years. So for the last year or some of my conversations are what is BACnet? How do you You know, what is this and what is that? And when I bring it into certain IT terms where, you know, like if instead of saying trends, you say the word logs, you know, and ⁓ you know, it's like, ⁓ the discovery that we're talking about is ⁓ this type of TCP kind of like thing going on, something similar to it, you know, and and you use a little bit of terminology, all of a sudden, like, ⁓ that's all it is, you know, it it's a broadcast message instead of unicast. It's like, yeah, yeah. You know, and so all of a sudden I I've seen our just bringing in him, our IT security reviews have gone so much faster. And it's I'll tell you, if you at least have one or two on your team that can understand, get past the ⁓ the acronym and terminology, you know, hell that we live in, you yeah, it it it's it's just necessary at this point. I'm seeing as well the industry move to IP based controls. Right. And so you're you wanna do analytics on the edge, you wanna do more at the edge, not just analytics, but maybe more trending, more this, more that. But like the ITOT conversion is definitely definitely something to keep an eye on. And if your teams don't have someone that can speak a little bit of IT and get convert BAS kind of acronym and terminology to to the IT world, you you will have a lot of issues. And so yeah, Rudy, you'll you probably experience the same thing I did. You get You get those ITOT kind of people in in there and they're lost for a second. And then once they hit the ground running, they're like, ⁓ man, it's not that you know, it's it's not that confusing. It's just backnet is a protocol you gotta deal with and modbus is a protocol you gotta deal with sometimes and lawn is still out there a little bit and it's like don't don't it's not scary. Like it's just it once you do it a few times, it's really not that we had we had ⁓ ⁓ Douglas Plumley from ⁓ Dartmouth College on and I asked him, you know, what ⁓ how how have you been able to sort of bridge the gap between IT and OT? And his answer was pizza parties. He said it was the just get people in a room, give them some food and let ⁓ talk. ⁓ and and really it was like getting over that just that initial nomenclature hump. And it was like turns out there's a lot more in common than there is ⁓ a difference. ⁓ and speaking of ⁓ you know, seeing ⁓ more and more advanced things, we do have to talk about the eight hundred pound gorilla in every room, ⁓ AI and machine learning. ⁓ you know, so software that learns patterns from data rather than following just the rules you set out is a pretty big shift, I would think, in the world of OT, ⁓ where everything has been for so long very schedule based, you know, ⁓ very simple sort of if then statements. ⁓ but ⁓ looking specifically at say anomaly detection for backnet IP, how feasible is the idea of AI running the show? Not just being somebody you ask for advice, but actually turning over the idea, which I mean that is the the the marketing, you know, terms that people would love. The end is, you know, set it and forget it. How feasible is actually something like that? What barriers do you think might be in the way? Is it even, or is this just like a straight up pipe dream? Should we pause for the groans of our audience, our viewers who are just, you know, ⁓ no, I'm not another AI question. We'll give them a s we'll give them a second to like they've they've now closed the podcast. We'll give them a second to load it back up and give us a second chance. So ⁓ ladies and gentlemen, thank you for coming back. We are with you. Let's continue. I think it's a requirement. It's a box you have to check when you do a type of podcast like this where you have to mention AI. Yes. Otherwise you don't get through the AI filters to get bumped to the top. Exactly. I acknowledge I will be listening to a conversation about AI at some point in this podcast. Yeah. Yeah, I I th remember seeing an article maybe about three weeks ago where one of the AILMs was like, the way it solved some problem they kept getting asked was it finally just deleted the database. And it's like that I think that kind of answers it for me. It's like you still need the human component. And when the human component is involved and before you do like say global commanding, most automation systems have global commanding, the human still needs to click the button that sends the zone temperature set point to 68 on every VAV box in the whole hospital. And so that's some of the it's almost like you got to firewall certain areas off. And ⁓ that's ⁓ I think that's kind of my answer is that if someone really expects it to fully automate and there's no human component, you probably need to step away for the next three, five, ten years. And come back. Yeah. No Skynet. No Skynet is what we're saying here. Not as of today. No, I mean i in some cases we're still trying to get operators to trust just the regular control system and and the schedules. So, you know, depending on where you're at, I still deal with operators that want to go out and feel the wind and then set their child water temperature for the day. So like I think it's baby steps there. And then the other part is where's the you know, if you have an issue or you have a hot and cold call, who's Where's this kind of decision making tree happening? You know, I think that we're not comfortable there yet from an operation standpoint where, yeah, they're it's a couple of years out for sure. Yeah, and I think ⁓ on the on the backnet or network diagnostics piece, you know, obviously AI ⁓ i is gonna be an important component going forward, but I think it can lack sometimes like operational context, right? What I mean by that is ⁓ example, if we're looking at a campus environment and we're looking at devices that are offline, for example, you know, the the tool may flag that as a an anomaly or as a problem to be addressed. Whereas, you know, maybe it's just, you know, it's wintertime, so the chiller has been turned off or whatever. I mean, that's a simple context that AI could certainly learn, but it's that type of thing, the operational context that's maybe missing from some of this ⁓ what AI is doing and and I think it can be improved on. it's probably going to take a couple more years before it's fully there. And that's a little bit of what I was talking about with that data integrity, too. Is like, you know, you don't have normal curves that operate the way you think they should. You know, you you might have a sensor that shorts and then all of a sudden you've got a 327 degrees and then it goes back down, or you know, a meter resets or something. Like it's not clean historical data most oftentimes. So, you know, we're starting to use it, but it's in choice areas. It's usually behind the scenes and support mechanisms. It's interesting because to to hear some people talk, it's it's almost past what you just said, Rudy, where it was like, you know, take take my historical data and tell me where it should be. It's take the historical data, I'll see you later. Send me a report at the end of the month. And and it's it's kind of that gulf between, you know, the marketing speak and the idea of what they we want it to be versus like you said, the reality, like a fuse could go and and all of a sudden it's making the worst kind of decisions. I'm gonna ⁓ move into ⁓ our second last question here. ⁓ because we are obviously a a software vendor company, we're we're we pay attention to a lot of the things we do, but we don't always get to see what everybody else is is out there offering to to people on on your side. And what we do know is there are a lot of vendors out there, analytics startups, fault detection things like that and peop you know, and and we've been lucky enough to to have a few on the show. ⁓ and then there's also, you know, incumbent BMS providers, ⁓ that are really bolting these sort of what they call the their advanced analytics and predictive analytics modules onto their products. ⁓ I guess the question here is, you know, what's actually delivering value in the market today versus what's sort of mostly marketing? What are the ones that are kind of making you roll your eyes when vendors are contacting you these days? And ⁓ What should a building owner sort of do to cut through that? What should they really be zeroing in when somebody's talking about our advanced analytics? I mean, probably some red flags to me are if you're getting shown something and it's only dashboards and they're not mentioning at all some of this things about like we talked about, like actually connecting to your existing systems, maybe talking about architecture. Hey, what you got? They're they're not, you know, the sales guy's not feeling you out. on those kind of things and they're just showing you the pretty dashboards. But if they're showing you things like that and they're showing you point A to point B to point C to point D, and they're not really hiding much and they're, you know, not sitting behind a pretty GUI, I I think I think you're on to something. Mm-hmm. Well and I I think what what Justin hit on too is kind of where I was going is bef before you're even evaluating something, you've got to look internally and figure out how you're going to work it into your workflow. Right. Right. And that's kind of the A to B to C to D that that Justin mentioned is if you don't have somebody that's going to be using it and kind of owning the results of it, then you could buy whatever system out there and it could be really flashy and cool. You're gonna get a lot of reports, you'll get some nice dashboards, but you won't make any progress on it because you don't have anybody to own it. Right. So I think, you know, before you really go out and look at any of the systems out there because there's a lot of really good ones. There are a lot of fluff out there, but you've got to figure out how you're going to use it and then or who's going to use it. And if it's not, you know, if you don't have a dedicated person that it's going to be their job, then it's probably something when you evaluate the technology, you want to evaluate the support system and kind of the consulting agreement that goes along with it. ⁓ and I totally agree with Justin that it's something you need to tie to some sort of work orders, but you want to be able to tie I found this I assigned it to, you know, you or I assigned it to building tech and then they fixed it and this was the result rather than I got this really cool thing and it's got AI and I can talk to it and everything else, but I don't actually make any changes. I really like that that piece of advice that both of you hit on there is figure out exactly like you said, figure out your c use case and I I I would maybe add, don't be afraid to ask them to answer it, right? Say this is what I need. Can it do it? Rather than, you know, letting them just go through their whole demo spiel and like you said, show the the pretty graphics. It's like, can it do A, B, and C for me in my situation? So yeah, those those are really good pieces of advice for sure. And ⁓ with that, I think we'll we'll hit up our last question here. Ryan, I'm gonna go off I'm gonna go off the board for one question if I can. A little surprise. Yes, you can. For the Spicy questions and spicy answers are encouraged. I love it. Okay. So here here's something that I've heard before in the context of of Optigo solution, and I wonder if it resonates for you guys and how you address that in the market, which is ⁓ FDD or analytics or anything that is finding problems and allowing the building owner or the their service provider to fix those issues. Are there value in these tools beyond, let's say, the initial one year deployment? Or are we finding everything in that first year and then, you know, selling them those pretty reports years two and onward? I'm just curious to get your guys' feedback. For our certain type of projects, say we have an energy as a service agreement that's twenty or thirty years. Absolutely. We we keep it in there, you know, kind of as an insurance policy almost. And I can tell you we've got one project that's maybe on a year seven, year eight, and then about year four of it, ⁓ maybe year five. Some big BAS upgrade happened and broadcast storm, this or that. It's your fault detection product. Nope. It's this right here. You know, Optigo shows where it started, where it's going. It's coming from the BAS server. I'm not gonna pick on the vendor, but it's a vendor we see a lot that that likes to do is real chatty with BACnet and they know about it. And the conversation went real fast to it's Emphra's fault detection platform to ⁓ it happened on March fifteenth, right at two PM and the guy had just come back from lunch, you know, and it installed another panel. And it's just like, Yep. Yep, there you go. Like I can tell you the minute it happened. And after that happened, I got a lot of buying internally, let's just say, to make sure this, you know, some type of product watching back net analytics state and and was installed on new projects. So that's our experience with it anyway. Yeah, I mean Buildings are not static things. All right. They're they're always changing. Somebody's always plugging something in. They're doing tenant fit outs. They're doing upgrades. And that causes changes everywhere, whether it's on the controllers, it's on the network. Something's always changing. And I think that sure you know, you used you may have found a lot of stuff the first year. But that doesn't mean it's not changing year two, year three, year four. It it's a little bit of, you know, your insurance policy, your continuous commissioning agent, your network analyst, you know, kinda all boiled into one and trying to make it easy to find that type of stuff. So I think there is ongoing, you know, value to to all of it. I love that. And I I'm glad you guys mentioned those stories. ⁓ because I'm a greedy salesperson, right? So I'm I'm clearly biased in this conversation. But I think you're right for what it's worth. I think ⁓ there are obviously things that happen in the building, you know, in year two and beyond that if you're not using an FTD platform or a network analytics platform, you've you've missed those things. And depending on how severe those issues become, the building owner will pay for them one way or another. If it's not for the, you know, the FDD product that's lumped into their service agreement, it's gonna be for all the time and material. along the way that gets them fixed the fix, right? So I I thought that was interesting. Because John, we've had I know I know you've you've related stories where we've had conversations with people that have used, say like Optigo VN and they're like, ⁓ it was great, fixed all our problems. I'm good. ⁓ I'm good now. Thanks. Bye. You know? Yeah, I mean in general you're you're sort of talking to I think ⁓ an antiquated way ⁓ that some you know ⁓ some people operate in our industry. ⁓ but you know, it it's conversations like these really I think that are starting to turn the tide on that. There's ⁓ you know, a a beginning, let's say, of an understanding within our market that some level continuous monitoring on the network side and of the performance of the building automation system beyond just what's built into the, you know, the BMS itself are valuable and providing value to the customer, or if not to the customer directly, then the service provider is cleaning value from it. ⁓ I we know of a particular service provider who ⁓ Who has ⁓ a unique, I guess, business model where their service delivery is essentially a fixed fee annually, and they don't get paid any additional time and material if they have to come outside of whatever bank of hours they've contracted or agreed to. So you can imagine in a scenario like that, having products like these that we're talking about are pretty darn valuable. Mm-hmm. And that's that's a perfect ⁓ segue to the final question here. you know. It's easy to talk when you have a large company or large budgets where you have money to burn on these things, but let's just say, you know, you're you're you're somebody running a mid size portfolio, something like that, you know, typical mix, H V A C lighting, access control. And they're thinking about whether or not it's time to jump into, you know, the extra investment in in these analytics software solutions. Would you say now is a good time to start? Wait a few years, stay with your current approach. You know, where do you feel like this technology is earning its keep today? And and where where do you think maybe staying with what you have right now is probably better? ⁓ it depends on the amount of risk, you know. ⁓ electricity's probably not going down, electricity rates, natural gas rates probably aren't going down. ⁓ and you You know, it's it's all about your risk tolerance and then how much capital do you really have? I mean, if you just don't have the capital, I I I heard this when I started in the industry. ⁓ you know, like sometimes people can't j afford to have a building automation system. And really what the guy meant at the time was, well, if you're not gonna install it right and you're not gonna choose a good vendor and you're gonna go low bid and you're not gonna take care of the sensors once they get five, ten years old and recalibrate, maybe you can't afford. you know that system. And I would say the same thing for fault detection. There's probably a point of entry. And if you're going low bid on everything and the sales guy's showing you pretty GUIs, right? And you're not he's not asking you good questions about how does this fit into your culture. You know, can I integrate to your work order system? There and there's probably a lot of companies out there that can help consult and, you know, firms that know the fault detection world stuff and tell you, yeah, you do have the capital for this or no, you don't. Fair. Yeah. Yeah, I think you do have to do a little bit of self reflection. Like I said, my my example was we had a customer that was like, I know my dampers are falling off the wall. I can't do anything about it right now. That's probably not worth the investment in fault detection. Right. ⁓ but the other part is exactly. And you c and you're not doing anything about now anyway. So like that's where you wait a few years, but you've got to kinda get your house in order first. ⁓ You know, and sometimes that's hard to kind of look internally and and recognize that. Well then you've got to change the culture and get into that kind of maintenance mentality before it's worth investing in something like this. ⁓ it's it's almost like can you afford to fix the problems it finds. Correct. Yeah. Yep. Right. And then the other part is is it this allows you to do more with less. You know, you've got you know, we kind of are in a in a ⁓ gap a little bit in the industry where you've got a lot of really experienced people and then a lot of really young people coming into the industry and there's not a whole lot in between. So depending on where you're at with your you know mix of staff and size of portfolio, this could be a good way to help bridge that gap or get some of those early entrants to the career up to speed a lot quicker by giving them better information. I think that kind of depends on you know where your staffing is and How big you are and some of those types of things. Yeah. And Justin talked about risk and all that cost, of course. ⁓ and I think that, you know, those two things need to be factored in heavily. But the other thing to consider is that ⁓ you know, if you're a a a building owner who's happy to get a bill for, you know, however many hours of work for your systems integrator and it's opaque as far as what that work was and what was done. Keep doing what you're doing. You know what I mean? Like that, you're probably good with that. If you want to actually get some insight into, you know, what's wrong with the building, the sensors, the controllers, how actually your, you know, energy bills could go down, how you can improve the speed of your building automation system, if you actually want to start getting some of those insights and visibility rather than just, you know, relying and paying the bill that is opaque each month or each year. ⁓ You know, F D D is a and analytics in general are are are are a good time to get in. You're gonna have that visibility and visibility breeds accountability, is the other thing, right? If you don't have access to these tools, you're not getting some of this reporting, you know, you you you really don't know what's going on. You're you're trusting whoever is doing the work in your building, to be honest with you. When you have access to those tools, now you've got a level of collaboration with your vendors. And a level of visibility and transparency that breeds better work and better outcomes for the building. So those are things to keep in mind as well. And I think that is a perfect place, gentlemen, to end this discussion. ⁓ Justin and Rudy, I want to thank you guys so much for joining us. We'll talk to you again soon. All right. Really appreciate it. Thanks. Thanks. Yeah, you too, Justin. Take care, guys. How handshake was created by Optico Networks. You can find us on YouTube and at optigo.net. Tune in in two weeks for the next episode.