Building Reliable AI For Industrial IoT And Edge Systems의 전체 녹취록

value, not just to users and users like myself, but but actual value in the enterprise world. 00:05:37.600 --> 00:05:41.439 And you know, one of those challenges has been integrating integration. 00:05:41.839 --> 00:05:45.680 Can IoT devices with AI begin to integrate themselves? 00:05:45.759 --> 00:05:54.560 And how close are we, do you think, which is you know one of your missions, to eliminate all that kind of manual integration, the stitching together type work? 00:05:55.519 --> 00:05:59.360 Yeah, well, well, surprisingly, we can do a lot today. 00:05:59.839 --> 00:06:03.920 I our focus and our research right now is around a lot of that. 00:06:04.000 --> 00:06:10.879 And the normalization, the parsing, and the decoding being automated, since those have been some of the the hardest problems. 00:06:11.759 --> 00:06:22.000 And normalization and parsing these the LLMs with paired with deterministic algorithms have have shown really great progress. 00:06:22.160 --> 00:06:27.120 They've we've pretty much solved that that case. 00:06:31.360 --> 00:06:46.480 As you know, you know, specialized binaries and formats that some people have still take a human hand, uh especially for for uh for validating the the decoding of different uh formats. 00:06:47.120 --> 00:06:50.399 But most of it can be automated as well. 00:06:50.560 --> 00:06:54.560 You can use very small sample data sets to get a lot of information. 00:06:54.800 --> 00:06:56.800 Same thing with the normalization. 00:06:57.120 --> 00:07:18.000 You can take in a small data set or historical data and just look at and understand the patterns of the data to be able to normalize it, to be able to find out, you know, three different desperate temperature type of readings that they do correlate and how they correlate and how they might uh fit into a similar data set. 00:07:18.240 --> 00:07:23.519 And usually much more accurately than we ever could in the past without these tools. 00:07:24.000 --> 00:07:24.720 Brilliant. 00:07:24.879 --> 00:07:31.360 And just to cut one here, be careful of just touching your table or the mic, it's coming through. 00:07:31.519 --> 00:07:33.040 So it didn't, it didn't. 00:07:33.279 --> 00:07:36.560 I'm just worried if we keep going and it'll be hard to edit out. 00:07:36.720 --> 00:07:42.160 So yeah, I use AirPods, so I don't have to worry about that in my hands. 00:07:42.560 --> 00:07:48.560 I haven't been in I haven't been in this room before, and it's like this really nice physical uh wooden table. 00:07:48.800 --> 00:07:49.680 I keep touching. 00:07:49.920 --> 00:07:52.160 Yeah, I'll back I'll back off from the table. 00:07:52.240 --> 00:07:52.959 That'll be a good idea. 00:07:53.199 --> 00:07:53.759 Okay, okay. 00:07:53.920 --> 00:07:56.000 So I'll I'll go back so to cut one. 00:07:56.240 --> 00:07:57.439 Next question. 00:07:57.839 --> 00:08:06.959 So everyone seems to be bolting these chatbots onto existing platforms as sort of stage one, but you know, that might

be dangerous, right? 00:08:07.120 --> 00:08:11.279 Particularly when you have hallucinations and real-world scenarios. 00:08:11.519 --> 00:08:14.160 Why is that approach perhaps not the best one? 00:08:14.240 --> 00:08:16.160 And what is your alternative? 00:08:17.199 --> 00:08:27.839 Yeah, so the project that we're working on right now, the focus was to architect around these new tools because the fact that they are still brittle. 00:08:27.920 --> 00:08:33.679 I mean, you're talking you're working with non-deterministic systems, in essence, when you work with LLMs. 00:08:34.559 --> 00:08:42.000 And the hallucinations and the errors may never really be solved, especialized for especially for specialized cases. 00:08:42.799 --> 00:09:08.159 Today, we we've done it, we do extensive benchmarking and testing in-house, very rarely rely on the public benchmarking, which I would advise people not to rely on, because of course, companies that make AI models, especially the the state-of-the-art large foundation models, they know about these tests and they do try to work them out and try to perfect them. 00:09:08.639 --> 00:09:22.399 But in-house, we've seen error rates and hallucinations out of the box within the IoT space and handling data being 60, 70% hallucination and error rate, which is fairly extreme. 00:09:22.559 --> 00:09:35.519 But once, but if you actually architect around the system, around the data with instructions, sometimes with specialized trained models, you you can get that error rate down to almost zero. 00:09:35.679 --> 00:09:55.360 You can you can get almost as good as the traditional expert systems and and yeah, and that that's kind of the goal for everybody, but you know, that a lot of people want to go into this more generalized space, which is useful because generalizing the models does give them more insight. 00:09:55.519 --> 00:10:07.279 They can solve problems that you wouldn't expect, and the nuance of a user interfacing and how they might go towards a problem set is going to be so different from the next user. 00:10:07.440 --> 00:10:18.240 So having that generalized knowledge in many different fields is useful in that way, but it also contributes to this creativity inside the model, this the hallucination rate. 00:10:18.879 --> 00:10:19.679 Yeah, I got it. 00:10:19.759 --> 00:10:27.200 So give us a bit of a practical example of you know AI configuring or operating an IoT system. 00:10:27.360 --> 00:10:30.879 What would that look like in the field in sort of reality? 00:11:16.529 --> 00:11:19.649 So there are multiple layers to that. 00:11:19.889 --> 00:11:32.129 So one, we talked about data normalization, just the onboarding of devices and integrating them greatly with that because of the nuances and and the variety of the different types of data. 00:11:32.210 --> 00:11:44.370 But there's also the interactivity of how to interpret and create uh complex alert systems, visualizations to understand when a problem happens. 00:11:44.689 --> 00:12:02.610 You can utilize AI systems for many different aspects of that to simplify the process, but also to expedite it, to make it much faster to handle as like say fleets of devices are being monitored at one time. 00:12:02.769 --> 00:12:06.769 People can only handle so much and address so many of those alerts. 00:12:06.850 --> 00:12:08.850 And then sometimes they need to evolve. 00:12:09.009 --> 00:12:37.169 I mean, false alarms is some of the biggest problems we've heard from customers that have hundreds to thousands of devices, is that you know, the these false alarm systems take a lot of manual process for somebody to come in and adjust and say, okay, well, this this anomaly detection is is is has been set or manually designed for a system that's drifted or changed over time that now just does not fit. 00:12:37.490 --> 00:12:48.610 So then an AI can go in and monitor that and give suggestions on it and adjust the parameters based on what a human decides as well. 00:12:48.769 --> 00:13:00.929 So keeping a human in a loop is still very important, but it's more of a back and forth and and more of a assistance than full automation at this point. 00:13:01.809 --> 00:13:02.529 Interesting. 00:13:02.689 --> 00:13:09.169 And you also talk about treating physical sensors and and cloud APIs as equivalent data sources. 00:13:09.330 --> 00:13:17.250 Talk about the source of uh of the data in a in a in a distributed IoT world and and how you make sense of all that. 00:13:18.450 --> 00:13:19.090 Yeah. 00:13:20.370 --> 00:13:27.490 So all the data out there can a lot of times be abstracted as a device as well, right? 00:13:28.129 --> 00:13:58.769 So we've we found that a lot of the problem sets out there, like energy efficiency for a building, might require outside weather information and forecasting or historical data through systems like Energy Star and getting, or even the power station or or or supplier that they're using, the energy rates fluctuate extremely quickly nowadays. 00:13:59.009 --> 00:14:16.529 And tracking all of that internal data and making it perform as a device next to the internal metrics of the building, of the you know, the heating and cooling systems and lighting, you can you can solve the problems much more efficiently that way. 00:14:16.769 --> 00:14:23.889 Because a lot of the a lot of the problem sets require a little bit forward thinking, anyways, to do them efficiently. 00:14:24.529 --> 00:14:37.730 And that's that scales with the size of a factory or building, like how much in the future you you need to help forecast this information, and then how much external data is helpful for a solution. 00:14:38.289 --> 00:14:38.769 Interesting. 00:14:39.009 --> 00:14:45.330 So talk about what which you know of these smaller specialized AI models are you running at the edge? 00:14:45.649 --> 00:14:46.450 Can you talk about that? 00:14:46.529 --> 00:14:47.649 Are you building your own? 00:14:47.809 --> 00:14:54.610

And what you know, when does it make sense versus the big frontier you know uh labs models? 00:14:55.809 --> 00:15:11.409 Yeah, I have I have a lot of experience building out small models, especially in in in high security regions, uh that a lot of customers they're afraid of where their data is going. 00:15:11.570 --> 00:15:39.889 And they, although a lot of these large language models, they have you know SOC2 compliant, uh, and they check all the boxes of not you know using or sharing data, they still feel there's that's not a guarantee because there's been too many reports of data leakage, and as as well as so that's the security is one of the biggest concerns that have pointed me down the road of developing custom small models. 00:15:40.049 --> 00:15:44.129 But the other benefit of that is efficiency sometimes. 00:15:44.289 --> 00:16:10.129 There's a scaling problem where the large models only become a are only affordable when you're within a it was like roughly, I think still today, like a hundred users, and then hosting your own starts to starts to make sense because the the cost for the server and operating it ends up ends up being more economical. 00:16:10.450 --> 00:16:14.049 And those models, you can specialize them even more. 00:16:14.129 --> 00:16:17.649 You can train them directly on your problem and data set. 00:16:18.450 --> 00:16:21.490 So you get a lot of benefits out of that. 00:16:22.769 --> 00:16:42.769 We've I've worked with government and large corporations that they they refuse, of course, to use the big model to use cloud that's outside of the outside of the internet, but they usually have an intranet of things of their own, right? 00:16:42.850 --> 00:16:57.490 They they build out large internal data sets to solve these problems, and they've noticed that they get less errors and hallucinations with these small models when they're trained specifically on their data set and their problem. 00:16:57.970 --> 00:17:05.890 And that's that goes for problems, everything from like image recognition to natural language. 00:17:06.610 --> 00:17:07.250 Interesting. 00:17:07.490 --> 00:17:15.330 And you know, lots of focus on the safety and reliability of the again, the frontier language models now for good reason. 00:17:15.410 --> 00:17:19.650 You you know, you're millions of people using them every day. 00:17:19.730 --> 00:17:22.690 But you know, what about AI safety for these smaller models? 00:17:22.850 --> 00:17:32.130 What's your philosophy there regarding autonomy and you know the deterministic engineering of those when safety and reliability are at play? 00:17:33.650 --> 00:17:35.809 So it takes a lot of testing. 00:17:35.970 --> 00:17:46.370 Um and then picking out models that are open weight are are very helpful because then you can see what the data, what data it was trained on. 00:17:46.610 --> 00:17:51.970 So that gives you a little bit of a picture of what you could expect from these models. 00:17:52.130 --> 00:18:03.090 Of

course, a lot of people are afraid of these of the flood of small models coming out of China that might be government funded or or or even US funded ones. 00:18:03.330 --> 00:18:12.050 But the most of the models can run fairly sandboxed easily, so a lot of that fear can be mitigated. 00:18:12.290 --> 00:18:31.010 But but we we have seen issues where using a model, even an open model, has had some consequences where the model will report or say things that were not expected and were not desirable and have to be trained out of the model or corrected. 00:18:31.570 --> 00:18:35.330 So there are concerns there. 00:18:35.730 --> 00:18:54.050 I know that that there are recent articles and things coming out, like even Anthropic just put out that article that they had a fear that that small models could self-replicate and actually go on to all the IoT devices and become swarms and and all right, it's a very large danger. 00:18:54.130 --> 00:18:58.690 But of course, somebody would have to specially design a model for that. 00:18:58.850 --> 00:19:03.810 Those aren't the kind of models that people pick out and train or or work with in our field. 00:19:04.050 --> 00:19:04.290 Yeah. 00:19:04.450 --> 00:19:08.450 And of course, you know, we have botnets and malware designed every day. 00:19:08.610 --> 00:19:11.490 That's part of what the bad guys, the the black ads, do. 00:19:11.650 --> 00:19:11.810 Yeah. 00:19:12.210 --> 00:19:14.530 So that's uh, you know, a known entity. 00:19:14.770 --> 00:19:15.490 So what what else? 00:19:15.570 --> 00:19:16.290 What have we missed? 00:19:16.370 --> 00:19:20.050 What are some of the big next technical challenges that you're facing, maybe that we haven't talked about? 00:19:20.210 --> 00:19:27.090 What should CTOs be thinking about now or or next, looking into the short-term one or two year future? 00:19:29.090 --> 00:19:34.210 So I believe people are moving a little too slow in a lot of this industry. 00:19:34.450 --> 00:19:35.890 It's it's hard to keep up. 00:19:36.130 --> 00:19:42.370 We've we have people that want to adapt this and want to understand it. 00:19:42.530 --> 00:20:02.050 And a lot of people are focused on the problem set, but we've got tons of these of these technologies coming out and changing so rapidly that it's it's it's a little hard for large industries to move, you know, it's the big ship trying to trying to navigate the waters. 00:20:02.210 --> 00:20:20.930 It's and the big issue right now is it's partially security, but these new technologies have opened up uh a way for people to find vulnerabilities and and and hack systems much more efficiently than they ever had before. 00:20:21.170 --> 00:20:26.450 I mean, I get I get bug reports or security reports in mass daily now. 00:20:26.530 --> 00:20:46.290 Like, and this is this is it's uh it's hard to keep up with, but to have a good policy in place and pretty much using the same tools, using automation to help address those rapidly is is key in the industry, particularly with IoT and factories and and large buildings. 00:20:46.530 --> 00:20:52.930 And we every day there's another report of some legacy system hacked. 00:20:53.010 --> 00:20:57.570 And the funny thing is, is legacy now is months instead of you know years. 00:20:57.810 --> 00:21:00.290 It's like yeah, no, it's moving so quickly. 00:21:00.450 --> 00:21:02.930 And I'm here on the nexto.ai website. 00:21:03.010 --> 00:21:05.410 Your your colleagues, I think, in in Japan. 00:21:05.570 --> 00:21:12.290 I see so many great use cases here from the home to the office, you know, intelligent office, shared office environment. 00:21:12.690 --> 00:21:33.650 Gosh, I wish I had this when I was working in an office, you know, person controlling screens and cameras and room availability all through IoT and of course a production line uh fleet management, you know, trucks across the country allowing for for tracking and planning purposes. 00:21:33.890 --> 00:21:38.610 Are there the any use cases, those, for example, that you're really excited about? 00:21:38.770 --> 00:21:42.530 And and how do you serve up uh those customers on the platform today? 00:21:42.690 --> 00:21:45.250 Maybe talk a little bit about the commercial side. 00:21:46.850 --> 00:21:50.290 So, yeah, we we've been working with multiple use cases. 00:21:50.370 --> 00:21:59.810 Uh the ones like you've discussed, and we've even looked into monitoring systems, cellular monitoring, remote environmental monitoring. 00:22:01.730 --> 00:22:05.410 Really, the the field is is so open right now. 00:22:05.810 --> 00:22:11.570 It can be applied from everything, yeah, from a home to a fleet of trucks and factories around the world. 00:22:11.730 --> 00:22:22.130 And where we used to, I've been on projects that were large multimillion dollar IoT kind of setups for integrating multiple factories. 00:22:22.290 --> 00:22:27.010 And those projects used to take years to get to to plan and to implement. 00:22:27.330 --> 00:22:33.090 And now the planning and implementation are down to can be down to weeks. 00:22:33.250 --> 00:22:40.130 Like I said, a lot of businesses don't want to run that fast or don't know how to, but the software has really enabled that. 00:22:40.450 --> 00:22:42.770 And it's it's this back and forth. 00:22:42.850 --> 00:22:47.970 It's like, like I said, keep the human in the loop, but you don't have to train the human for the specialized job. 00:22:48.130 --> 00:23:07.010 It's like the thing that we that I went through and the whole industry went through with the whole low code became out, it came out to be kind of a false promise where people were like, okay, we're gonna make all these complicated things simplified, but just by adding these options and these good interfaces, right? 00:23:07.170 --> 00:23:14.370 But then what happened was they just kept adding layer and layer and layer to so that they could generalize for those problems. 00:23:14.610 --> 00:23:18.930 And then you ended up just needing another type of specialized engineer to solve that problem. 00:23:19.090 --> 00:23:21.330 I mean, it's like something like ServiceNow. 00:23:21.490 --> 00:23:30.050 And it's like there are those are specialized engineers that actually run those systems, but the promise was that it was a general person and even a manager might be able to use it. 00:23:30.210 --> 00:23:35.010 But now we're actually hitting a point in technology where that is possible. 00:23:35.330 --> 00:23:49.250 The natural language systems can parse people's language so well that if you architect the system to direct their questions and their asks, you can build out almost any complicated system. 00:23:49.810 --> 00:23:50.450 Well, I love that. 00:23:50.690 --> 00:23:52.690 And that's almost a mic drop moment. 00:23:52.770 --> 00:23:57.730 With I think looking at your your your slogan here, AI built in, not bolded on. 00:23:57.810 --> 00:23:59.330 I think that says it all. 00:23:59.490 --> 00:24:04.690 Thanks so much for joining and sharing a little bit a peek behind the curtain and congrats on the mission. 00:24:05.010 --> 00:24:05.970 Yeah, thanks, Evan. 00:24:06.690 --> 00:24:08.770 And thanks everyone for listening and watching. 00:24:09.010 --> 00:24:13.170 Check it out at n equto.ai, snekto.ai. 00:24:13.570 --> 00:24:17.970 And also check out our TV show, Tech Impact on Fox Business and Bloomberg once a month. 00:24:18.050 --> 00:24:18.690 Thanks, everyone. 00:24:18.850 --> 00:24:19.650 Thanks, Anthony. 00:24:20.210 --> 00:24:20.850 Yep, thank you. 00:24:21.810 --> 00:24:22.370 Bye bye.