Can You Really Trust AI-Generated Code? - JSJ 699의 전체 녹취록

lot of concern there. 38 00:01:55.560 --> 00:01:57.920 And then the other piece of it is is, okay, 39 00:01:57.920 --> 00:02:00.799 I've got this powerful tool, Am I using it right? 40 00:02:01.400 --> 00:02:04.159 And so let's talk about the code reviews first, since 41 00:02:04.159 --> 00:02:08.280 that's where you're kind of living these days. And I 42 00:02:08.280 --> 00:02:10.639 don't think we've really gotten into the AI code reviews, 43 00:02:10.759 --> 00:02:13.840 to be perfectly honest. I saw that GitHub does some 44 00:02:14.400 --> 00:02:15.840 like you can turn this on for some of your 45 00:02:15.840 --> 00:02:18.919 repos and I haven't even tried it because it's just 46 00:02:18.960 --> 00:02:21.800 like I don't know, I mean, at work, you have 47 00:02:21.840 --> 00:02:24.759 to have a human code review your stuff anyway. 48 00:02:25.479 --> 00:02:26.800 And then on my personal stuff. 49 00:02:26.800 --> 00:02:28.960 It's like, like, I guess I did turn it on 50 00:02:28.960 --> 00:02:33.400 on one project and I was not that impressed. So 51 00:02:33.159 --> 00:02:36.400 so tell me where this fits in and maybe where 52 00:02:36.439 --> 00:02:38.479 I'm not using it to its full potential. 53 00:02:39.000 --> 00:02:42.879 Yeah, these are good pointers. So you touch an interesting point. 54 00:02:43.199 --> 00:02:46.280 They're related and different. So it's suckles almost theim like 55 00:02:46.319 --> 00:02:48.759 first about losing your job, Like you get to gut 56 00:02:48.840 --> 00:02:51.520 that point really quickly. I think like, first of all, 57 00:02:51.560 --> 00:02:53.719 the next few years. I'm not talking about fifteen years. 58 00:02:53.719 --> 00:02:56.960 That's it's hard to predict, especially the future. So let's 59 00:02:57.000 --> 00:02:59.000 focus on five years and we can we can go 60 00:02:59.080 --> 00:03:00.960 to further and out. I do have like my opinion, 61 00:03:01.000 --> 00:03:03.840 strong opinion about about the five next year. Forget about it. 62 00:03:05.039 --> 00:03:05.360 Sorry. 63 00:03:05.560 --> 00:03:08.479 Dario froment Tropic and Sam Altman cleaning. I don't know 64 00:03:08.520 --> 00:03:10.159 half a year ago, one year ago to the twenty 65 00:03:10.240 --> 00:03:13.280 twenty five. You don't need more developers. Ninety percent of 66 00:03:13.280 --> 00:03:16.840 code generated by developers. Yea sweet bench if you know 67 00:03:16.919 --> 00:03:19.840 this benchmark software engineering benchmark going to be ninety nine 68 00:03:19.840 --> 00:03:20.479 by nine of the year. 69 00:03:20.879 --> 00:03:22.240 We're far from all of that. 70 00:03:22.520 --> 00:03:26.560 And now I claim that while they decrease their predictions 71 00:03:26.560 --> 00:03:28.759 for twenty twenty six lower than twenty twenty five. I 72 00:03:28.759 --> 00:03:31.520 think even there they're wrong. So but that's not doesn't 73 00:03:31.560 --> 00:03:34.280 mean that the future is not going to be strong 74 00:03:34.439 --> 00:03:36.599 right now. It all already is, and it's going to 75 00:03:36.599 --> 00:03:39.520 be more and more AI empowered. And yes, at some 76 00:03:39.719 --> 00:03:42.120 point there is going to be a flection point where 77 00:03:42.280 --> 00:03:44.759 we're actually going to see end to end automation of 78 00:03:44.840 --> 00:03:48.639 software certain aspect of software development. But we can talk 79 00:03:48.680 --> 00:03:51.919 about it via a core review specifically about you know, Copata, 80 00:03:51.960 --> 00:03:54.000 et cetera. We are actually good friend, Like we have 81 00:03:54.080 --> 00:03:58.280 multiple clients that we are partnering together. They focus more 82 00:03:58.319 --> 00:04:02.319 on code generation and agentic workflow out of or pushing 83 00:04:02.360 --> 00:04:05.960 like get up issue into a PR much quicker. We're 84 00:04:06.000 --> 00:04:08.680 more focused on how do you note that that you 85 00:04:08.680 --> 00:04:11.759 can trust that line code? And I'll elaborate about that now. 86 00:04:12.039 --> 00:04:15.400 Sometimes there are some you know, feature like think about 87 00:04:15.400 --> 00:04:19.000 the cloud, like cloud have cloud observability tools, but still 88 00:04:19.000 --> 00:04:21.279 you have the data dog off the world. And that's 89 00:04:21.279 --> 00:04:24.079 a difference between like a few features that the cloud 90 00:04:24.120 --> 00:04:27.600 have around observability to a full fledged platform that gives 91 00:04:27.639 --> 00:04:30.040 you the confidence. So I won't go deeper on that 92 00:04:30.079 --> 00:04:33.120 competition because it's actually quite a symbiotic like, the more 93 00:04:33.279 --> 00:04:36.279 people use copilot, the more you need CODO in order 94 00:04:36.639 --> 00:04:39.120 to trust the code, So it's actually good. 95 00:04:39.639 --> 00:04:43.879 But code review specifically, I think like some might. 96 00:04:43.759 --> 00:04:47.079 Think about a synonym for where you deal with software quality, 97 00:04:47.240 --> 00:04:50.000 but actually difference like it is part of it. But 98 00:04:50.079 --> 00:04:53.879 let's think about quote review the purpose. It's meaningful purposes 99 00:04:53.920 --> 00:04:59.319 first like owning the code and learning two different very 100 00:04:59.360 --> 00:05:03.639 close relate buckets, like as a team. First the developer 101 00:05:04.000 --> 00:05:08.680 mostly single handed, like developing a feature or a sub feature. 102 00:05:08.680 --> 00:05:12.120 And then there's that moment where the team take responsibility 103 00:05:12.120 --> 00:05:13.879 if that person is going to be in vacation, who's 104 00:05:13.959 --> 00:05:17.439 going to do root code analysis if something happens right, 105 00:05:17.480 --> 00:05:21.040 So it's a moment where you you like, learn to 106 00:05:21.079 --> 00:05:24.240 each other code, learn the software, and own it together, 107 00:05:24.319 --> 00:05:27.000 even if an AI

generated it completely one hundred percent 108 00:05:27.000 --> 00:05:29.720 from one prompt to a PR which is rare. But 109 00:05:29.800 --> 00:05:32.360 let's assume we're going to maybe talk about aspector and development, 110 00:05:32.360 --> 00:05:35.519 et cetera. But let's say you did you respect. Yeah, 111 00:05:35.560 --> 00:05:37.920 let's say you did write respect completely like perfect man, 112 00:05:37.959 --> 00:05:40.720 and you've got a feature. Even then a human needs 113 00:05:40.759 --> 00:05:42.639 to take over at least for the next few years 114 00:05:42.879 --> 00:05:45.920 and take responsibility. So that's one one thing about Court 115 00:05:45.920 --> 00:05:49.319 of View, and the second thing is that like until now, 116 00:05:49.360 --> 00:05:50.720 and it's going to be in the next few years, 117 00:05:50.879 --> 00:05:53.079 even if you do the best spec and the best 118 00:05:53.120 --> 00:05:57.240 like work, you're still going to have AI that needs sorry, 119 00:05:57.319 --> 00:06:00.000 code that needs to be reviewed. And there's a different 120 00:06:00.279 --> 00:06:03.800 between a tool that is meant to help you review 121 00:06:04.240 --> 00:06:07.040 and startulize quality to the one that is trying to 122 00:06:07.040 --> 00:06:09.639 help you generate code. Same off, how there's a difference 123 00:06:09.639 --> 00:06:13.879 between code of reserveability and thought. So to till the 124 00:06:14.000 --> 00:06:17.319 r hey, don't worry, in my opinion about your job. 125 00:06:17.319 --> 00:06:20.639 I would even recommend people as different than all other I. 126 00:06:20.560 --> 00:06:21.759 Think nine and a half the industry. 127 00:06:22.120 --> 00:06:25.839 Do go learn like you know computer science, but do 128 00:06:26.399 --> 00:06:30.839 like work with AI as one of your main tools 129 00:06:30.920 --> 00:06:34.759 throughout the STLC and code generation, Code of view will 130 00:06:34.839 --> 00:06:38.279 cot analysis, generating your respect everything and you will just 131 00:06:38.399 --> 00:06:42.759 alleviate together with with the profession, and that I think 132 00:06:42.959 --> 00:06:43.959 my point about that. 133 00:06:44.800 --> 00:06:47.160 Yeah, I agree with a lot of what you said there, 134 00:06:47.199 --> 00:06:49.199 and I'm going to back up through some of it, 135 00:06:49.240 --> 00:06:50.800 and then I've got a couple of questions as we 136 00:06:50.879 --> 00:06:52.920 go through. But one is you know, because yeah, you 137 00:06:53.040 --> 00:06:55.240 kind of wove in the am I going to lose 138 00:06:55.240 --> 00:06:55.759 my job? 139 00:06:56.000 --> 00:06:57.920 Or where where do I go with my job? Along 140 00:06:57.959 --> 00:06:59.079 with hey, where. 141 00:06:58.879 --> 00:07:02.360 Does the code you stuff fit in? And and that's 142 00:07:02.360 --> 00:07:04.439 where I have more questions. But I just want to 143 00:07:04.439 --> 00:07:06.839 reiterate a couple of things. One

is is that we're 144 00:07:06.879 --> 00:07:10.040 not getting away from people having to be involved in 145 00:07:10.040 --> 00:07:12.920 the process. Is what I heard you say. And so 146 00:07:13.079 --> 00:07:15.800 to a certain degree, yeah, you may have tools that 147 00:07:16.319 --> 00:07:19.240 take on certain part aspects of the job or you know, 148 00:07:19.360 --> 00:07:22.839 do some of the things that are involved in you know, 149 00:07:23.279 --> 00:07:26.839 understanding and figuring out what's going on. But at the 150 00:07:26.920 --> 00:07:29.199 end of the day, yeah, you need a human that's 151 00:07:29.240 --> 00:07:31.560 you know, that's going to take responsibility for this stuff 152 00:07:31.600 --> 00:07:35.279 and and shepherd it through. And if you're looking to 153 00:07:35.439 --> 00:07:38.519 enhance your career, you need you need to understand and 154 00:07:38.600 --> 00:07:42.040 be able to use these tools because they do make 155 00:07:42.079 --> 00:07:45.040 people more productive. And so the adoption of this stuff 156 00:07:45.079 --> 00:07:49.959 is inevitable basically. And so those are the triitions and 157 00:07:50.040 --> 00:07:52.800 the role right and the role of soffer interrupting of 158 00:07:52.879 --> 00:07:55.439 software developer right now in five year is completely different. 159 00:07:55.519 --> 00:07:58.639 I love the word that you shepherd, like like basically 160 00:07:58.839 --> 00:08:01.480 you're going to deal a lot with writing spects, writing 161 00:08:01.560 --> 00:08:06.800 your rules, writing your best practices, following down like navigating 162 00:08:06.839 --> 00:08:10.920 your army of agents, specializing in how to deal with 163 00:08:10.959 --> 00:08:12.639 specific problems that might. 164 00:08:12.639 --> 00:08:16.000 Come around with that. How everything I said is going 165 00:08:16.040 --> 00:08:17.519 to be evolved, et cetera. So it's going to be 166 00:08:17.600 --> 00:08:23.000 completely different, like higher level, more architecture, guardrails stuff. And 167 00:08:23.120 --> 00:08:25.959 it's a process that happened throughout the last twenty thirty years. 168 00:08:26.000 --> 00:08:26.160 Right. 169 00:08:26.199 --> 00:08:30.279 We used to punch cards and write notes and assembly 170 00:08:30.360 --> 00:08:33.360 and etcetera. So it evolved. So it's going to be different. 171 00:08:33.360 --> 00:08:35.399 But there's a lot of things for a human to 172 00:08:35.799 --> 00:08:38.000 take responsibility and ownership. 173 00:08:38.799 --> 00:08:40.919 Yeah, and wherever we end up, right, because I think 174 00:08:40.960 --> 00:08:44.320 it's optimistic of anybody to say, well, this is really 175 00:08:44.360 --> 00:08:46.519 where I think we're going to be even in two 176 00:08:46.600 --> 00:08:50.000 or three years, because things are moving so quickly, right, 177 00:08:50.080 --> 00:08:52.399 And you pointed out, you know, Sam Altman and some 178 00:08:52.440 --> 00:08:55.279 of these other folks, they thought they knew where this 179 00:08:55.480 --> 00:08:58.320 was going, and they just you know, it's impossible to 180 00:08:58.440 --> 00:09:01.639 really predict it. But if you're on top of what's 181 00:09:01.679 --> 00:09:05.440 going on today, then it's a whole lot easier to 182 00:09:05.759 --> 00:09:10.519 adapt to whatever comes later. And then as far as 183 00:09:11.039 --> 00:09:14.720 the like the AI run code reviews and things like that. 184 00:09:15.279 --> 00:09:18.240 So you talked a little bit about somebody's going to 185 00:09:18.320 --> 00:09:22.919 have to understand and maintain and take ownership of the code, 186 00:09:23.440 --> 00:09:25.679 and a lot of that knowledge has passed through a 187 00:09:25.720 --> 00:09:29.759 code review, right, So are you are you advocating that 188 00:09:29.840 --> 00:09:35.919 you have a human reviewer and an AI reviewer or so? So, yeah, 189 00:09:35.960 --> 00:09:38.600 how does this fit into the life cycle of my 190 00:09:39.039 --> 00:09:40.320 features and things like that. 191 00:09:40.919 --> 00:09:44.360 Yeah, we'll talk about a bit about nostalogy just before, 192 00:09:44.399 --> 00:09:46.240 Like you remember those days, like I think me and 193 00:09:46.279 --> 00:09:49.440 you are like sorry for iman selling old model enough 194 00:09:49.480 --> 00:09:51.440 to remember today's where we actually use books. 195 00:09:53.240 --> 00:09:55.000 Do you remember before. 196 00:09:54.679 --> 00:09:58.200 We went to Google or whatever was other websites, we 197 00:09:58.200 --> 00:10:00.600 were actually using books to learn about things, et cetera. 198 00:10:00.679 --> 00:10:02.480 And I still use books. 199 00:10:03.120 --> 00:10:05.720 That's kind of my maybe I'm going to sound old, 200 00:10:05.759 --> 00:10:09.879 but and mostly yeah, if it's technology, I do spend 201 00:10:09.919 --> 00:10:12.639 a lot of time on the internet kind of picking 202 00:10:12.720 --> 00:10:15.360 up the new things, right because the books get out 203 00:10:15.399 --> 00:10:19.240 of date kind of fast, depending on how quickly technology moves. 204 00:10:19.279 --> 00:10:22.639 But yeah, I prefer picking up the books and just 205 00:10:22.720 --> 00:10:25.759 getting kind of the classic ideas that don't change it. 206 00:10:26.360 --> 00:10:27.200 Everything runs on. 207 00:10:27.360 --> 00:10:29.480 So so we are nostalgic about it. 208 00:10:29.519 --> 00:10:31.679 But if I have to force you, like choose one, 209 00:10:31.960 --> 00:10:33.879 I think I know what you're going to choose. And 210 00:10:33.919 --> 00:10:37.960 think about how much we delegate to Google or other 211 00:10:39.039 --> 00:10:43.720 technologies on the Internet and interanet to that we're not 212 00:10:43.799 --> 00:10:46.480 going to validate right now to the deepest like part 213 00:10:46.519 --> 00:10:49.639 of it. So I think it's right now already like 214 00:10:49.720 -->

00:10:53.320 a very human digital we call it I or not 215 00:10:53.559 --> 00:10:57.879 like technology based like development already as it is today. 216 00:10:58.480 --> 00:11:00.919 And I think like the same thing is going to 217 00:11:00.960 --> 00:11:03.840 happen with court review, where actually if you have a 218 00:11:04.000 --> 00:11:06.759 one hundred and we are seeing customers that have more 219 00:11:06.799 --> 00:11:10.559 than that, like a set of guardrails, rules, standards, and 220 00:11:10.600 --> 00:11:12.840 I can give you an example like having a human 221 00:11:12.879 --> 00:11:15.799 reviewer doing that like all at once in the time 222 00:11:15.879 --> 00:11:18.840 the limited time is actually problematic and if you can 223 00:11:19.240 --> 00:11:22.679 free the time to focus on what you know where 224 00:11:23.120 --> 00:11:25.559 could be the most useful for that courde review, it's 225 00:11:25.600 --> 00:11:29.360 actually going to have better learning, especially if the tool 226 00:11:29.440 --> 00:11:32.759 AI tools that are using cares about learning. So even 227 00:11:32.759 --> 00:11:35.000 if it caught something, it's just like, oh I fix it. 228 00:11:35.000 --> 00:11:37.080 It's rather hey, this is the reasoning, and that's what 229 00:11:37.120 --> 00:11:38.679 we learned from it, and by the way, this is 230 00:11:38.720 --> 00:11:42.279 how we changed our second brain like that. And so 231 00:11:42.399 --> 00:11:44.240 let me give you an example, like there could be 232 00:11:44.279 --> 00:11:46.840 that someone in the company really cares about you know, 233 00:11:48.039 --> 00:11:51.399 eat your flags, and they didn't manage to tell everyone no. 234 00:11:51.519 --> 00:11:55.360 And the five hundred developers seven thousand developer organizations, that's 235 00:11:55.399 --> 00:11:58.879 what you care about. You can configure that in the 236 00:11:58.960 --> 00:12:01.440 system and I system. And now if you developed a 237 00:12:01.480 --> 00:12:03.919 new feature, you get an alert, Hey you didn't put 238 00:12:03.960 --> 00:12:07.360 a feature flag. Oh show you shouldn't you know? It's 239 00:12:08.360 --> 00:12:11.320 this is also learning, like it's like, hey, oh, the 240 00:12:11.320 --> 00:12:14.120 company cares about feature flag. I have a one hundred 241 00:12:14.120 --> 00:12:17.200 example like this, but just one, you know, very I 242 00:12:17.200 --> 00:12:21.480 think famous, funny or scary or not funny. Min I'm 243 00:12:21.480 --> 00:12:23.519 a developer. This is like I'm a developers. I wake 244 00:12:23.600 --> 00:12:25.559 up in the morning, I have a slot forty minutes 245 00:12:25.919 --> 00:12:29.360 to review a PR, a pull request. If you give 246 00:12:29.399 --> 00:12:31.919 me five lines of code in that PR, I'll give 247 00:12:31.919 --> 00:12:35.039 you fifty comments. If you give me fifty lines, I'll 248 00:12:35.039 --> 00:12:37.639 give you five comments. If it's five hundred lines, looks 249 00:12:37.679 --> 00:12:41.919 good to me? Right, Like, in forty minutes, how can 250 00:12:41.960 --> 00:12:45.039 I five on lines of Code's that's a reality. And 251 00:12:45.039 --> 00:12:47.519 by the way AI could even help us with classifying 252 00:12:47.600 --> 00:12:49.440 like which review which pr? 253 00:12:49.480 --> 00:12:51.320 You can actually do it for forty minutes. 254 00:12:51.679 --> 00:12:53.759 That's how Also doctor works like they get an AI 255 00:12:53.840 --> 00:12:57.519 tool that helps them prioritize, not necessarily tell them if 256 00:12:57.559 --> 00:13:00.600 that person is whatever diagnostic, but just serve facing. Right, 257 00:13:00.679 --> 00:13:03.600 So the opportunities are awesome. Actually, the problem is real 258 00:13:04.279 --> 00:13:08.960 already existing around learning and catching issues. Tell me if 259 00:13:08.960 --> 00:13:11.200 you didn't have a major issue, which developer did not 260 00:13:11.279 --> 00:13:12.080 have a major issue? 261 00:13:12.080 --> 00:13:12.559 And production? 262 00:13:12.919 --> 00:13:15.440 I can tell a few horror stories myself, and so 263 00:13:15.480 --> 00:13:19.000 the opportunity is not reducing the learning or increasing bugs, 264 00:13:19.080 --> 00:13:22.440 is actually trying to to you know, get this better. 265 00:13:22.799 --> 00:13:25.120 But we do need to write you xui and the 266 00:13:25.200 --> 00:13:28.200 right mindset of the team. That's the building the dev tool. 267 00:13:28.320 --> 00:13:29.279 So obviously like. 268 00:13:29.360 --> 00:13:33.000 Sorry, shameless plug bug code, but it's not only us, okay, 269 00:13:33.039 --> 00:13:34.440 like or not going to be the only one who 270 00:13:34.480 --> 00:13:35.080 cares about it. 271 00:13:35.559 --> 00:13:37.960 Yeah, I think I think the how do I put it? 272 00:13:38.039 --> 00:13:38.159 So? 273 00:13:39.000 --> 00:13:41.600 Programmatic checking of your code has been around for a 274 00:13:41.639 --> 00:13:44.600 long time, right, So you have the static analysis tools, 275 00:13:44.600 --> 00:13:47.879 You've got the linters and stuff like that. The electronic 276 00:13:48.000 --> 00:13:51.840 with those is that typically at least the ones that 277 00:13:51.879 --> 00:13:55.320 I've seen they break your code down with the abstract 278 00:13:55.360 --> 00:13:58.000 syntax tree and then they look for patterns. 279 00:13:58.240 --> 00:13:59.440 Is basically what they do. 280 00:13:59.480 --> 00:14:04.440 And the AI systems they kind of do that. But yeah, 281 00:14:04.480 --> 00:14:07.480 you can be a lot more I guess, a lot 282 00:14:07.559 --> 00:14:10.120 less prescriptive as far as what the patterns are, right, 283 00:14:10.440 --> 00:14:13.120 and so you can train it on how to look 284 00:14:13.120 --> 00:14:16.440 at code and what to look for, but you can 285 00:14:16.480 --> 00:14:19.200 be a lot more broad and it'll figure out how 286 00:14:19.240 --> 00:14:21.080 to do

some of the stuff that you have to 287 00:14:21.159 --> 00:14:24.240 explicitly tell the other the static analysis tools and the 288 00:14:24.399 --> 00:14:27.279 linters how to do right. And so you can say, 289 00:14:26.960 --> 00:14:31.000 like your feature flag example, right, you know it can 290 00:14:31.320 --> 00:14:33.200 you know, I mean, depending on the training of the 291 00:14:33.240 --> 00:14:35.720 system and you know how good the data set is 292 00:14:35.759 --> 00:14:37.879 and things like that, and how good your prompt is. 293 00:14:37.919 --> 00:14:39.759 But you can teach it. 294 00:14:39.879 --> 00:14:43.799 Essentially, this is how you identify a new feature and right, 295 00:14:43.919 --> 00:14:45.240 and then you have to make sure that there's a 296 00:14:45.240 --> 00:14:48.039 feature flag around it. And it can figure out the 297 00:14:48.080 --> 00:14:51.120 other steps, like it can infer what to do and 298 00:14:51.159 --> 00:14:53.320 so on a lot of those things where you're saying 299 00:14:53.360 --> 00:14:56.320 you set up a rule. Effectively, what you're saying is 300 00:14:56.320 --> 00:15:00.559 is my code has to conform to these ideas, and 301 00:15:00.639 --> 00:15:04.200 then it can with its latent space, you know, the 302 00:15:04.200 --> 00:15:06.240 stuff that it's been trained on, anything else that you 303 00:15:06.360 --> 00:15:08.960 add to the mix, with its context and things like that, 304 00:15:09.200 --> 00:15:12.639 it can then go in and intelligently figure a lot 305 00:15:12.639 --> 00:15:14.840 of that stuff out where trying to figure out how 306 00:15:14.840 --> 00:15:18.039 to explicitly program it to say, here's how you find 307 00:15:18.120 --> 00:15:21.559 something like this in the abstract syntax tree, or here's 308 00:15:21.600 --> 00:15:25.240 how you break down this idea. Especially on you mentioned 309 00:15:25.279 --> 00:15:28.720 like the five hundred line pr right, it's like across 310 00:15:28.840 --> 00:15:32.440 all of these files and all of these changes. That's tricky. 311 00:15:32.639 --> 00:15:35.799 But the LLM can consume it and figure it out 312 00:15:36.279 --> 00:15:39.240 much much easier than you can figure out how to 313 00:15:39.320 --> 00:15:42.039 explicitly program it for every case you're going to run into. 314 00:15:42.519 --> 00:15:44.720 And it's not going to be perfect, but it can 315 00:15:44.720 --> 00:15:46.399 do a lot better job, and it can do it 316 00:15:46.440 --> 00:15:48.879 a lot more quickly and a lot more thoroughly. 317 00:15:49.120 --> 00:15:49.639 Than I can. 318 00:15:49.759 --> 00:15:52.440 Yeah, just kind of browsing through it and going, you know, 319 00:15:52.720 --> 00:15:55.840 hoping my pattern matching brain goes, yeah, that's a problem, 320 00:15:55.960 --> 00:15:59.279 and that's also a problem. And hey, there's so much 321 00:15:59.360 --> 00:16:01.919 context in here that I'm just trying to understand that. 322 00:16:02.120 --> 00:16:04.080 You know, I'm going to pick out all these little things. 323 00:16:04.279 --> 00:16:08.440 But then when I go in as an experienced programmer 324 00:16:08.480 --> 00:16:11.200 to do the code review, you know, I can see 325 00:16:11.200 --> 00:16:14.399 what it caught and I can say, yeah, ninety nine 326 00:16:14.480 --> 00:16:16.759 percent of this is great. You don't have to worry 327 00:16:16.759 --> 00:16:19.360 about these couple of things that you know, it may 328 00:16:19.399 --> 00:16:21.919 not be quite right on that or it's close, but 329 00:16:21.960 --> 00:16:24.159 you should do this instead of that. And then the 330 00:16:24.200 --> 00:16:25.960 other thing that I can see with it is that 331 00:16:27.320 --> 00:16:29.840 there may be things that are just kind of aesthetic 332 00:16:30.200 --> 00:16:36.200 or organizational or other things that we do that we 333 00:16:36.279 --> 00:16:39.440 haven't codified into the rules where it's like, no, we 334 00:16:39.799 --> 00:16:43.639 do things this way, not that way. And then you 335 00:16:43.679 --> 00:16:46.360 can also go back and you can retrain your you know, 336 00:16:46.399 --> 00:16:48.399 you can rewrite your prompt to include those on the 337 00:16:48.440 --> 00:16:51.440 next one. But yeah, I can see where the LM's 338 00:16:51.559 --> 00:16:54.799 looking at the code could do a much broader and 339 00:16:55.600 --> 00:16:58.360 more nuanced analysis than you can get out of some 340 00:16:58.399 --> 00:17:01.120 of these other tools and be more thorough than a 341 00:17:01.159 --> 00:17:02.039 programmer doing it. 342 00:17:02.679 --> 00:17:06.599 I'd love to relate to the you know, traditional quote 343 00:17:06.640 --> 00:17:11.359 unquote old world satical analysis versus the semantic AI. But 344 00:17:11.519 --> 00:17:15.400 just just before that, about philosophy around like, how would 345 00:17:15.400 --> 00:17:19.039 you how would you actually exploit l lams like AI 346 00:17:19.640 --> 00:17:24.759 to work with rules or standard standards that are written down. 347 00:17:25.839 --> 00:17:28.440 So it's for those who are watching the video I'm 348 00:17:28.480 --> 00:17:32.319 actually having I have here like a black shirt that 349 00:17:32.319 --> 00:17:35.200 it's really where you really I wear purple. And I'll 350 00:17:35.200 --> 00:17:39.640 explain why, because I think the world's roughly sticking divide 351 00:17:39.680 --> 00:17:41.839 into too and I'm relating to how you use rules 352 00:17:42.279 --> 00:17:44.400 in the standard and mark down, et cetera. I'm going 353 00:17:44.440 --> 00:17:48.119 to explain it's connected, I swear. So there's the blue 354 00:17:48.160 --> 00:17:52.000 team and the Red team. Like principle at CODO, we 355 00:17:52.440 --> 00:17:55.119 see yourself like more in the red with mixed. That's 356 00:17:55.160 --> 00:17:57.920 one of the reasons we chose the purple. So I'll 357 00:17:57.920 --> 00:18:01.559 explain the

blue team, which we had like the Winter 358 00:18:01.640 --> 00:18:04.519 of the Cursor of the World, et cetera. Right QUOD 359 00:18:04.599 --> 00:18:08.519 code compile everyone Basically they take those rules you can 360 00:18:08.559 --> 00:18:11.200 write them in different and they mostly put it as 361 00:18:11.240 --> 00:18:15.119 part of their context, which in many cases is actually 362 00:18:15.200 --> 00:18:15.759 listening to. 363 00:18:16.000 --> 00:18:18.799 But there's two problems. One I said many cases. 364 00:18:19.319 --> 00:18:22.920 The second is that maybe it took it as part 365 00:18:22.920 --> 00:18:25.640 of the context. But as you usually it doesn't work. 366 00:18:26.240 --> 00:18:28.160 You don't finish a feature like at least not a 367 00:18:28.160 --> 00:18:31.440 meaningful feature from promptu code, even if it's an agent 368 00:18:31.480 --> 00:18:33.839 that runs for ten minutes usually or one hour, and 369 00:18:33.960 --> 00:18:40.039 how sometimes you usually like get to prompt it over like. 370 00:18:38.640 --> 00:18:39.480 Like navigate it. 371 00:18:39.880 --> 00:18:44.480 Then at your tenth you know, prompt it actually very 372 00:18:44.920 --> 00:18:49.880 very probably like missed something from your spec or or 373 00:18:50.400 --> 00:18:53.359 your rules or your even your own prompt. And the 374 00:18:53.440 --> 00:18:56.680 second one that's a blue way of thinking, would they 375 00:18:57.160 --> 00:19:03.640 prioritized like their KPI is like u x ui. We 376 00:19:03.680 --> 00:19:07.240 admire on them on that and speed from prompt to code, 377 00:19:07.240 --> 00:19:10.839 et cetera. The red team is how we take every 378 00:19:11.039 --> 00:19:15.160 fact that is there as an intent they're functional or 379 00:19:15.160 --> 00:19:18.440 non functional, and verify it. That's a totally different process. 380 00:19:18.759 --> 00:19:20.839 So you take the list of your one hundred rules, 381 00:19:20.880 --> 00:19:22.920 whether it's ten or or one thousand by the way, 382 00:19:23.519 --> 00:19:27.240 and you and you check them with and write lam 383 00:19:27.559 --> 00:19:30.119 to turn to really increase. I can't say one hundred percent, 384 00:19:30.200 --> 00:19:32.559 but really ninety nine whatever and free is a chance 385 00:19:32.960 --> 00:19:35.720 that they're actually actually checked. I just want to you 386 00:19:35.759 --> 00:19:40.400 know that it's important to differentiate between different philosophies and 387 00:19:40.440 --> 00:19:43.359 it actually leads to a different uxui, et cetera. And 388 00:19:43.480 --> 00:19:46.200 that relates to the first topic of ASTs. I think 389 00:19:46.240 --> 00:19:49.599 that they're actually very powerful tools like sonar for example. 390 00:19:50.079 --> 00:19:53.079 I would definitely I have like high expectation of that company. 391 00:19:53.119 --> 00:19:55.839 And but yeah, they do a great job within the 392 00:19:55.880 --> 00:19:57.920 set that they're you know, they're capable of doing. 393 00:19:57.960 --> 00:20:01.039 They do two and I think like eventually, if you 394 00:20:01.079 --> 00:20:04.400 want to exploit AI, really explore, move fast with confidence, 395 00:20:04.759 --> 00:20:08.759 you want that mix of you know LLLM AI empowered 396 00:20:09.279 --> 00:20:13.839 code review, code quality with that static and it would 397 00:20:13.880 --> 00:20:16.559 be the best if you can actually mix them together 398 00:20:16.960 --> 00:20:20.200 and you would see integrations like we have integrations with 399 00:20:20.200 --> 00:20:23.440 with those tools because they might catch in many cases 400 00:20:23.480 --> 00:20:25.400 the same thing and you don't want that annoying like 401 00:20:25.519 --> 00:20:29.920 double double reviewing double thing. So but but the basic 402 00:20:30.000 --> 00:20:33.519 of the technology, you want this technology to to to 403 00:20:33.599 --> 00:20:36.640 work together because otherwise you don't have a confidence to 404 00:20:36.759 --> 00:20:40.319 really like, okay, there's one to clock code. In my 405 00:20:40.400 --> 00:20:43.279 experience like you often I wouldn't say daily but weekly, 406 00:20:44.240 --> 00:20:47.720 Like with one prompt, I can can get one thousand 407 00:20:47.759 --> 00:20:50.920 line of code in five minutes being changed. Go review 408 00:20:51.000 --> 00:20:54.359 that right and right and and now and now, like 409 00:20:54.839 --> 00:20:58.960 you want the maximum help of AI also or sorry, 410 00:20:58.960 --> 00:21:02.799 any technology to help you navigate to getting confidence about 411 00:21:02.799 --> 00:21:08.119 that code quality, correctness, maintainability, et cetera. So I have 412 00:21:08.200 --> 00:21:11.480 like a high expectation of these twols working working together, 413 00:21:12.160 --> 00:21:15.319 being that hunter part the red versus the blue. So 414 00:21:15.519 --> 00:21:18.200 you can get a purple as a dev organization. 415 00:21:19.400 --> 00:21:22.440 So I mean going into this, right, and I kind 416 00:21:22.480 --> 00:21:24.599 of I mean, we may change the title when we publish, 417 00:21:24.640 --> 00:21:27.559 but I kind of tongue incheek put the title is 418 00:21:27.599 --> 00:21:30.559 can I trust AI generated code? And so now you're 419 00:21:30.599 --> 00:21:33.039 talking about this other end of things right where it's 420 00:21:33.079 --> 00:21:37.079 like you've got the copilots or cloud codes or Google 421 00:21:37.119 --> 00:21:39.440 Gemini or chat GPT. 422 00:21:39.400 --> 00:21:42.519 Or whatever, right or whatever? 423 00:21:42.599 --> 00:21:46.000 Right, Yeah, and so you know they're using these models 424 00:21:46.039 --> 00:21:49.039 to generate code, right and then yeah Kiro we did 425 00:21:49.079 --> 00:21:53.160 an episode with Eric Hanchett where you know, in Kiro 426 00:21:53.839 --> 00:21:58.720 it helps you generate the entire spec and then basically 427 00:21:58.799

--> 00:22:01.119 it kind of iterates through step by step and does it. 428 00:22:01.119 --> 00:22:03.119 It was funny because we did that episode. I'm sitting 429 00:22:03.119 --> 00:22:06.400 there going, well, I've been using Cursor and vs Code 430 00:22:06.400 --> 00:22:09.079 with Copilot and neither of them do it, and literally 431 00:22:09.119 --> 00:22:13.519 a week later is showed up in Cursor. And so 432 00:22:13.240 --> 00:22:16.039 so I've used that feature a handful of times and 433 00:22:16.079 --> 00:22:18.480 it's really nice. The other thing that I'm going to 434 00:22:18.519 --> 00:22:20.480 point out because you're like, yeah, you know you have 435 00:22:20.559 --> 00:22:23.200 claud code generate you a thousand lines of code or whatever. 436 00:22:23.319 --> 00:22:26.400 I was having a conversation yesterday with Obi Fernandez. He's 437 00:22:26.799 --> 00:22:30.279 a Ruby developer, but I've seen the same thing with 438 00:22:30.279 --> 00:22:34.240 from JavaScript friends, where you know, they use claud code 439 00:22:34.279 --> 00:22:37.000 and they have it generate a bunch of the code. 440 00:22:37.039 --> 00:22:39.079 Right, and then they kind of review it on their own. 441 00:22:39.640 --> 00:22:43.480 And the thing that's he basically said because he used 442 00:22:43.519 --> 00:22:46.680 to run a consultancy, one of the bigger consultancies out 443 00:22:46.720 --> 00:22:51.279 there in the Ruby space, and he was saying, I 444 00:22:51.400 --> 00:22:53.920 built this app in a month. That it would have 445 00:22:53.960 --> 00:22:57.519 taken us six months and probably five hundred thousand dollars 446 00:22:57.599 --> 00:23:01.920 of developer salary to build this, and I've done it 447 00:23:01.920 --> 00:23:04.960 by myself in a month with AI help. And so 448 00:23:05.200 --> 00:23:07.279 I'm looking at it and go, well, so to a 449 00:23:07.279 --> 00:23:09.279 certain level, I guess the answer to can I trust 450 00:23:09.319 --> 00:23:13.079 AI generated code is yeah, I mean to it because 451 00:23:13.240 --> 00:23:16.359 it works, right, But yeah, so if you have the 452 00:23:16.799 --> 00:23:19.960 AI generating the code, I mean, where do you run 453 00:23:20.039 --> 00:23:24.279 into problems there? And then you know, especially with what 454 00:23:24.319 --> 00:23:26.920 you're seeing, where you have this system that then reviews 455 00:23:26.960 --> 00:23:29.279 the code. So if you have the LLM generate the 456 00:23:29.279 --> 00:23:32.000 code and the LLM review the code, right, is that 457 00:23:32.119 --> 00:23:35.279 kind of a circular dependency that does or doesn't work? 458 00:23:35.319 --> 00:23:38.680 I mean, this is where I'm starting to get into. Okay, 459 00:23:39.319 --> 00:23:41.480 you know, I've got these tools that are supposed to 460 00:23:41.559 --> 00:23:46.279 empower me, but do they actually work nicely together to 461 00:23:46.319 --> 00:23:48.839 give me that red team blue team work out? 462 00:23:49.000 --> 00:23:49.400 Yeah? 463 00:23:49.440 --> 00:23:52.559 And I guess the other concern is is you know, 464 00:23:52.640 --> 00:23:55.519 am I actually moving faster or am I moving faster 465 00:23:55.640 --> 00:23:57.079 with a gun into my foot. 466 00:23:56.920 --> 00:23:57.480 The whole time? 467 00:23:58.799 --> 00:24:00.440 Yeah, we heard a lot last point. 468 00:24:00.519 --> 00:24:03.559 I think, like short, what I'm mostly want to relate 469 00:24:03.599 --> 00:24:06.440 about the main question off the topic of today, like 470 00:24:06.480 --> 00:24:09.839 should you trust AI generated code? I do have an answer, 471 00:24:10.319 --> 00:24:12.599 but I just say, like I do think the future 472 00:24:12.680 --> 00:24:15.000 is already happening now, like pol people of a client 473 00:24:15.240 --> 00:24:18.759 connecting CODO with cursor, CODO with COPAAD or or whatever, 474 00:24:19.319 --> 00:24:20.799 and that's more. 475 00:24:21.519 --> 00:24:23.519 Yeah. I just want to like it, Okay, Yeah, I 476 00:24:23.599 --> 00:24:25.319 just don't want to like to spend too much time 477 00:24:25.319 --> 00:24:25.440 with that. 478 00:24:25.720 --> 00:24:29.319 I guess it's a lot of commercial today's, so it's 479 00:24:29.319 --> 00:24:30.720 possible it's going through that direction. 480 00:24:30.880 --> 00:24:31.359 More and more. 481 00:24:31.720 --> 00:24:34.720 I do want to answer the question, but with a 482 00:24:34.759 --> 00:24:38.240 metaphor before if I ask, do you you do you 483 00:24:38.359 --> 00:24:42.640 trust human code? Wait, way, don't answer, let me give 484 00:24:42.640 --> 00:24:44.839 you give you a second, but just think about a 485 00:24:44.880 --> 00:24:48.480 for a second. Do you trust human the like code? 486 00:24:48.960 --> 00:24:53.440 Think like, let's let's well think about a little bit. Sorry, no, 487 00:24:53.519 --> 00:24:56.720 because because you're you know, it's a it's a good point. 488 00:24:56.799 --> 00:24:59.319 The flip side, though, is that because I don't know 489 00:24:59.359 --> 00:25:01.559 that it's necessar do you trust human generated code? A 490 00:25:01.599 --> 00:25:04.640 lot of it depends on the human right, and you know, 491 00:25:04.720 --> 00:25:06.640 it's like, Okay, how much experience do they have? Have 492 00:25:06.680 --> 00:25:07.559 they done this before? 493 00:25:08.200 --> 00:25:08.440 You know? 494 00:25:09.039 --> 00:25:13.119 Are they taking security and scaling and all the other 495 00:25:13.160 --> 00:25:14.079 things into account? 496 00:25:14.200 --> 00:25:14.400 Right? 497 00:25:14.759 --> 00:25:18.440 But the other thing is is that I don't think 498 00:25:18.480 --> 00:25:20.799 that that's the baseline that we have to look at 499 00:25:20.839 --> 00:25:25.079 AI from because it's not whether or not I trust 500 00:25:25.200 --> 00:25:27.680 human code and whether or not I trust AI code, 501 00:25:27.799 --> 00:25:31.000 because at this point, the

human generated code is the baseline. 502 00:25:31.000 --> 00:25:32.759 We've been doing that for years and years and years 503 00:25:32.799 --> 00:25:33.359 and years. 504 00:25:33.559 --> 00:25:36.480 Is the AI code better or the other way you 505 00:25:36.519 --> 00:25:39.480 look at it from a business standpoint, is is it 506 00:25:39.640 --> 00:25:42.400 close enough? Given what I had to do to generate it, 507 00:25:42.440 --> 00:25:46.440 which is usually a prompt and not nearly as much time. Right, 508 00:25:46.559 --> 00:25:50.200 so maybe it's not as good as human code in 509 00:25:50.240 --> 00:25:55.160 a number of ways, but it's good enough and it's. 510 00:25:55.039 --> 00:25:58.200 A lot cheaper, and so it may be worth it anyway. 511 00:25:59.880 --> 00:26:00.880 I'll use you on it. 512 00:26:00.920 --> 00:26:04.119 But still still bear of me with the metaphor, like 513 00:26:04.160 --> 00:26:07.720 if I ask, and I'll borrow something that you said 514 00:26:07.759 --> 00:26:09.319 to you to make my point. If you ask, like 515 00:26:09.480 --> 00:26:13.160 do you trust human the developed generated code, written code 516 00:26:13.240 --> 00:26:17.720 or coded code, then then I think that the minute 517 00:26:17.759 --> 00:26:19.359 answer is like, yes, that's what we do. But then 518 00:26:19.480 --> 00:26:21.000 you think about it a bit more like you just did, 519 00:26:21.079 --> 00:26:24.519 and you're thinking, if I trust that person, even if 520 00:26:24.519 --> 00:26:27.759 it's a senior developer, to write the code and noteped 521 00:26:27.839 --> 00:26:30.880 plus plus on the airplane and then push it to production. 522 00:26:31.119 --> 00:26:34.039 I'm not saying I never did that, but but but 523 00:26:34.039 --> 00:26:36.519 but push a production. I think your answer is no, 524 00:26:37.319 --> 00:26:40.519 And then you're what you are saying, like a senior 525 00:26:40.559 --> 00:26:44.160 developer that is going through the processes that is required, 526 00:26:44.759 --> 00:26:48.359 uh checking also for security reviewing for the standards and 527 00:26:48.400 --> 00:26:51.279 all that and going through the review process and checking 528 00:26:51.319 --> 00:26:53.599 the CI results and et cetera. You do trust if 529 00:26:53.599 --> 00:26:56.960 you want generated code. That's my answer, Like you should 530 00:26:57.039 --> 00:26:59.799 you trust AI generated code? Spit it out, you write 531 00:26:59.799 --> 00:27:03.119 a you wrote it, prompt you via code, trust it. No, 532 00:27:03.839 --> 00:27:06.440 I suggest you don't also do that. For the most 533 00:27:06.440 --> 00:27:09.640 senior developer, you have a process, the process could be quick, 534 00:27:09.960 --> 00:27:10.720 and I think that's what. 535 00:27:10.759 --> 00:27:11.400 We're going to see. 536 00:27:11.400 --> 00:27:14.200 Also for the AI generated code, you're going to see 537 00:27:14.359 --> 00:27:18.359 AI being used to automate more and more and more parts, 538 00:27:18.759 --> 00:27:21.920 and then AI generated code is going to look like 539 00:27:22.160 --> 00:27:28.559 more more a generated code like AI generated software development, 540 00:27:28.880 --> 00:27:32.440 AI software development, where AI is going to do a 541 00:27:32.640 --> 00:27:37.680 proper process. Humans, by the way, in certain percent, which 542 00:27:37.720 --> 00:27:40.799 in the beginning would be very big, very large, sorry, 543 00:27:41.359 --> 00:27:43.880 are going to be involved throughout the process, and over 544 00:27:43.960 --> 00:27:48.160 time the process is more like you know, like verifying 545 00:27:48.200 --> 00:27:50.240 that the pipeline works, you know, like like we do 546 00:27:50.359 --> 00:27:54.319 with the MANUF in a manufacturing like a lab. Right 547 00:27:54.359 --> 00:27:58.160 like it started with human labor and slowly like more 548 00:27:58.200 --> 00:28:01.319 and more and more automatic humans are still involved there, 549 00:28:01.400 --> 00:28:04.319 et cetera. It took like fifty years to do that. 550 00:28:04.440 --> 00:28:07.720 It will take also like fifteen twenty years. We've talked about. 551 00:28:07.519 --> 00:28:10.599 Predicting for totally being machines. 552 00:28:11.079 --> 00:28:14.079 Meanwhile, like I just invent in a number like fifteen years, 553 00:28:14.119 --> 00:28:16.440 like twenty forty one just to have to guess, like 554 00:28:16.480 --> 00:28:18.400 the four and one looks like AI, so I guess 555 00:28:18.440 --> 00:28:21.079 it's a stayer for me. But until then, like we're 556 00:28:21.119 --> 00:28:23.519 just going to see like more and more portioned being 557 00:28:23.559 --> 00:28:26.640 automated and maybe some portion like automated end to end, 558 00:28:27.079 --> 00:28:29.519 and at that point you will trust it because it's 559 00:28:29.559 --> 00:28:32.640 not just like spitting statistically code and even if it's 560 00:28:32.720 --> 00:28:35.960 trained really really well, but it's also going to validate itself, 561 00:28:36.079 --> 00:28:38.319 verify itself going through the process, et cetera. 562 00:28:38.680 --> 00:28:40.759 And that's a future. That's a future we believe that. 563 00:28:40.960 --> 00:28:43.119 Code of right, that's where we focus on code quality, 564 00:28:43.160 --> 00:28:44.200 codelorification and etc. 565 00:28:44.920 --> 00:28:45.160 Right. 566 00:28:45.200 --> 00:28:48.759 I think the point is well taken that because a 567 00:28:48.759 --> 00:28:54.279 lot of people they conflate where things are with where 568 00:28:54.319 --> 00:28:57.119 we're going to end up. And what you're saying is 569 00:28:57.119 --> 00:28:59.119 is that, yeah, we're going to get better and better 570 00:28:59.160 --> 00:29:02.279 tools to do more more things, and they're going to 571 00:29:02.319 --> 00:29:06.240 manage more and more pieces of the process, right, And 572 00:29:06.240 --> 00:29:09.480 and yeah, I think that that is

absolutely true. It's 573 00:29:09.480 --> 00:29:15.279 funny because for several years, you know, I've been using 574 00:29:15.519 --> 00:29:19.880 rock or chat, GPT, or Claude. I've used all of 575 00:29:19.920 --> 00:29:21.039 them for different things. 576 00:29:21.119 --> 00:29:21.279 Right. 577 00:29:21.319 --> 00:29:24.720 It's like, hey, I'm trying to explore this thing, right, 578 00:29:24.759 --> 00:29:28.160 and so'll it'll give me all kinds of feedback on 579 00:29:28.440 --> 00:29:31.799 you know, health or whatever, right, and so I kind 580 00:29:31.799 --> 00:29:33.640 of use it as a coach or at least, you know, 581 00:29:33.680 --> 00:29:36.720 and then sometimes I'll go fact check it or verify 582 00:29:36.799 --> 00:29:40.440 this or that, or you know, refine whatever it gave me. 583 00:29:40.599 --> 00:29:42.839 But you know, a lot of times it shortcuts a 584 00:29:42.880 --> 00:29:44.640 whole bunch of research that I would have to do, 585 00:29:44.720 --> 00:29:46.640 and so then I can just justify the pieces where 586 00:29:46.680 --> 00:29:49.559 I'm like, it doesn't seem quite right, but it's it's 587 00:29:49.599 --> 00:29:52.279 gotten more and more correct the longer, you know, the 588 00:29:52.359 --> 00:29:55.400 longer we go, because the models get better, the data 589 00:29:55.440 --> 00:29:58.039 that's you know, in that latent space gets better. 590 00:29:58.640 --> 00:30:02.200 And so that's I definitely see with software. 591 00:30:02.920 --> 00:30:05.799 Yeah, But as far as conflating where we are with 592 00:30:05.880 --> 00:30:08.440 where you know, a few years ago, I think it 593 00:30:08.480 --> 00:30:10.640 was just last year actually, I was having a conversation 594 00:30:10.720 --> 00:30:14.119 with my father in law. Now, granted he's a general contractor, 595 00:30:14.359 --> 00:30:17.000 and by general contractor, I mean like he fixes crap 596 00:30:17.039 --> 00:30:20.279 in people's houses general contractor, and you know, and so 597 00:30:20.400 --> 00:30:22.720 he just heard about like the goof ups in the 598 00:30:22.759 --> 00:30:25.240 news where it was like, well, I heard somebody ask 599 00:30:25.359 --> 00:30:28.359 chat GPT this thing and it told him this bogus thing. 600 00:30:29.039 --> 00:30:31.519 And I'm like, yeah, dad, but we've moved like four 601 00:30:31.559 --> 00:30:35.000 models ahead since then and it doesn't do that anymore. 602 00:30:36.000 --> 00:30:38.039 And he's like, yeah, well, you just can't trust it 603 00:30:38.079 --> 00:30:40.319 for anything, and you know, and again I'm looking at 604 00:30:40.400 --> 00:30:43.000 him and going, well, actually, I use it all the 605 00:30:43.079 --> 00:30:47.200 time for this other stuff because you can trust it, right, 606 00:30:47.519 --> 00:30:50.240 But yeah, I go and fact check stuff. It's like, 607 00:30:50.279 --> 00:30:52.319 you know, I think you have a bias here, and 608 00:30:52.359 --> 00:30:54.640 so I'm going to go fact check these pieces because 609 00:30:54.680 --> 00:30:56.960 I think I think the data that you were trained 610 00:30:57.000 --> 00:30:59.960 on isn't one hundred percent in line with my worldview 611 00:31:00.240 --> 00:31:02.319 or the way I think things are. But at the 612 00:31:02.359 --> 00:31:05.440 same time, yeah, it's gotten way, way, way more accurate, 613 00:31:06.079 --> 00:31:08.319 especially when it gets into a lot of the you know, 614 00:31:08.599 --> 00:31:12.359 like easier for like meal planning and you know, hey, 615 00:31:12.359 --> 00:31:14.559 I got to modify this workout or this or that, 616 00:31:14.599 --> 00:31:17.119 and it you know, it's terrific because it's got all 617 00:31:17.119 --> 00:31:19.799 that data in it. And I think that's where we 618 00:31:19.839 --> 00:31:23.880 go with the software. Is My point is so even 619 00:31:23.920 --> 00:31:26.279 where we see it kind of fall short. Is it 620 00:31:26.480 --> 00:31:29.720 a much better and much more accurate, much more thorough 621 00:31:30.039 --> 00:31:33.400 than I can be as a code review reviewer. And 622 00:31:33.480 --> 00:31:36.680 it looks like the answer to that question is undoubtedly. 623 00:31:36.759 --> 00:31:37.000 Yes. 624 00:31:37.680 --> 00:31:40.559 It may give me some false positives and some false negatives, 625 00:31:40.599 --> 00:31:43.079 but it's going to be more thorough and much faster, 626 00:31:43.200 --> 00:31:44.680 and I can pick through that and it's still going 627 00:31:44.720 --> 00:31:46.319 to save me a whole bunch of time and effort, 628 00:31:46.440 --> 00:31:49.240 and it's only going to get better. Right, So where 629 00:31:49.279 --> 00:31:51.799 we end up in a few years it may be 630 00:31:51.839 --> 00:31:54.640 completely different, but it's almost certainly. 631 00:31:54.279 --> 00:31:56.720 Going to have better data to run on and make 632 00:31:56.759 --> 00:31:57.559 the process better. 633 00:31:57.920 --> 00:32:02.680 I think, like I truly believe there is meaningful improvement 634 00:32:02.720 --> 00:32:05.440 in the LLLM. Like some people over time claim that 635 00:32:05.440 --> 00:32:11.400 it's like diminishing the velocity of right because it consumes 636 00:32:11.400 --> 00:32:14.200 what it's been putting out. I've heard that for example, 637 00:32:14.240 --> 00:32:16.680 for that reason, et cetera, I think the are meaningful improvement. 638 00:32:16.720 --> 00:32:19.920 We have internal benchmarks around quality of code, et cetera. 639 00:32:20.240 --> 00:32:22.519 It's it's it's going up. And having said that, I 640 00:32:22.599 --> 00:32:28.240 think there's other reasons where like extracted value is bigger. First, 641 00:32:28.319 --> 00:32:30.240 we use it in more areas the l ll ms 642 00:32:30.240 --> 00:32:32.400 and more areas. But specifically I want to relate to 643 00:32:32.400 --> 00:32:34.480 what you said. I think it's also we're learning how 644 00:32:34.519 --> 00:32:34.960 to

use it. 645 00:32:35.119 --> 00:32:37.119 So you know, like as a developer, you learn how 646 00:32:37.160 --> 00:32:39.319 to Google, or you you learned how. 647 00:32:39.240 --> 00:32:41.599 To use stack overflow, but now I guess, but you 648 00:32:41.720 --> 00:32:43.799 learn how to have a good Google. That's still very 649 00:32:43.920 --> 00:32:46.720 relevant and I think like we are learning how to 650 00:32:47.359 --> 00:32:49.640 you know, prompt or use it could be one of 651 00:32:49.640 --> 00:32:54.119 the differences in between you and the front or family, 652 00:32:54.160 --> 00:32:56.759 et cetera. And that's really actually to for example, what 653 00:32:56.759 --> 00:32:59.319 we talked about w as Hero and Spectra and development 654 00:32:59.440 --> 00:33:02.799 that like we we're learning that the better we the 655 00:33:02.799 --> 00:33:06.079 more information should be concise and accurate. But the more 656 00:33:06.079 --> 00:33:08.559 information we provide as part of the prompt if we're 657 00:33:08.559 --> 00:33:12.400 talking about self development, like maybe spec in most cases, the. 658 00:33:12.400 --> 00:33:13.319 Better job it will do. 659 00:33:13.480 --> 00:33:15.680 By the way, I am like once in a while, 660 00:33:15.720 --> 00:33:17.720 like giving a disclaimer or trying to be careful because 661 00:33:17.759 --> 00:33:20.079 because there's a lot of research down for example and 662 00:33:20.119 --> 00:33:22.119 Tropic are really good in it is that if you 663 00:33:22.240 --> 00:33:25.599 think that you can push like as much as context 664 00:33:25.640 --> 00:33:29.720 and instruction as you want and expected to really work well, 665 00:33:29.759 --> 00:33:32.920 they're actually seeing it diminishing returns even like worse if 666 00:33:32.960 --> 00:33:34.920 you give it like a spec of like a full book, 667 00:33:35.160 --> 00:33:37.680 even if the context is bigger. But putting that aside, 668 00:33:37.799 --> 00:33:41.200 it is a really good idea to. 669 00:33:40.559 --> 00:33:41.920 Learn how to use these tools. 670 00:33:41.960 --> 00:33:45.160 And I think we're actually consciously and consciously like like 671 00:33:45.160 --> 00:33:48.599 like doing doing that and and for software development specifically, 672 00:33:48.599 --> 00:33:51.839 and I think even JavaScript, where you know the language 673 00:33:51.880 --> 00:33:55.200 is maybe not that descriptive, et cetera. Like having a 674 00:33:55.240 --> 00:33:59.599 proper spec is a good idea. And although I have 675 00:33:59.640 --> 00:34:02.079 to say I'm not a big believer that spectrum of 676 00:34:02.119 --> 00:34:04.720 development is going to be the last thing that survived. 677 00:34:04.720 --> 00:34:06.599 It's going to be the biggest thing that actually make 678 00:34:06.680 --> 00:34:09.719 the difference. Is an important concept, but it's not going 679 00:34:09.760 --> 00:34:13.119 to be the what's solved everything. 680 00:34:13.440 --> 00:34:14.480 No, I think. 681 00:34:14.639 --> 00:34:16.599 And I'm just going to piggyback on what you're saying 682 00:34:16.639 --> 00:34:19.800 because I think you're correct in one way how we 683 00:34:19.920 --> 00:34:22.400 use the tool, right, and so you know, spec druven 684 00:34:22.480 --> 00:34:25.119 development is one way that we've you know, this is 685 00:34:25.159 --> 00:34:26.639 a new way to use. 686 00:34:26.559 --> 00:34:30.639 The LM and you know, maybe have a wider. 687 00:34:30.320 --> 00:34:32.400 Context on what it's doing and give it a step 688 00:34:32.400 --> 00:34:35.960 by step cohesive plan. But yeah, I don't think that's 689 00:34:36.000 --> 00:34:37.800 where we end up. I mean, we're going to invent 690 00:34:37.880 --> 00:34:40.480 other ways of using these tools, and this may be 691 00:34:40.519 --> 00:34:43.480 a stepping stone to something else. The other thing, though, 692 00:34:43.559 --> 00:34:46.960 is that it's not just for me. Hey, we're getting 693 00:34:46.960 --> 00:34:50.039 better at using these tools, but also as it takes 694 00:34:50.079 --> 00:34:52.719 things off of our plate, we're able to refine in 695 00:34:52.800 --> 00:34:55.679 other areas, and I think those get better because the 696 00:34:55.719 --> 00:34:58.039 next versions of the models pick up some of those 697 00:34:58.119 --> 00:35:00.880 changes to the way we do things outside of how 698 00:35:00.880 --> 00:35:03.760 we use the LLM and make it better that way too. 699 00:35:04.119 --> 00:35:09.079 And so at some point does it kind of you know, 700 00:35:09.199 --> 00:35:12.360 are we getting smaller increments of value? 701 00:35:12.559 --> 00:35:13.039 Maybe? 702 00:35:13.800 --> 00:35:18.000 But again I I just see the ingenuity of people 703 00:35:18.079 --> 00:35:22.559 as we go continue to just be really cool and awesome, 704 00:35:23.360 --> 00:35:25.360 and so for the time being, we just see these 705 00:35:25.400 --> 00:35:28.400 astronomical leaps every time we get a major version update 706 00:35:28.440 --> 00:35:29.360 on these llms. 707 00:35:30.000 --> 00:35:35.199 Yeah. By the way, I think once upon a time. 708 00:35:36.119 --> 00:35:39.920 My background is the machine learning since two thousand and six, oh, okay, 709 00:35:39.960 --> 00:35:43.880 annual networks in twenty ten, so I allow myself talk 710 00:35:43.880 --> 00:35:48.119 about a history. I think once upon a time until 711 00:35:48.840 --> 00:35:53.760 roughly speaking GPTs three three point five. Every time you 712 00:35:54.800 --> 00:35:57.440 train them all like really like nine nine percent of 713 00:35:57.519 --> 00:36:00.159 the cases. You try it to be better than the 714 00:36:00.199 --> 00:36:03.039 others in a specific niche, right, even if it's a 715 00:36:03.039 --> 00:36:06.599 big niche, still GPT three point five, I think we 716 00:36:06.679 --> 00:36:08.639 had like a year or two or more that we 717 00:36:08.679 --> 00:36:14.880 were under

assumption level matters. You're not the market that wow, Like, 718 00:36:15.079 --> 00:36:18.519 look at this GPT three point five winning every benchmark, 719 00:36:18.519 --> 00:36:21.000 even human benchmark if you remember those graphs, amazing graph 720 00:36:21.000 --> 00:36:24.840 on opening eye exacts as like different professions like from 721 00:36:25.079 --> 00:36:28.719 lawyer stuff to history and the waxes like the percentile 722 00:36:28.840 --> 00:36:33.039 on their official tests, and like GPT three point five 723 00:36:33.599 --> 00:36:37.119 like cross every model on all of these like fifteen 724 00:36:37.159 --> 00:36:40.679 different professions, and then GBT four the same, et cetera. 725 00:36:41.039 --> 00:36:43.679 At this point, it's not the case anymore. I think 726 00:36:43.760 --> 00:36:46.760 since roughly speaking, so that's three point five if you're 727 00:36:46.760 --> 00:36:48.920 familiar from on Tropic, et cetera, then and I think 728 00:36:48.960 --> 00:36:53.440 that model was suddenly better. Some claim much better than 729 00:36:53.840 --> 00:36:57.639 GPTs on coding, but probably not at all. 730 00:36:58.760 --> 00:36:59.760 Quite a few cases. 731 00:37:00.079 --> 00:37:03.400 Now, like there was a moment where seeing people thought 732 00:37:04.119 --> 00:37:06.320 only open air on Tropic or whatever, Google are going 733 00:37:06.400 --> 00:37:10.719 to generate like foundation models, but I'm seeing like dedicated 734 00:37:10.760 --> 00:37:15.079 foundation models and in health medicine, customer success, et cetera. 735 00:37:14.920 --> 00:37:18.239 And and like we do do see like GPT. 736 00:37:17.840 --> 00:37:21.280 Five for example, and maybe there's new new versions the 737 00:37:21.519 --> 00:37:24.519 the are coming that are that are better on specific 738 00:37:24.599 --> 00:37:28.000 aspect of software development, like specific in software develop right, 739 00:37:28.360 --> 00:37:32.400 So so like there is evidence that that is it's incremental, 740 00:37:32.440 --> 00:37:37.800 and I think that's actually somewhat meaning that we're not maturing. 741 00:37:37.920 --> 00:37:40.159 And I'm not sure if that's what people thought. Aren't 742 00:37:40.159 --> 00:37:42.800 going to say, like we're maturing. Okay, let's get to 743 00:37:43.119 --> 00:37:46.119 let's get to those EDU cases. Let's get Okay, we 744 00:37:46.199 --> 00:37:49.599 probably need a specific LLM and a specific agent for 745 00:37:49.679 --> 00:37:53.519 specific quality measure that we want to track or help with, 746 00:37:53.599 --> 00:37:56.239 et cetera. So I think I think like it doesn't like, 747 00:37:56.599 --> 00:37:59.960 it doesn't mean that the whole solution doesn't keep evolving 748 00:38:00.119 --> 00:38:04.159 upwards right in the same or even bigger speed, you know, 749 00:38:04.199 --> 00:38:05.400 like the same. 750 00:38:05.119 --> 00:38:07.960 As uh uh. 751 00:38:07.440 --> 00:38:10.840 And book we talked about book like carts will that 752 00:38:11.440 --> 00:38:13.880 similarity is near. We're seeing like this is a notion 753 00:38:13.960 --> 00:38:17.199 that we're seeing like exponential growth and technology. But if 754 00:38:17.199 --> 00:38:22.199 you're zooming in, then you're seeing like skurve and and 755 00:38:22.239 --> 00:38:25.280 the thing is that each escort the time. The difference 756 00:38:25.320 --> 00:38:28.760 between that each escort just slower and smaller and smaller, right, 757 00:38:28.800 --> 00:38:31.719 and if you zoom out, it looks like exponential. So like, yeah, 758 00:38:31.760 --> 00:38:37.880 maybe that specific GPT sorry, specific LLM attention architecture and 759 00:38:37.920 --> 00:38:41.559 specific training, et cetera, is slightly like the low hanging 760 00:38:41.599 --> 00:38:44.599 fruits are over, but we are we will. We are 761 00:38:44.679 --> 00:38:49.840 seeing in the igentic world and and other like technologies 762 00:38:50.239 --> 00:38:52.920 more breakthroughs and I can mention more and and overall 763 00:38:53.280 --> 00:38:56.480 like the we're going to see AI like keep keep 764 00:38:56.519 --> 00:39:00.519 going upwards and so so I wouldn't like, like say, 765 00:39:00.880 --> 00:39:04.760 you know, like incremental LLM the solution we're getting, it's 766 00:39:04.800 --> 00:39:07.159 going to get better and better and we should adopt it. 767 00:39:07.719 --> 00:39:10.280 Yeah. All right, Well we're getting toward the end of 768 00:39:10.280 --> 00:39:11.920 our time. I hate to cut this short because I 769 00:39:11.920 --> 00:39:17.239 could sit here and talk about this forever. But yeah, 770 00:39:17.320 --> 00:39:20.159 so we're gonna just roll into our picks. I've got 771 00:39:20.199 --> 00:39:24.000 like five minutes before my work, Yeah, my work anyway, 772 00:39:24.800 --> 00:39:26.719 so I'm going to jump in. I have a work meeting, 773 00:39:27.440 --> 00:39:29.639 so I'm going to jump in and move to picks. 774 00:39:29.679 --> 00:39:29.840 Now. 775 00:39:29.880 --> 00:39:32.000 Picks are just shout outs about whatever it is that 776 00:39:32.039 --> 00:39:35.400 we've been up to and enjoying lately. So the first 777 00:39:35.559 --> 00:39:38.440 pick that I have is, so on Friday, I'm going 778 00:39:38.480 --> 00:39:42.760 to be teaching board games. I do this periodically. Hang on, 779 00:39:43.639 --> 00:39:48.880 So I yeah, I'm teaching board games at a board 780 00:39:48.960 --> 00:39:52.559 game conference, and I've picked most of the games we're teaching. 781 00:39:53.159 --> 00:39:56.480 The one game I haven't picked is well, there are 782 00:39:56.480 --> 00:39:58.239 two of them. One of them I'm learning tonight and 783 00:39:58.280 --> 00:40:01.719 the other one earned last week. And this one's called 784 00:40:01.800 --> 00:40:05.800 far Away. It has a board game weight on board 785 00:40:05.840 --> 00:40:08.719 game Geek of one point nine to one, which means 786

00:40:08.760 --> 00:40:14.599 that it's fairly approachable for the average board game player. 787 00:40:14.719 --> 00:40:17.519 And so what it is is you how do I explain? 788 00:40:17.559 --> 00:40:21.760 It's it's mostly cards. So you're playing cards in front 789 00:40:21.800 --> 00:40:24.199 of you, and you have like eight slots, and so 790 00:40:24.320 --> 00:40:26.440 you play your first slot and then your second slot, 791 00:40:26.440 --> 00:40:28.960 and then your third slot. But when you score it, 792 00:40:29.000 --> 00:40:32.320 you score it back the other way. And so the 793 00:40:32.400 --> 00:40:34.360 last card you put down is the first one you 794 00:40:34.440 --> 00:40:39.480 played to score, and it's it's available for scoring on 795 00:40:39.559 --> 00:40:43.000 all the other cards that you played, and then you 796 00:40:43.039 --> 00:40:45.280 flip over the next to last card that you played 797 00:40:45.760 --> 00:40:48.199 and you score it against the two cards that you 798 00:40:48.239 --> 00:40:48.719 have down. 799 00:40:49.599 --> 00:40:51.880 If you play cards. So let's say you play. 800 00:40:51.760 --> 00:40:56.079 The the twelve, and then you play the fifteen after 801 00:40:56.119 --> 00:40:59.280 the twelve, then you also get some other cards. I 802 00:40:59.280 --> 00:41:02.119 can't remember what they're called, but those ones count through 803 00:41:02.159 --> 00:41:05.280 the whole scoring process and anyway, so you just kind 804 00:41:05.280 --> 00:41:07.360 of build up this deck and then you score it 805 00:41:07.679 --> 00:41:10.679 back up the other way it is. It was really fun. 806 00:41:10.800 --> 00:41:12.280 I think it took us like a half hour. There 807 00:41:12.280 --> 00:41:14.320 were four of us playing. It says that you can 808 00:41:14.320 --> 00:41:18.920 play it with ten ages ten plus. You probably can 809 00:41:19.159 --> 00:41:24.840 if you're going to play competitively, as far as like, hey, 810 00:41:24.840 --> 00:41:26.719 you know, I'm stacking all these cards up so that 811 00:41:26.760 --> 00:41:30.159 all the resources on the earlier cards play nicely on 812 00:41:30.199 --> 00:41:33.360 the later cards. A ten year old might struggle with, 813 00:41:33.760 --> 00:41:36.599 you know, planning ahead that far figuring out what to do, 814 00:41:37.159 --> 00:41:40.320 but they definitely play the game right, and enough of 815 00:41:40.360 --> 00:41:43.440 it is common sense enough to where they can probably 816 00:41:43.480 --> 00:41:45.599 at least wrangle their way through a lot of it. 817 00:41:45.719 --> 00:41:48.280 So so yeah, so I'm going to pick that it's 818 00:41:48.320 --> 00:41:50.880 called far Away on the board game. 819 00:41:50.719 --> 00:41:53.920 Pick, and then let's see other picks. 820 00:41:54.159 --> 00:41:57.599 So I think I might have I think I might 821 00:41:57.639 --> 00:41:59.159 have picked this last time, but I'm going to just 822 00:41:59.199 --> 00:42:02.159 pick it again there's a movie that came out. It's 823 00:42:02.159 --> 00:42:05.400 called Truth and Treason by Angel Studios. My wife and 824 00:42:05.440 --> 00:42:08.239 I are members of Angel Guild, so we pay every 825 00:42:08.280 --> 00:42:11.280 month to be part of that. We get to vote 826 00:42:11.320 --> 00:42:14.119 on the movies that they make, and it's also part 827 00:42:14.159 --> 00:42:18.760 of our subscription to the Angel app that Angel where 828 00:42:18.880 --> 00:42:20.760 you know, you can watch videos and you can say, 829 00:42:20.800 --> 00:42:22.800 I don't want any of this kind of profanity or 830 00:42:22.800 --> 00:42:25.280 any of this kind of content, right, so it'll cut 831 00:42:25.280 --> 00:42:27.199 all the sex scenes out of your movies and stuff 832 00:42:27.199 --> 00:42:31.239 like that. But so we wind up getting tickets as 833 00:42:31.280 --> 00:42:33.440 part of our Angel Guild membership to all of these 834 00:42:33.440 --> 00:42:35.480 movies when they come out to theater. So this one 835 00:42:35.559 --> 00:42:37.960 is a World War II film. It's the story of 836 00:42:38.239 --> 00:42:43.880 three young men become disaffected with the Nazi regime during 837 00:42:43.960 --> 00:42:48.320 World War Two after their Jewish friend gets disappeared by 838 00:42:48.360 --> 00:42:53.719 the SS and so they start distributing leaflets by putting 839 00:42:53.760 --> 00:42:57.119 them in mailboxes and stuff and on cars and things 840 00:42:57.159 --> 00:43:01.159 around Hamburg and they get caught, and the movies about 841 00:43:01.199 --> 00:43:04.320 them and you know what happened to them, and so anyway, 842 00:43:04.400 --> 00:43:06.800 it was it was really really good. One of the 843 00:43:06.800 --> 00:43:09.039 things I like about these kinds of movies. I mean, 844 00:43:09.119 --> 00:43:11.039 it's it's a sad story, you know, in the end, 845 00:43:11.639 --> 00:43:13.480 you know the way, the way that it all goes 846 00:43:13.519 --> 00:43:16.360 for them. But it's like, look, you know, how how 847 00:43:16.400 --> 00:43:18.079 willing are you to stand up for what's right and 848 00:43:18.119 --> 00:43:18.639 what's true? 849 00:43:19.039 --> 00:43:20.760 And I think in. 850 00:43:20.800 --> 00:43:24.440 Today's world, in certain parts of the world, yeah, you 851 00:43:24.480 --> 00:43:28.360 may be risking your life, you know where I live. Yeah, 852 00:43:28.599 --> 00:43:30.320 I guess they do kill people for that because it 853 00:43:30.440 --> 00:43:34.079 killed Charlie Kirk. But I don't feel like somebody's going 854 00:43:34.159 --> 00:43:37.480 to kill me for standing up. But I've had people 855 00:43:37.559 --> 00:43:40.280 come after, you know, my reputation and things for things 856 00:43:40.280 --> 00:43:43.400 that I've said. But again it's down to how, you know, 857 00:43:43.480 --> 00:43:44.960 are you willing to stand up for truth? 858 00:43:45.679 --> 00:43:46.639 Are you willing to. 859

00:43:48.239 --> 00:43:50.599 You know, do the right thing even if it costs you. 860 00:43:50.679 --> 00:43:52.639 So anyway, it's called truth and treason. I don't know 861 00:43:52.679 --> 00:43:54.280 if it's still in theaters or not. I think it 862 00:43:54.360 --> 00:44:00.559 still is probably until like Thanksgiving, So yeah, definitely worth seeing. Terrific, 863 00:44:00.639 --> 00:44:03.639 terrific film. And then my wife, My wife and I 864 00:44:03.639 --> 00:44:06.840 are still playing Jaws of the Lion, which is one 865 00:44:06.880 --> 00:44:11.480 of the gloom Haven board game setups, so and it's 866 00:44:11.519 --> 00:44:15.440 basically without a Dungeon Master, self directed D and D 867 00:44:15.719 --> 00:44:21.800 kind of game, and so you know, uh, anyway, the 868 00:44:21.840 --> 00:44:24.159 difference being that you're it's not as free form. You 869 00:44:24.199 --> 00:44:26.880 actually have cards to give you your abilities and so 870 00:44:26.960 --> 00:44:30.239 you play the cards. Yeah, anyway, very fun. So I'm 871 00:44:30.239 --> 00:44:32.840 gonna pick that as well tomorrow. 872 00:44:32.880 --> 00:44:34.239 What are yours? Okay? 873 00:44:34.320 --> 00:44:43.320 Like, I just saw the movie Good Fortune, and uh, 874 00:44:43.559 --> 00:44:45.760 the first time I'm seeing a movie in a in 875 00:44:45.760 --> 00:44:48.039 a theater in New York. I just relocated here, and 876 00:44:48.719 --> 00:44:53.159 I felt that there was a somewhat funny mix of uh, 877 00:44:53.320 --> 00:44:59.199 you know, today's a tech tech bro you know thing, 878 00:44:59.519 --> 00:45:03.519 just this cushion related to together with like all the 879 00:45:03.679 --> 00:45:09.360 apps that we're using daily and how how like people influence? 880 00:45:10.480 --> 00:45:14.760 How are these apps that are designed by supposedly tech 881 00:45:14.800 --> 00:45:18.360 bros Our influence day to day? Kenor is like playing 882 00:45:18.360 --> 00:45:21.239 there as like an angel I don't give like ten 883 00:45:21.239 --> 00:45:24.039 out of ten, but coming from the tech industry, et cetera, 884 00:45:24.199 --> 00:45:28.280 I thought like it's an interesting movie also to see 885 00:45:28.360 --> 00:45:30.760 a bit a little bit how like other people like 886 00:45:30.840 --> 00:45:32.719 see our industry, et cetera. 887 00:45:33.159 --> 00:45:35.079 So I definitely like. 888 00:45:38.280 --> 00:45:38.760 Very cool. 889 00:45:39.639 --> 00:45:42.159 All right, Well, one last thing and then I have 890 00:45:42.239 --> 00:45:43.800 to jump off for a work media, I'm already late 891 00:45:43.840 --> 00:45:47.039 for if people want to check in see what you're 892 00:45:47.079 --> 00:45:47.480 working on. 893 00:45:48.000 --> 00:45:50.280 Check out codo. Where do people go for any of 894 00:45:50.320 --> 00:45:50.760 that stuff? 895 00:45:51.280 --> 00:45:54.400 Yeah, totally so first of all qodeo, dot ai. 896 00:45:54.559 --> 00:45:57.440 From there, we have like everything we are at social 897 00:45:57.719 --> 00:46:02.519 obviously as well personally, I a mar underscore. Mar is 898 00:46:02.559 --> 00:46:07.760 my handler at x Twitter, and we have multiple open sources. 899 00:46:07.840 --> 00:46:10.320 We're actually going to contribute some of them to one 900 00:46:10.320 --> 00:46:11.960 of the open source foundations. 901 00:46:11.960 --> 00:46:14.079 Still learning which one is the best one. 902 00:46:14.400 --> 00:46:18.159 So for example, we have like a pull request code 903 00:46:18.159 --> 00:46:19.960 review agent, so you can find it. 904 00:46:19.960 --> 00:46:21.119 It's called pr agent. 905 00:46:21.400 --> 00:46:23.599 It's very different than our main product, by the way, 906 00:46:23.800 --> 00:46:26.239 very very different, but it is part of like our 907 00:46:26.599 --> 00:46:29.840 collaboration with the community. So this is a bunch of 908 00:46:29.960 --> 00:46:33.119 ways to reach out and we love like hearing the 909 00:46:33.119 --> 00:46:37.079 community code reviews subjective quality is subjective, while we do 910 00:46:37.159 --> 00:46:39.199 need to standardize that. People think about it differently, so 911 00:46:39.440 --> 00:46:43.280 hearing everyone, please reach out and anything and we'll be 912 00:46:43.280 --> 00:46:43.639 in touch. 913 00:46:44.039 --> 00:46:46.760 All right, cool, Well, thanks for coming. This was fun. 914 00:46:47.639 --> 00:46:49.079 Yeah, same here. I really loved it. 915 00:46:49.880 --> 00:46:52.280 All right, folks, we'll wrap it here till next time. 916 00:46:52.480 --> 00:46:52.960 Max Out