Full Transcript for Technically We Have Code Reviews and the LLM Semantic Layer

thing we try to do is always like provide some sort of link to whatever the content is. So is there like a physics engine that you constantly use to model that stuff or a particular I don't know, interface or tool that you're utilizing or something that you just built up in code directly? Mark (06:09.016) Hmm. So I I guess like I kind of I kind of like built up this like physics engine over time in in in in like Haskell to to play around with it and a visualization engine too. I g I guess like if it's in in the sense that's relevant to my story, like it it it taught me a bunch about programming. And I can I can kind of share. Like I I said this in Haskell, like part of my background is like I'm a functional programming like obsessive. Warren Parad (06:18.268) huh. Okay. Mark (06:38.902) And and like that's that was kind of like the stepping stone for everything else in in my career. Warren Parad (06:44.787) Honestly, Mark (06:47.148) And if if you want like if there's like an underlying principle, I I guess it's like of course I work on other things like besides like functional programming. Like I have to like we I'm a data software company. Like we don't it's not all in Haskell. And like I do sales too. So the the I guess the underlying pr principle is is it's kind of like my it was kind of my stepping stone to doing everything else. Whether that be like learning other things or or like le learning about what I needed to start this company Warren Parad (06:59.779) Yeah. Well Mark (07:16.822) Or yeah. Warren Parad (07:17.467) Yeah. I I think this is something I even may want to include in the actual episode, more than a a pick for the end. I I think it's a really great little piece of anecdote of how you got to where where you are today, more so than just you know, random throwaway thing. So just to give you like maybe some exam more examples that could be helpful, some people pick like a like a particular bike that they have, like they live biking and they bike or hiking shoes or let's see what else have people done recently? A a particular protein bar for like a new company or like a a particular smoothie or shake from a from a shake shop or a particular recipe that you like making if you cook. I don't know if these are giving you any any inspiration in like sort of the non technical direction. Mark (08:08.129) I guess if if you want something a non technical direction, like I guess I started doing a bunch of like burpees for for exercise. Especially I I like travel a bunch to do sales and and stuff like that. I I like exercising, but the problem is when you're up to a bunch of of of stuff, like traveling all over the place, then you fall out of the habit. But like jumping up and down your hotel room is pretty good. Like I'm in my hotel room right now, if it wasn't Warren Parad (08:21.763) Okay. huh. Yeah. Yeah. Warren Parad (08:35.791) Yeah. Yeah. I love that. I love this. I I I no, I I think I think this is great. I I I I think we're gonna have to get Mark's exercise routine travel exercise routine. it's gonna be a whole a whole new product. you can push this training out, it could be a learning. this this is fantastic. for the Mark (09:00.854) Okay. Sure. I I can talk about hotel room exercise. Yeah. Warren Parad (09:05.465) Exercises. Hotel room exercises. I l I love I love this. This is this is this is quite a new direction. Okay. sure. and if you think of anything during the episode, you know, feel free to switch your pick over to to that in instead. sometimes they do do do come up spontaneously. and so second and last thing, at the end of the episode, when I say bye and we close out, please just stay on Mark (09:24.226) Okay. Warren Parad (09:33.147) the recording so the post processing can happen. I'll let you know when you can actually help. Mark (09:37.537) Okay, cool. Warren Parad (09:39.813) And the last thing is y your name and role. I think I have here co-founder and CTO at TextQL and it's Mark Hay. Is that right? Okay. so before we jump in and get this started, any outstanding questions for me? Mark (09:47.404) Yep, that's right. Mark (09:56.03) is like the camera used here at at all? Okay. Cool. Warren Parad (10:00.847) Yeah. So we do release on YouTube, but it's a very small fraction of our audience that actually listens to the to the recording via that mechanism. Mark (10:12.226) no, that that's that's fine. I I just wanted to know whether I should be cognizant of like what's in front of the the camera and and and and stuff. So so it sounds like yeah. Warren Parad (10:20.921) Yeah, you know, if so if if something happens, I'm like we can't we we we have to cut that, like, don't worry, I'll I'll I'll call that out. And if I don't, you know, our our editor will definitely take care of it. Mark (10:30.35) Okay. Yeah, no problem. Warren Parad (10:35.641) Is that it? You're just like, let's do it. Okay. Mark (10:36.268) Yeah, that's it. Mark (10:40.665) Is there anything else I should be asking? Like Warren Parad (10:40.719) The ones got the sh no, I don't think so. honestly it People come from different areas and have different expectations. So it's it's not it's hard for me to say what what will be valuable or not. If you feel like we're doing the recording and you get this like overwhelming urge to just have an answer to a particular question that you're not sure about, feel free to be like, you know, off camera, you know, hey Warren, can we just, you know, I have this question I just want to answer and we'll just cut that part out of the episode. Mark (11:21.62) Okay. Yeah, sounds good. Warren Parad (11:23.845) Okay. Well then we're we're gonna get this show on the road then. let me just finish setting up here. Okay. Warren Parad (11:39.76) Okay. Welcome back to Adventures in DevOps. This week we turn over some of the largest rocks in the ML Tech area as we dive into what take what it takes to run ML classification. Our guest previously led Meta's Text Classification

Infrastructure and is now the CTO and co-founder at TechsQL. Welcome Mark Hay to the show. Mark (12:01.058) Thanks for having me on. Warren Parad (12:03.801) You know, when I think classification, there are aspects like sentiment analysis, like how does a text read and entity recognition, identifying relevant aspects, as well as automatic labeling. And I'm sure there's so many more that I haven't I since I haven't done this in such a long time, even even remember at this point. While everyone is focusing on like LLMs, just predicting the next word, we're stuck over here actually trying to solve the hard problems. Mark (12:28.778) yeah, for for sure. I mean I I I guess like in the sense of, hey, I worked on it at at Meta. fortunately or unfortunately, classification is a lot more primitive than that, just by the definition. The definition was just like is it like one or zero? Like just just like true or false. There's it it's not necessarily linked to any specific technique, like not not linked to to like logistic regression or or like GBDTs or anything like that. The thing I worked on at Meta was trying to take in ev basically every single event that we could on the platform. So that that includes like Instagram posts, Facebook posts, comments on each one, messages sent, even even stuff like likes and friend requests and follow requests. And just like r classifying it as as like should we do something about it, like is it abusive or or not? Abusive meaning like is it selling drugs or is it just generally activity we don't want on the platform or is it even e even like criminal or or or or just spam anything like that. And and and kind of the the the whole deal was like how do you know whether whether something's bad or not and what technique do you use? It depends. And and kind of like being on the classification infrastructure side of it meant that we had to capture all that it depends. So on all of these events we have to run like literally thousands of of of different techniques, whether that be like regex on on like the actual text. looking at like various like language embeddings, like media embeddings, stuff like that. to to actually stuff people might not e expect. Like people I I think when when one says like, hey, finding like spam or bad stuff on on on Facebook, it's all about like content. Like can you can you recognize the image or can you like classify the text as abusive or not? But really for for for for a lot of it. Warren Parad (14:34.661) Mm-hmm. Right. Mark (14:41.986) the vast majority of the value comes from behavioral features. so like features about the graph. Like has this person just like sent out too many friend requests to to like seemingly unfamiliar people in it l for for the last bit? Or is there is there something wrong with like the the rate or or the unfamiliarity of the of the folks they're send they're they're sending it out to such that you can Warren Parad (14:53.307) Mm-hmm. Right. Mark (15:11.358) you can maybe like classify a a drug dealer like without looking at a single piece of content that's actually posted, just by looking at like what is their pattern compared to what is their pattern compared to like a normal person. Like maybe a drug dealer is they get they get a new friend or like five new friends every week. They send like two messages to them and then and then they get two two more messages from the same person like three weeks later, and it's it happened over and over again. Warren Parad (15:18.289) Mm-hmm. Warren Parad (15:24.678) Hmm. Warren Parad (15:28.773) Yeah. Mark (15:40.162) You can think you can kind of see like, okay, people try and like hide their activity i in in the content, but the the patterns of of what's going on underneath all all tend to be very much more easily detectable. Warren Parad (15:54.98) It's it's the whole woman in the red dress coding in the matrix, right? Like you can look at the code on the screen and have no idea what the actual message is or what's being rendered for real inside the matrix, but see the pattern of of that, right? And I feel like that's really interesting what what you're getting at there, which is that it doesn't matter what the content of the message is. You can just look at the the pattern of the content to identify the the type of intuit in individual. So even if you're obscuring your content or using some sort of code in your message. You can't necessarily obscure how you interact with the platform. That's quite interesting. And so one aspect is the accuracy, and another one is the scale. As you mentioned, you're doing this potentially thousands of times per request. How do you improve the accuracy really? Like how did you how did you actually identify what the metadata around the connections or frequency rates that a, you know, hypothetical drug dealer would would take versus someone that's just messaging their their friends and family. Mark (16:58.876) so the the good thing about the good thing about working at at at Facebook is that okay, even even though I I'm even though we try and do everything in an automated manner, there's still like a huge amount of human reviewers checking things for or whether that be just a small sample, or or even like a larger amount for okay, i is this thing like is this thing legitimate? like for w what would a general like pattern be for for like good content or or bad content or a good event or a bad event. And that helps a lot in the ground truth, especially for especially for like various statistical classifiers that need like r retraining. So the interesting thing about like abuse classification is that it's like somewhat adversarial. Like they can kind of catch on to like are you Like what what are they catching, what are we catching in in in their behavior and then try and change it like back and forth. So we need actually a lot of like readjustment, like rethinking of the approach and so on, simply because you're playing against an

opponent, kind of. And and your your opponent's gonna make moves to to try and get around that. of course, like the human is a lot the human people at at Facebook meta are probably a lot better than than than the robots. I mean Warren Parad (18:07.281) Mm-hmm. Warren Parad (18:10.639) Yeah. Warren Parad (18:16.176) Yeah, for sure. Mark (18:24.994) Definitely like pre LLM this was this was the case. and so what we do is like we we just constantly take out like samples to to check, okay, is is this classifier like still reasonable or not, or is it like f completely fall under like thresholds for like false false positives, like accuracy rate and so on. And then if we can like retrain it and try and get back above the threshold, Warren Parad (18:38.363) Mm. Mark (18:54.496) If not, then like go back to the drawing board and see, okay, how do we change our policy mix to actually like better solve the thing? Warren Parad (19:03.035) How do you at scale identify that say a malicious actor on your platform has I correctly identified what the classification is keying off of and changing their behavior? like for instance, ha well, I guess it's two questions. Number one, how do they learn about where the limits are? Is it that they're somehow creating lots of accounts on the fly and then trying different things and seeing which accounts get banned and using that to reason about your sort of back end validation system? Mark (19:34.846) I th I think something like that, along with like trying like trying new content and the and then seeing like what what sticks and and what doesn't. That kind of means that that kind of means that like looking the only really the only real way to to like stay ahead is by having like a person like look at the data, whether that be inaggurate Warren Parad (19:44.689) Mm. Mark (20:04.674) Like based on their intuition or even like specific pieces of content to to check like what's what's going on because Warren Parad (20:05.094) Mm-hmm. Mark (20:12.724) Oftentimes like the the trick is very like cul culturally based, if that makes sense. Without going into specifics, like Warren Parad (20:22.361) go go go into p specifics. Like let let's let's let's get to it. I I think that's incredibly interesting. Mark (20:27.562) Okay, sure. Well, I I it might not be appropriate for this podcast, but the the specific I always go to is people trying to sell penis enlargement pills. That's probably one of the most that's probably the the example I can most clearly remember. and so you can think of it like it it starts out as it it just says that. Like we're we're we're selling pills to make your junk bigger and Warren Parad (20:40.097) Yeah. Warren Parad (20:44.731) Okay. Warren Parad (20:53.999) Yeah, for sure, right? Yeah. Mark (20:57.134) You can imagine that's like pretty easy to to detect. Like you can just detect the the yeah, you can regex or just like OCR some some like text and and and so on. But then Warren Parad (21:00.966) that's a

regex, right? Warren Parad (21:07.075) Right. But because then the text moves into an image, right? Like that's the that's the upgrade. Like first it's first is the text, and then it's like a picture of the text that's it that's in there. and I mean you see this in a lot of like game forums or or community chats, right? Like where they're like, Well, what if we misspell some words where everyone still knows what the what the thing is, but it's spelled in such a way that the regex doesn't catch it? Or like what happens if we put stars between characters? That's definitely all like first level evasion. Mark (21:13.09) Yeah. Mark (21:34.688) Right. So there's there's like, yeah, write down text and then like misspell it, like move into an image. once it seems like we're detecting it in an image, start like putting lines and stuff in the image to to kind of like trick a like a naive text detector and and stuff like that. And then yeah. Warren Parad (21:55.001) Yeah. It's like a it's like a it's like a reverse capture, I feel like. Like you're going the opposite direction, right? It's it's not you're trying Yeah, right? Mark (22:03.266) Right, you're on the side of the robot and and then and like the the person the and and then like the the the abusive actor is is the one like making the capture, trying to to go around with stuff. Warren Parad (22:14.585) Right. Right, right. Mark (22:17.974) And the and and and I guess like the the kind of like highest level is to like go into the symbolic realm. Like just have a picture of of like like two cucumbers. Like the like the left one is smaller than than the right one. And then and like not nothing else other than like like go to this website or or or or like all this number. And then and and that's that that's really hard. And and that's kind of like you have to you have to have some some some like people. Warren Parad (22:25.061) Yeah. Warren Parad (22:31.643) Yeah. Warren Parad (22:36.943) Yeah. Yeah, it's a very It's a human contextual problem, right? Mark (22:46.328) trying to apply their own in like symbolic intuition to try and figure out how to best detect it. but definitely like in generality like to totally unsolved territory without like a lot of cultural expertise. 'Cause we had to do this for for like not just the United States and not just Warren Parad (22:49.241) Right. yeah. Warren Parad (22:58.661) Yeah. Warren Parad (23:02.149) Right. So I am I imagine that there there is this huge I mean there's this interesting aspect here, which is that unlike other adversaries where the the target is sort of infinitely far away, whatever the malicious actor on your platform is actually trying to do, they still have a need for their content to be understandable by their their mark in some way, right? Because so I can imagine your goal is actually to not only identify like you you don't have to identify Mark (23:26.392) Right. Warren Parad (23:33.775) the malicious actors on the platform, if you can make it that their message doesn't result in any any negative impact to the platform. I mean, I feel like the the c having two cucumbers of different sizes, you know, may be just a legitimate meme that could be there as a real picture that someone could find humorous and not actually in the land of problematic content that could lead someone astray or convert them to whatever Mark (23:56.546) Yeah, yeah, that that that can be true. I've never thought of that. Maybe you can get so good that you just like totally exhaust the space of like of symbols that that that one might be offended by. Warren Parad (24:04.653) Right. R right, right, exactly. I I because I mean, like at this point, like do you ban eggplant emoji, right? It it's so perversive pervasive in our in our culture, in in at least in the Western world that I'm aware of, that it seems like it it doesn't do you any good to have concerns over you know, the printing of that. and then there's another aspect of the underst like is it understandable by the the other party? and then so here's here's another aspect. That I've always been sort of curious about. And we can we can pivot pivot off of this in a moment, but real realistically, there is this aspect of a way to deter someone from identifying what the defense mechanisms are is by separating out in time the moment in which you actually perform like have a retribution action against a a spammer or whatever. because humans are pretty bad at long-term feedback loops. When an action is performed at T0 and 10 minutes later something happens, your account gets banned, you probably can guess somewhere in the last 10 minutes, an a violation has occurred. But if you ban them after an hour or a day or a week or a month, it makes it very difficult for them to narrow down what the problem is. So there's like some sort of mean time to like a delay or threshold that you automatically let spammers have so that they're unable to actually understand. the merits of the classification system that's in play. Mark (25:38.026) Mm. Yeah, no, I I I I think that makes sense. And and there I guess there are kind of ways you can you can do it. They're consistent with a rule system, like like having strikes for for example. Like like you have some number of strikes or you have some number of points and it's unclear like how many points you get for what. And also like how many points you get for like various like behavioral like strikes because Warren Parad (25:48.251) Yeah. Right. Warren Parad (25:53.563) Yeah. Warren Parad (26:07.055) Right. Mark (26:07.5) I I think that's not that's not something, at least for an individual actor, that's not something they'll necessarily like think about hard. They'll just think about like content, like content, content. but instead, like if we can detect, okay, like who are they, who's the scene like they're trying to target, is their pattern like just someone interacting with their friends, or is it someone like trying to find

like new like like new marks to scan? that's Warren Parad (26:14.127) Right. Warren Parad (26:17.552) Right. Warren Parad (26:32.697) Mm-hmm. Right. Mark (26:36.214) I think that's a lot harder for peop for people to kind of reverse engineer. And like going back to to like like the cucumber an analogy, like totally sidesteps this like trying to go trying to like level up in in like abstract space because there's like no there there's no like content you can change if if like we're not detecting the content. Warren Parad (26:41.585) Mm-hmm. Warren Parad (26:58.745) Right. Well, there's another aspect here that I I just thought of, which is that if changing the validation or classification mechanism to better identify malicious actors on on the platform, just causes them to change their behavior. It doesn't actually reduce the amount of spam or like unwanted messages that you have to process. So it's almost in some way better off to let those messages be sent, not block the user, and then just block like send them into the abyss. Right. Like it's it's fine if someone performs you know, a violation on a platform if no one is around to witness it, right? And if you don't ban them because of it, they'll never find out that it was a problem. Mark (27:40.16) Yeah, that that's another I I think it's another good one. A good technique for from I'll call it like the actual actually passing on d judgment, which is like throttling or or or shadow banning where we okay, we think someone's like trying to spam a bunch on on Facebook. Well like just let them, but but like kind of kind of like pull back on the reach a a a little bit. Of course like Warren Parad (27:50.843) Mm. Warren Parad (28:06.063) Yeah. Mark (28:09.85) if you get it wrong that th you're kind of like torturing them. in so you definitely don't want to do to to do that. Not that I on Facebook it's not so bad. Like people definitely like complain about it on on on say like YouTube. For I don't I don't know whether it's actually like abuse views or not, but I I I frequently see like people on YouTube say, I didn't I don't have as many views as I did like like a year a year ago. Like is is YouTube punishing me? And again, I don't know if it's 'cause they did something bad with their content or not, or if they just like fell off or the audience fell off or YouTube changed their algorithm in some arbitrary way. But for you can imagine like if if they if they aren't like actually like doing bad stuff, i it it w it wouldn't like feel bad. Like at at like at best, like you feel like your friends hate you now 'cause now they don't interact with your post, whereas they they once did. Warren Parad (28:37.359) Yeah. Yeah. Hmm. Yeah. You're right. Warren Parad (29:06.617) Well and maybe maybe this is the thing where you're better off lying about the engagement mechan like the engagement metrics for your for that particular user. Like don't tell them that they're getting less reach. I mean, it doesn't doesn't

really matter, what's in but I guess you're you're sort of have this hole to deal with, which is that if users' fundamental goal is to get a larger reach, you still want to maybe you want to actually provide them the tools to understand why they're not reaching their actual personal goals or whatever their goal is on the platform. And so you have to deal with this duality. Whereas you want to help people who who are improving the platform, but you don't want to help people who are diminishing value of it. And so now you have this whole issue of how do we sort of lie to the people who we don't want there. So they continue to do things that are easy for us to detect and avoid and and sanitize out and help those that that are out elsewhere. Mark (30:06.624) Another way of looking at it, and I I guess I can say this 'cause I don't work in I don't work at Facebook anymore, even though I still still like the company. I I I guess a good way to a good analogy for what you're what you're saying is like ads. Ads in a really uncharitable way are just like like legal spam. Like spam the like Facebook like Warren Parad (30:24.569) Mm-hmm. I don't know if I say le legal. It is for sure spam. We can maybe leave the the legal part out of it. Yeah. Mark (30:31.49) Or it's like a allowed spam because you pay for it. And and and and what happens when it's like allowed spam that you pay for? Well, f like Facebook like does everything they can to try and like increase your reach for that you're you're like paying you're paying for it. Like they'll they'll I don't have you ever like ran Facebook ads or or anything like that before? Or like any pain? Warren Parad (30:55.27) me personally, no for a couple of reasons. but I don't know if that's worth getting into. Mark (31:02.644) Okay. Well, if if if you do, like you can kind of something interesting that i is once once you like first put out an ad, there's it's kind of like running the algorithm on it. It's like testing it on on like a small subset of of audiences. It's like trying to figure out like who's it gonna reach well well. And then based on your parameters, like minimum spend or like maximum spend, like who are you who are you trying to reach, what goal do you want. Warren Parad (31:12.049) Mm-hmm. Warren Parad (31:22.438) Right. Mark (31:31.714) Like maybe you're selling a thing and you want them to hit the purchase button, or maybe you just want them to look at the thing. it it will like optimize that for you. And and so so that's like the total like flip side of of of like trying to kill spam. It's trying to figure out like what makes your advertisement like seen by like seen more by more people, like interactively more, like clicking the button at the end of that. I didn't work on that part, but it is kind of funny how. Warren Parad (31:35.889) Right, right. Warren Parad (31:41.798) Mm-hmm. Warren Parad (31:47.321) Right. Warren Parad (31:54.001) Right, right, right. Yeah. Mark (32:00.536) They're they're

like two like two sides of of of the same coin, i if you think about it. It's like what content do we what content do people pay for? try and and and do the best we can to like be good business partners and and and boost that as much as we can. And then what kind of stuff do we feel like is like a harm to the platform and is is kind of especially like s people like selling like bad stuff. and then how do we like do the opposite? How do we throttle that? How do we even even like kick those people off? Warren Parad (32:24.165) Yeah. Warren Parad (32:29.712) Yeah. Warren Parad (32:36.015) Yeah. Well, I I think the uncharitable view of this is it'd be totally solved if users could control their their I I you what, I know this is such a ridiculous statement, and we know no platform could ever support this, but just have full control over their feed. and then they would be able to say that they don't want certain things in there and that would be the end of it. like, you know, I only want to see messages from my friends. And if one of them starts posting pictures of cucumbers, I can probably guess that their account was hacked. And I can I can stop following them and that's the end of it. but I I I do appreciate that in order to make money, companies want to poison your feed with content and from that regard you have this problem. you also have the problem of of public groups, right? where you allow anyone to post, and which case you th those groups by nature ha have this problem where malicious actors can come in and and and post by default. Mark (33:26.606) Right. a a really interesting thing we we worked on was like WhatsApp integrity. And I I I I talked a lot about like you have to look beyond the content to make to to make a the best judgment. Well in WhatsApp you have to because WhatsApp content is like totally encrypted. and so y like you can't like read the messages even if you w even if you wanted to. In some circumstances they can like Warren Parad (33:35.003) Mm-hmm. Warren Parad (33:41.233) Mm-hmm. Mark (33:56.204) Someone can else can like report a message and then and then you can see it. But that that's like at that point they they're they did your job well already for you and you're just like confirming it. but for for for WhatsApp, you have to figure out everything based purely on like behavioral features, like who's messaging who, like groups and and so on. and I guess the most obvious thing is I don't know if you have like WhatsApp, but every so often I get added to like Warren Parad (33:58.585) Right. Warren Parad (34:05.295) Yeah. Warren Parad (34:12.891) Right. Mark (34:25.868) Like crypto token like pump group. Like like number number ninety eight. Like I I don't even like trade crypto, but I say add to these groups and then they say like we're gonna b all buy this token and then Yeah, and then you're you're missing out you don't buy as well because you'll be a millionaire if you buy at the same time and then sell when we w when

we tell you to. Warren Parad (34:28.015) Uh-huh. Warren Parad (34:37.829) Yeah. Maybe you're missing out. This is this is your one opportunity. Warren Parad (34:47.383) Only a millionaire? I mean with what the US dollar is at, I feel like that's that's shooting pretty low there. you know, anything anything less than than ten or a hundred million. I I mean what what what's even what's even the point? Mark (35:00.174) I'm I I mean for for for the for for like the shit coins that they try and pump. I I don't even think they get to a cap of of like over a hundred million. Warren Parad (35:06.02) Yeah. Warren Parad (35:09.583) Yeah. Yeah, for sure. I mean and Ugh, I mean that's pretty interesting about the the behavioral the behavioral metadata that's associated with the messages is actually what you're performing the analysis on. so one of the things I wanted to do, especially because of your your pr professional pivot now to being the CTO at a TextQL, I I'm sort of interested in I I just if I had to guess, you've ported some of your learnings and knowledge from working at meta classification over to providing a similar solution to other companies that are doing something. And like I'm sort of curious, like what is the the depth of the technology at TechSQL doing? Mark (35:53.664) so if if you haven't if you haven't heard of of it or or or looked into it, PACGL like kind of simply is is like it's kind of like chat GBT but for querying like data warehouses and databases and big data systems, like doing any of like data engineering, data analysis, data science. And even if you're like non technical and specifically like even able to like work on on like messy data and and stuff like that. Warren Parad (36:10.554) Okay. Mark (36:23.886) So can think of it as and I can go into like why is this a why is this something you want you even because e even over just like Claude by itself. but you I guess you can hopefully you can imagine like the problem or like why someone who doesn't like know SQL or programming might want this capability themselves. like kind of in addition to or in substitute of like a full-time like data. engineering data a analysis team of o of like humans. Warren Parad (36:56.305) Well let's talk about the technical side then potentially. Maybe there's there's some value in it, especially because I think the audience here is much more technical, so it could give them some inspiration into what they're dealing with. I think historically a lot of companies had some sort of business analytics and that was replicated data MS SQL databases if you were lucky or Oracle databases if you were unlucky. And then there were teams who basically wrote SQL on top of that or managed the SQL, the DB admins, and then teams who didn't understand the SQL and used like SQL crystal reports or whatever to expose the data. And I think the natural evolution is, of course, to throw an LLM-based query at it. And it the LM will generate you some SQL to run against

your database. And I think what we realized collectively in the last six years or so is that it's incredibly expensive to figure out how to do the semantic search on your database with all the columns. And whatnot. And it's also super expensive to do to put that data in a rag database or in a vector database in order to perform the embeddings on all of that data that you have for your entire company, as well as perform the embeddings on every single request or query that's coming in and match it to what's in the database. And now I feel like where the world is at, and maybe you can correct me, my my limited understanding is that especi the most of the advan more advanced companies have switched to some sort of semantic layer. Whereas instead of Performing embeddings on the original query and what goes into the database and using embeddings to do the similarity match, you just parse the schema and run that through your embeddings engine basically and then match incoming LLM requests or prompts that users have about the data to specific queries. Like you're you're not dynamically matching in the database. You're dynamically matching a particular query to be able to even interact with the database in the first place. It's a and so you don't have to really, you don't have to parse your data. You don't have to run it through an embeddings model. You don't have to run it through an embeddings model every single time you change your embeddings model, which I think historically has been a huge challenge. And I don't I don't know how much of this you're actually doing, but I I find the space now to be much more Warren Parad (39:03.731) Yeah. Mark (39:05.352) no, I I I think your evaluation of of like where did the industry start out and and where where has it gone is is is totally spot on. Kind of the current the the current like conventional wisdom is to kind of decompose your schema into a bunch of like metrics and dimensions and then kind of construct these rails such that like you can search over them and then any combination of like metrics and dimensions you have. is kind of like correct by construction, uses like the right formula and stuff like that. it's interesting you go there because kind of like our like our approach to Text Ul is trying to get to the next stage of of where that is. And why we need a next stage, because the the the main problem like as I see it with semantic layers is time to value. I mean kind of the same with with with with like rag Warren Parad (39:52.538) Okay. Mark (40:04.32) and and like vector databases to a degree. In order to have it's once you have like this correct by construction layer, then you get this like next level of correctness and assurance that okay the language model isn't gonna go totally off the rails. the kind of downside to that is you have to set it up first. And that can take a long, a long time and a lot of validation. Warren Parad (40:06.255) Mm-hmm. Mark (40:33.26) And also like the kind of the

what happens if someone like mentions a metric that's like a slight variation on one that's in the semantic layer. Well now it can't really it's kind of broken as it can't it it's like stuck on the rails. Warren Parad (40:46.139) Let's let's I I want I want to really dive into that. maybe first get your perspective on how you would define semantic layer because I know my my definition is totally convoluted and not accurate for someone that is not an expert in in the area, and then also understand more about the complexities that you currently see in trying to have the semantic layer work correctly. Mark (41:07.956) so I I I guess like so I mean I guess I'm neglected to go into into this, but like my background is in programming languages. And so my my definition of a semantic layer would be kind of a A configuration of of like of I'll I'll call it like analytical definitions on on top of a database. So that's one part, like your config layer of definitions. And then the second part is kind of a do a domain-specific language, like a very simple programming language language on top of that config, such that like if you can express a query in that, it is like ostensibly correct if you provide that you like did the configuration right. If if that makes Warren Parad (42:00.281) almost like it's almost like you're saying that SQL isn't the best language for querying a a a pool of data. Mark (42:07.306) well it I I I guess like it is and it it is and isn't. so I I'm I I I've been like a semantic layer isn't necessarily good. but but it is good in the sense that like Warren Parad (42:17.393) Mm. Mark (42:23.922) it is good in the sense that like if you have a query and you can write it in the semantic layers DSL, it'll probably be right, or at least like you can be assured of of like its structural correctness to a degree that you can't with like raw SQL. The problem is that you have to define everything you want to refer to in this like configuration layer. And that takes and and that means you have to write the configuration layer. Warren Parad (42:28.049) Mm-hmm. Warren Parad (42:39.738) Okay. Warren Parad (42:48.923) Right. Mark (42:52.972) And then that takes a lot of time. The good thing about SQL is that it's it's like practically like a Turing complete language on on top of like your tables and columns. That means you can express anything, which means you don't have the guardrails, but also like what are LLMs good at? They're good at like looking, they're they're good at like trying many hypotheses, they're good at like at Warren Parad (43:19.057) Mm. Mark (43:20.6) creativity and flexibility and coming up with things you haven't thought about. And so that's like that's kind of one of the other like by having this like rigid configuration layer, you're kind of like killing some of the juice that makes people really like using language models, w which is the ability to take on like any task in a super flexible manner. Now you're kind of straight jacketing it. Warren Parad (43:40.313) Mm. Warren Parad (43:45.03) I I like that I like that flavoring. I mean it sounds like right, if we rely on I so in in a previous episode where we talked about embeddings a little bit, one of the things that w I sort of identified was that when you do a similarity match in a vector database, so you you get in all your data, you run it through an embeddings model, you get out a bunch of binary numbers. I like to think of this as the transformation between like the coordinate system, like X and Y, for packets versus the Fourier transform in the frequency domain. So you're changing domains basically. And then you're doing the similarity match by running the same embedding model on the new query that's coming in to figure out whether or not there's data in your database that matches. Now, with that, the thing that was sort of identified is that the values in similarity is An inherent property of the embedding model that you're utilizing, that how close two words are or their hypothetical meaning is going to be based off of whatever, as you said, special creativeness is in the in the model itself. And so while it's expensive and it may be problematic to manage, especially at scale, switching off of that and moving the flexibility to the SQL layer means you're losing whatever core aspect, whatever. the I hate to say value or soul was in the original embedding model that allowed it to even perform the similarity in the first place. So I I get that. there's also this complexity where it sounds like we were saying that with a semantic layer where we're dynamically generating SQL from im incoming prompts, that the accuracy of the SQL generation is still potentially problematic. I mean you can of course try to improve it using guardrails, et cetera, but at the end of the day you could still end up with a syntactically invalid SQL that you're running against the database. I I think that's what you're saying. Mark (45:40.684) so I I'm I I guess I'm I'm I'm kind of describing like two worlds. Like one world is is like LM like writes any SQL it wants. It's the exact same as like giving it your Snowflake credentials in in like cloud code. And then the second world is like make it like pick the the the things in the can in the semantic layer configuration that it wants, submit that to the semantic layer program, and then get back correct SQL. now that's Warren Parad (45:44.816) Yeah. Warren Parad (45:51.406) huh. Right. Warren Parad (46:09.136) Right. Mark (46:10.562) Now that SQL will be like structurally at least like structurally correct, provide you did the configuration right. the problem is yeah, sure. You could like refer to stuff that like doesn't exist or like totally misinterpret like the user's question. Warren Parad (46:15.483) Maybe. If you're using an L L you could still end up with hallucinations, right? So you Warren Parad (46:24.847) Right. Right right. Right, but I mean that accuracy is a different problem

for sure. Mark (46:31.947) Right. And and and I guess like I'm contrasting these two worlds is like the semantic layer world, like you get structural rigidity, which is bad for for kind of language model creativity, but good for kind of predictability. Exact same as like moving from embeddings to like deterministic text search. Embeddings is like fluid, but you have no idea like how it'll act. Warren Parad (46:51.398) Right. Mark (47:00.33) and and it's super sensitive to like small changes in your pipeline. Whereas like text search is like it it some text search can be complicated, but at the end of the day, you feel a lot more confident and you can kind of like pick out if two things are close together, where exactly in the pipeline of of like your algorithm did they come to be close together? Whereas like semantic or or not embedding search, you're kind of just like throwing up your hands and saying, I think open the eye or whoever, like change a good model and because it's a good model, these two things are are close and so they're they're that's that's probably like actually correct that they should be close. Warren Parad (47:44.294) No, I I get it. So I mean it does seem like where we're going is that even so there's a problem with the current state, right? That we've we've removed some of the you're calling creativity. I don't know if I love that term, but there is something here that may have captured the in intuitiveness that whoever labeled the original data, whoever refined the data sets that were used to train the LLM in the first place, knew subconsciously. And included that in the creation of the model. But as we shift layers up, we we lose some of that understanding fundamentally of how these two things are related. And like we were talking about the context of the eggplant versus the cucumber, right? and since some contexts, those are very similar. And other ones, they're fundamentally different. And I I think this keeps gone going up though. I mean, you mentioned the sort of accuracy of the generated SQL. it sounds like an inevitable direction we're going. And I I hate that you said that SQL is a Turing complete language, and so that's good. but there there is this aspect where we can say that a a DSL that is internally consistent and guaranteed to be objectively correct in what we're generating is still better if it's a dynamic DSL DSL. And using the appropriate language and then converting that to something that the database understands in order to do the query in the first place. Is that the eventuality of where we're going with technology in in the space to be able to search databases effectively at at scale? Or is there something on top of that that you're already envisioning? Mark (49:12.05) so I I think if so if that existed, like of course that'd be way better. I I guess like the problem I'm trying to I'm trying to highlight is what does it take to actually bring this layer into existence? It kind of requires defining like what structurally correct means for like your database or like your business or your organization or it and s and stuff like that. That's like not really Warren Parad (49:26.129) Mm. Mark (49:40.738) That's not really like a programming problem. That's kind of I'll I'll call it like a some combination of like a social problem and like a UX problem and and a project management problem. And and that's kind of like where I feel like all of the all of the minds you might accidentally step on in like s in the semantic layer world might lay. and and so that I I think like Warren Parad (50:05.23) Mm-hmm. Mm-hmm. Mark (50:09.826) Kind of like when we're selling TaxUL, like the thing we hammer on is like like time to value, like time to value, time to value. Like how fast can you like get in someone's hands and happily using it? Semantic layers, if you use them naively, unfortunately, are are kind of like counterproductive of this because they they they kind of make you define everything before you use it. And and that's also like kind of why I brought up creativity. Like what if you what if you want the language model to think about something. you haven't like exactly defined already, well, then you can't if if you're only using the semantic layer, like no, you can't do it. hence what we try and do is kind of like like semantic modeling is great. Correct by construction is great. Let's try and get there like incrementally. Like write some some like SQL. It'll it'll be wrong sometimes. It'll be right like more than often than you think though, because language models are pretty smart nowadays, especially if you can if you like connect them to all the definitions and business documents they actually need to do their work. and then like over time, like incrementally build that up in the same way that like if you're like vibe coding, you're incrementally building up like an application from maybe start with some front end stuff, then you add a database, then you add some endpoints, all all of that. Warren Parad (51:31.835) So what so from this regard, you're obviously trying to approach the innovation in this space in how we're doing searching in large scale databases. some companies call them data lakes. I guess most of them are actually data data swamps where nothing of value is actually stored in them whatsoever. Where is the challenge today to actually do the query design or semantic search or embedding based search or the correct by construction DSL generation to be able to do the query? Like where is the biggest challenge in even being able to do that? Mark (52:14.88) I think the I I think right now the the the natural way of thinking is okay, what's like the best system? Like what's the best retrieval system for like data assets? What's the best DSL? Like what's the best like how do we define like a correct like semantic layer with all of our domains mapped out? kind of like Warren Parad (52:38.8) Right. Mark (52:43.394) the the the direction I

want to move the the market here is like your agentic analyst or really any AI system sorry excuse me your agentic analyst any or really like any AI system is actually like a dynamic one that like starts out with nothing and then eventually like maybe like us for now eventually has like all the components you want from from it and kind of like A lot of are you a is your organization or are you a successful user of AI is not really like how well designed is your end state, but how like how good is your system at like every intermediate state between now and the end state? You kind of brought it up earlier with like what's the embeddings are great, like semantic search and rag are are are great, but there's this like large cost in setting them up. Warren Parad (53:32.528) Mm-hmm. Mark (53:42.892) And so if you like go about that wrong, you're gonna like take a a ton of time to get to this like state where finally everything is like raggable. And like pray that you didn't do it wrong because if you did, now you have to like go back to the start and then do it all do it all over again. kind of like I w the the thing I I I would like people to think about more is like Warren Parad (54:02.226) Isn't it? Mark (54:10.7) All the states in in between like zero and and and one hundred and making sure you have a good product like at every every point in in in that Warren Parad (54:18.411) Good luck with that. Getting people to actually evaluate or they have a good product. That's a that's for sure a challenge. I I'm sort of curious. Like so while you're building TextQL, are you taking insight and innovation from your from your days at Meta? Is there something specific that you are working on internally that you like right now that seems like it's the it's the showstopper or the next biggest innovation that you're currently working on to achieve? Mark (54:46.816) so the I I I guess like the the biggest like new thing we've done is I'll I'll call it like our take on the semantic layer, which we call or we call it an ontology, but our biggest thing is like it's the fastest time to value and time to build semantic layer. And so if you take my analogy of like going from zero to one hundred, we make it so that let's say you have like no documentation whatsoever in your data. Warren Parad (55:05.765) Mm-hmm. Warren Parad (55:16.016) Yeah. Mark (55:16.246) It it's like pretty decent. It'll help you like build out the semantic layer on its own using AI. Let's say you're like in the middle, it's still okay because maybe it it like tries to answer one question and it can do it with the semantic layer and then tries to answer an another one and it can't. That's okay because we can like break down break out into the unmodeled part of the database and then write like a raw SQL query and then notify you that it was a raw SQL query. as to like what did I use from meta? Warren Parad (55:21.605) Right. Mark (55:46.088) kind of in interestingly enough, like a big part of a a big part of like doing classification work is user experience. The users here being defined as like all of the ML scientists and like policy experts and like single region experts who are trying to define these policies without necessarily being like expert programmers Warren Parad (55:59.334) Mm-hmm. Mark (56:14.732) Or maybe they are an expert like ML programmer, but they're not like an expert in terms of like orchestrating something to be run at like the millions of requests per second, like QPS. You can think of like if you combine scale plus audience, you run into like a representation problem. Like how do you represent or how do you provide like the best represent representational interface for a non expert to get as close to expert results as possible? At meta, the the the Warren Parad (56:25.253) Right. Mark (56:44.34) expert results were like perfect classification of spam and not spam. I I I guess at a text UL, the re the the perfect representation is trying to get to as close to expert level like data science, data engineering, with as little actual like knowledge of of the inner workings of those as as as possible. So you can think of them as actually like pretty similar. And again like my my background i is Warren Parad (57:09.745) Mm-hmm. Mark (57:13.356) Like I'm I've been super into like programming languages and functional programming. Kind of like even if that's not like even if you're not designing a programming language for these things, that's the toolkit. The the toolkit is to think about like how you represent information the the best. and then how you do that given your audience isn't the same between like customer to customer. Warren Parad (57:39.384) you s you spoiled this a a bit before during the pre before we went live, you were spoiling a bit for me how you were secretly at heart a Haskell engineer, and that you absolutely love Haskell. and I I feel like you're not the first guest on the show for in in the recent time that has suggested how great functional based programming is. And I'm wondering if LLMs have inspired a new level of lack of control over the world and needing to be more refined in how we're actually communicating with the systems that we're building or passing on our expectations. And if this is just our cry for help in in the night to be like, you know what, I'm gonna use a functional based programming language because it's gonna make me feel more secure in what what I'm building. any thoughts there? Mark (58:29.832) I'm well you you said I'm secretly a Haskell pro program at heart. Like I don't it's not really that secret. Like you can look at my my GitHub and see that like almost all the repos are in are are in Haskell. But I mean I I think you're I I I I I think you are getting at at at at something, and and that's kind of The the amount of of of just like output, whether that be like text or code, that an LLM can do compared to even a team of humans, is like is is absurd. Like maybe ten to a

hundred X. And that's if you're like being responsible about usage of it. and like I I I think we've all kind of thrown up our hands and said, like, okay, mi like technically we have code review and and I do try and like look at everything. Warren Parad (59:26.171) Technique technically we have that. Mark (59:27.288) like as closely as as as as possible. But at like the end of the day, like people's attention like does slip from from from like time to time. And and like the more volume, the higher pressure there there is for your attention to slip when you're like checking an L L M's code or trying to figure out like how something was was was done or or like whether you just like wrote a huge L L PR and and if you wanna like look really look at every line before submitting it. Warren Parad (59:42.353) Mm. Mark (59:57.068) The thing I I guess like the good thing about like typed functional programming and other like very structural ways of of like outputting code or or or text is that they're like principled like and and highly rigorous and and so having that structure grants a level of of assurance. Maybe we overstate that assurance because like maybe the principles that like the LLM built the Haskell on are also like suspect. But at at the end of the day, it is it is kind of I I I think reassuring to to see that okay I submitted this like 2000 line PR and according to this like according to the compiler that that unless and compilers have bugs sometimes, but much less than like than like LLM code. according to this compiler, everything between like the assumptions declared in like the interface and the as and the assumptions declared in the code are the exact same. And and I I think that I think that makes people feel a lot safer. Warren Parad (01:01:09.681) Yeah. Warren Parad (01:01:17.709) you're definitely getting at a sore point for sure. I think early on, maybe a couple of years ago, there was sort of a controversy where some of the core Linux modules were converted from C to Rust. And I believe the goal was to avoid all the sort of problems that Rust solves, memory related, etc., as far as vulnerabilities go. And we can be sure that none of those exist in the compiled modules. But the problem was that new issues showed up through the transformation process. And so I think know you're definitely on to something there to say that we are avoiding certain concerns and maybe it doesn't actually solve all of them. But on the flip side, I I I think that I am also a little bit more optimistic where we're able to define the the semantics or the invariance with our our code or our business logic in such a way that actually does give us additional guarantees there. So I I can appreciate this migration. I know my own preference is Rust when I can, because I feel like using Rust does avoid some of the complexities that show up in larger systems. I also think that yeah. Mark (01:02:20.674) Yeah, I I try not to say it like too much because it it'll like

poison people's brains, but the the the kind of feeling of of like if it compiles it works is is definitely there. Of of course it's not always true. Like you can you can add two when you when you meant to add one, but it it there's definitely like a a high level of reassurance there. Warren Parad (01:02:29.913) Yeah. Yeah. Right. Well Warren Parad (01:02:38.403) No, I Yeah, I definitely I definitely agree. actually this came up in the episode that we were recording earlier earlier with Cassidy Williams on basically that there's correctness in what we're what we're generating and a a commit to getting the correct answer. But more importantly, is that there are vulnerabilities, just safety checks or security issues in what we're building today when we're not using one of these languages that just compiles. and if it compiles, it it's correct and it it works to some degree. And such that we'll migrate to them because there's clear wins in doing that. And from there, there are still innovations that we can have on top of that. I think one of them is known as formal verification, which we just can't do with like a scripting language or even a a weekly type language. And this is the aspect of ensuring that what we built or the code that's running is not only syntactically correct, but and avoids these sort of vulnerabilities that come up because of memory leak leakage, et cetera, but is actually doing what the business declared. There's some aspect to the code which makes it objectively correct on a higher level. And I think we'll get there. Mark (01:03:51.104) Mm-hmm. Right. I I I guess like when you think about it, if you say like define like your reasonably definable invariance like in the types and then have your program like off the types, well even with like ten thousand LLM written lines of code per day, you can be rather sure that like the invari the type invariance will be a much smaller service area than than like all of the business logic. Warren Parad (01:04:05.585) Yeah. Warren Parad (01:04:15.825) Mm. Yeah, I I I mean I I definitely I definitely like the optimism there. One of the problems is that like I still think that LMs tend to generate new stuff rather than pulling the semantics out of what have already been generated. Like we have the same type for a sixteen character string generated over and over again, except this sixty this one's a sixteen character string that represents, I don't know, the order ID. And this one's a sevent fifteen to seventeen character string that represents the invoice ID. When both of them, you know, still have to go in the same column in the database because it's an auditing Mark (01:04:48.961) Mm-hmm. Yeah. Warren Parad (01:04:49.075) table and the foreign key ID, you know, actually has to be sixteen characters. And when it gets it seventeen characters is gonna be a problem. And so well thank you LLM for generating the invariant that says the invoice ID Mark (01:04:58.084) yeah, i it it likes to break out of it likes to break out of like centralized abstractions. Have you ever heard of this like debate or tension between like Warren Parad (01:05:03.718) Yeah. Mark (01:05:10.988) Locality of behavior and don't repeat yourself. Warren Parad (01:05:16.06) yeah, absolutely. It's one of the I I think it's one of the biggest struggles I've had in mentoring engineers for my last twenty years for sure. Mark (01:05:26.892) Right. So I I think it it's definitely like w it it's a debate that that's like come up in engineering, like stuff like and and I I think like the case it was like all don't repeat yourself for thirty years. And now I think there's some advocates for locality behavior. I fortunately or unfortunately, an extremely large advocate of locality of behavior or large language models who prefer to do everything within Warren Parad (01:05:41.785) Mm yeah. Warren Parad (01:05:52.377) Yeah. Yeah. Mark (01:05:56.546) got like literally one hundred percent of the logic defined within their context window. Warren Parad (01:06:02.32) I I think this is one of the things that the pendulum swings back and forth to one of the extremes over and over again. just for context here, for anyone who's not familiar with these ideas, the the don't repeat yourself or or dry of the solid principles basically says if you're doing the same thing in multiple locations, you should abstract out some sort of abstraction or function or method or class that encapsulates that functionality. So like a sum method if you're adding numbers over and over again, and then just call the sum method. The the problem is that in practice you get extra complexities that are added into it, like what are the Parameters that are allowed in? Are they just strings? And does it parse strings to in so are doubles or floats before it does the arithmetic? What about overflows? What about negative numbers? What happens with irrational or complex numbers? And so do you end up with a single method that's just called sum that and I'm sure some mathematicians have a opinion here about how you define a group and the operations on the group or on the set in order to decide, you know, whether or not it is a group and whether or not that's the appropriate function. And I'm sure I just lost everyone when I said that. so welcome to real analysis for for mathematics and how or group theory really. And so I think that's one thing. the other the other thing is that realistically when when we are deciding where the aspects or complexity of our program should go, it really does require a little bit of a design philosophy on whether or not it makes sense to do the subtraction and have one infinitely configurable method, which then loses all its value, or have something incredibly opinionated. And obviously the optimal is somewhere in between, whereas the locality of the behavior of the function defines where it's like, well, in this place, we only need to add two integers. So we will just write add integers

and that we'll be done with that. And we won't care about the all these Warren Parad (01:07:53.732) Dead cases, but you lose some of the understanding and the expertise that has been brought by building up your single abstraction of the sum method over time. And so I think what's interesting here is to be aware of the controversy or the discrepancy and the duality of the opinions in the space, and then figure out what actually makes sense. And I feel like LLMs just this is one of the areas where they fail. And I feel like this stems from all the failures that LLM generated code always has, which it doesn't fully grasp. Why to pick one of these solutions over the other one? I mean it will if you ask it when should I do this, like should I use this A or B, it will give you a whole list of things, but it's not going to pick the right one for sure. And I I my pick in a previous episode was this paper that compares the Linux operating system to E. coli bacteria, as far as the DNA for protein creation, where it says evolution has concluded that the most critical functions are highly replicated throughout the DNA, which in a way represents similarly to what we see with LLMs with the locality of behavior for the functions. Because if a function is so critical, if there's a mutation in that function or a bug. That's introduced or a regression, then the whole organism ceases to function. But if you have that same function replicated everywhere, only where it's necessary, and it's a different instance of that function. If one of them introduces a mutation in the DNA, which creates a protein, which means that that thing can no longer function, the rest of the creature or organism still can survive as long as it's not like. I don't know, ha handling cell division in some way. Or, you know, it does get a cancer and and die. And so like there are some critical failure modes, but for most of them, it can still survive. And maybe that mutant form is actually better than what was previously. And I feel like the real challenge is identifying, okay. Warren Parad (01:09:45.636) Are we in a scenario where we're creating an abstraction that directly duplicates that functionality and has to be the same everywhere? Or are we the Linux operating system where, you know, there's only one version of that module? And if there is a failure there, then that means that every single version of Linux is now susceptible to a security vulnerability because of it. Mark (01:10:02.742) wow. Yeah, I never really thought about like redundancy in the code as a potential defense mechanism. I mean, like you might just say like, don't screw up like the the the single version of of of of the function. But I I guess like the need for defense i is i is probably a hundred times more than than than ten years ago. Warren Parad (01:10:20.283) Yeah. Warren Parad (01:10:28.966) yeah, for sure. I I think the the wisdom that I've shared here a lot is look at the function that you're creating and decide, is there another function that has the same core values and expectations for how that function should evolve over time as the new one you're creating? And that's not a simple thing to just answer on the fly, but I feel feel like that's fundamentally the aspect of doing software development, software engineering is doing this activity. And so we still have to make those decisions. And I feel like I feel like people who spend more and more time generating more code are increasingly avoiding making those decisions and understanding what's going on there and skipping that part of the review. And so we're going to end up with a lot of the locality of functionality, but avoiding the question of does this actually have to work the same? And those are where bugs creep in. Mark (01:11:16.93) Yeah, no, totally. Warren Parad (01:11:20.549) I have a lot of controversial controversial theories on this. actually, maybe I'll throw one more at you. the value that is the business value that is created by software you're writing is proportional to the time the a human has spent creating that software. Mark (01:11:38.198) Hmm. I would say at the Well It it it might be true if you define it as like an upper bound because there there's certainly like like like random like like Java product with a hundred thousand engineers versus like more pleasurable to use product with like two hundred right. Like it's not the I mean it it might be written in Java too, but but definitely the the the legacy one was written in Java. Warren Parad (01:11:53.529) Mm. Warren Parad (01:12:02.054) Yeah. Warren Parad (01:12:07.29) Not written in Java. Mark (01:12:18.806) Yes, I I it it's it's if you define it as like an upper bound, that's I I think that's definitely true. Like keeping in mind that you you can't you can totally like waste human hours on making the the the software better. but Warren Parad (01:12:19.279) Yeah. Mm. yeah, okay, let's let's Yeah, okay. Yeah, no, I I totally agree. Yeah, for sure, right. Yeah, we've seen lots of organizations waste human hours in the name of software development. So I I think I think for sure that that that has to be true. Mark (01:12:39.956) but yeah, I I'm I'm like leaning towards I I'm like leaning towards yes, the the th this seems to be true because I I think like the there are claims floating around about like how this or that person is like a hundred times more productive with like AI software engineering. I I I'm not sure I I see like a hundred times better like software products compared to like fat compared to like ten years ago. Like ostensibly, like Jira now has like seventy percent of the or or more of Jira developers all use like AI in their in their coding. is like the Warren Parad (01:12:59.662) Mm. Yeah. Warren Parad (01:13:09.104) Yeah. Yeah. Mark (01:13:24.12) the the Jira like profit or or like or customer satisfaction or anything like tripled what it was com compared to to like eight years ago b before anyone was

doing AI coding. Like I like I I really think so. I I it so it seems like AI helps a a lot, but there is some like underlying variable that that like equally influences quality and and does not seem to be moved and and AI like anything or automated anything doesn't seem to move the needle. The only thing that moves the needle there is is like lots of of like thoughtful like human like groundwork, at least with the current level of AI capability. Warren Parad (01:14:07.929) Yeah, you know what? Honestly, I'm I'm like this close to writing a blog post called the like Warren's Laws. sort of like Maxwell's laws for the electromagnetism and the nature of the universe. to like this one is one of them. The the upper bound on the relative gain is proportional to to the human. I I I do think that there's something to be said about there there's clearly some value here with using LLMs and I feel like understanding c the core of of what it is is super important. but maybe that's context for an potentially another episode. One of the things I do wanna sort of circle back around to, and maybe this is a a question that's still relevant in the space, is so it it it seems to me that you know you're being very careful about which where you're using LLMs in the software development process and how you're interacting with them for the benefit of the customer ingestion for the data, for how you're generating DSLs or the complete Sorry, what was the term used? for the generated queries that you're making that are provably correct. what yeah. Correct by construction. I I I I I love that term. do you see that there is a corresponding challenge for scale? Like if you only had to perform a hundred requests per day on against the database versus Mark (01:15:21.665) like like correct by construction. Warren Parad (01:15:42.448) I think somewhere in the classification architecture for Facebook, you were at 50 million. Does this change your approach for interacting with that data? Or do you is that sort of like a different orthogonal axis that just talks about how to make the the solution reliable? Like is it, okay, well, we're doing 50 million, we need to make sure we're using different software languages or or doing some sort of performance testing, or is there a meaningful difference in how we're approaching doing the the development or the interaction with the L LMs. Mark (01:16:13.25) Hmm. Frankly, I don't have a don't have a super good answer to to this. Like theoretically, theoretically, like yes. Like like in like I was just writing SQL, like at at Facebook to do analytics, like and under like underneath the hood, like if you have correct like primitives like like sharding for for example, like everything should like work structurally the same at on on like one gigabyte to to like a million gigabytes. Right. the like yeah, th theoretically, but it's it's it it's kind of hard to say because always it's always the case that like one like Warren Parad (01:16:48.613) Theoretically. Mark (01:17:02.24) when you actually go to the million gigabyte or more scale, like implement like tiny details do always like creep in. Like you have to you have to think about you have to like think about how data is distributed like not just the shard, like s on on like some for some function of of of like the shard. So you kind of already broke the mental model with with w w with that. and kind of like going to the flip side. Like what if you only had to deal with like ten gigabytes, which is like more often you think. I that's probably like years for for TextQL where all the relevant information could fit in in in like ten gigabytes. there's actually like a lot of power that you can do that you can get just by saying like screw like sc screw scale, like I don't I'm not gonna think about this. Let's say I'm I'm at like ten gigabytes forever. Now I can like use what what I call like laptop tooling for for everything. Like like SQLite, like DuckDB, like random like PHP server like like living on on on on like my my laptop, like bash scripts and and and so on. And the amount of composability you get from like living on on like one small machine is is like absolutely insane. Like I I think you can do the same Warren Parad (01:18:04.465) Mm-hmm. Mm-hmm. Mark (01:18:30.15) Again, it it's theoretically just SQL in at Facebook too, but somehow if it's just like bash and SQLite, I can do the same thing like a thousand times faster just on my computer. Warren Parad (01:18:40.697) Yeah, I I I think you're absolutely right. if you only have ten gigabytes, you're like, well, I can just load this all into memory and then we can do an optimized investigation of whatever the data is. Like querying in memory is just so much easier. We have all the data constructs that we want to use or whatever data structures that make sense. Right, exactly. I mean, obviously the challenge is like how do we load the data in from the database so that it's like Mark (01:18:54.53) Right. You can like jump anywhere, like arbitrarily, like all Warren Parad (01:19:02.529) Exactly in the data structure which makes it easier or faster to query over and over again. But it's gonna be so much faster if we just do that at the start than trying to read from disk. And obviously at at scale, you have s some of these other problems where there is a real latency between a particular kind of request and another one if one shard is in this data center and another shard is in a data center that's you know halfway across the country or something like that. Mark (01:19:25.604) sure, or or you like want to or you're like trying to query in like the absolute worst way that like like takes a tiny scoop from every single shard, which is probably like much easier than than you think. Warren Parad (01:19:39.203) Mm. Right. I mean, you just name your indexes wrong or you don't you didn't even think about doing that in the first place and why w you know, of course some data will end up

getting sharded ineffectively. And ha you'll have to deal with that then. And I the problem is that I see often a lot of times these show up at scale but are sort of orthogonal to utilizing LLMs or the technology that we've built up recently, vector databases, et cetera. it's like almost completely separable until we get to the point where like, we want to have LLMs write the source code for us or the schema or, you know, where the indexes are, because then it will get that wrong, or generate the queries for us because it won't take into account that whatever our infrastructure is. And so I I do see that there is still some sort of meaningful distinction still, but it's not as important as just understanding that at scale there are certain things that we've already figured out and have to do. Mark (01:20:30.722) Right. I'm like I I guess like the promise of of like having the cloud and and being able to stand up an entire app from like the Ib US SDK is that like it's kind of like the correct by construction version of like like DevOps or or infra where hey hey like I'm I'm using like ECS. it it appears to horizontally scale like right out of the box. Or hey, I'm using like Warren Parad (01:20:42.758) Mm-hmm. Mark (01:20:59.146) or like or or or Aurora or or something. Hey, that also like horizontally scales, like right out of the box or or or like Dynamo D B. But then like you run into all of these like it's so easy to like accidentally like make some kind of locality assumption. And then like you go up one order of magnitude and everything explodes. Warren Parad (01:21:01.874) Mm. Warren Parad (01:21:17.252) Yeah. I I mean you say that and then I I just think back to I think it was a couple episodes ago where we were discussing about RTO and RPO, about handling either malicious attackers like encrypting all your data and how do you recover from that or losing a data center and how you recover from that. And the the clouds don't give me that burprining. I mean, it's w it's not even like you pay for it and then it happens. You have to actually invest in understanding what the building blocks are that the cloud provider offers you so that you can even utilize them correctly. Like you mentioned, like RDS versus Aurora serverless versus DynamoDB in AWS. Some of one of those provides you multi region backups and active active configurations at scale. By default. The other two make it incredibly painful to make it happen. And in practice, it could be problematic. So I feel like, yeah, I do agree that the c w going to the cloud does solve some of the problems with scale. I think the original promise was more on the lines of the zero to one velocity to make that happen. And over time, I think they've gotten better with adding the necessary building blocks to go further than that. But I don't think the user the the usability or the user experience there helps make people make the right decisions. That being said, maybe that's more of a tangent for one of

the previous episodes that we had. So maybe now's a good time for us to switch over to picks for the episode. So Mark, what did you bring for the audience today? Mark (01:22:55.942) sure, I I I think the thing we we we discussed beforehand was like burpees or hotel room exercise in general. Like I'm in the hotel room right now for all kinds of like events and like selling, like I guess like perks or what happens when you're when you sell to other businesses, you have to be on the road a lot. I like to exercise, but I used to always like fall back on or it I was I always let it go to the wayside whenever I was like on the road or especially busy or or anything. And to solve that I just started doing burpees in my ho hotel room. that's it. I mean it's a it's a workout where you're just like standing up and like jumping up and down and and and doing push ups. Good enough for me for for like a couple day couple day road trip and and it means like when I do get do get back home, like I'm not out of the habit anymore. Warren Parad (01:23:52.582) I feel like there's always sort of enough room somehow to do a burpee in the hotel room. Mark (01:23:58.006) Right. Unless you're in unless you're in some kind of like hostel with with that's like six bunk beds in a two hundred square foot room, then yeah, yeah, definitely you'll have space. And even if not, like go to the sidewalk and then and then do them. Warren Parad (01:24:12.74) I the hands on the sidewalk. I I I think I I I think would be a little bit of a challenge. I mean, for some people, I'm sure that's fine. Honestly, the hallways in the hostels always were incredibly wide and s for some reason. It's like the rooms incredibly small and you know, double or triple bunks or whatever, but in the h the hallways super wide, you could absolutely you know, just jump up and down like that. I I love it honestly. B I mean when I when I travel, I feel like I definitely use that as an excuse not to exercise. Mark (01:24:35.166) there you go. Mark (01:24:43.276) Right, well now you don't have one anymore. Warren Parad (01:24:46.13) you've you've ruined my my whole travel life. My whole my whole conference speaking circuit was always like, this is for me. I travel, I enjoy talking with people and discussing new topics, and I just get to chill out and regress in a lot in a lot of ways and and now you're like, well, you've got no excuses left. There were always so a long time ago when I tr when I was traveling, I used to try to go if there was a swimming pool, I used to try to utilize it in the hotel. and if there was an exercise room. But there was definitely a huge barrier to have to overcome in order to do that. Like finding the time in the day to add that in was definitely problematic. And as you mentioned, if you're traveling a lot, like especially in a sales position, you may be traveling three three months, three weeks out of every month. And then at that point you're like, well I'm I'm pretty much just not exercising ever. that week I get back, that's for me. I have to then relax and recover during during that time. So you you pretty much completely avoid exercising completely in that regard. Mark (01:25:46.486) Mm-hmm. Yeah, I I I think the thing about or like whatever exercise you can do, just like standing up in in in like the space you are right now, is that it like totally removes the friction. Like and like motivation and like love for for like doing an activity, whether that be like exercise or even like like programming even, it it like wanes. Sometimes you don't feel like doing it. But what's the best way of continuing to do that regardless? Just like taking out all the fiction. Warren Parad (01:26:18.414) Is there a whole Mark Hayes exercise routine for traveling hotels or is Burpees it? Mark (01:26:24.474) so I I mean at at minimum the the like frictionless thing is is like just just like burpees, do do like a hundred or or something or or like split it up as well. And then if there's an exercise room, then I can build up the motivation to go go there and and like do do like pull ups or like exercises or run or or something. All the stuff that like burpees doesn't like cover super well. But Once you once you realize you can do exercise like right in your room, then all that stuff's just a cherry on top and you already you already like passed the test, so to speak. And now it's just about passing to to going to like an a an A or B. Warren Parad (01:27:08.856) Mm. Do you do you change your exercise routine based off of where you are? Like if you're not traveling, you do one set of things and when you're traveling, you do something separate, or did Mark (01:27:16.502) because I I I guess like because once I'm home, I actually have a gym and and stuff. I definitely go for like way more d diversity. Like if I was only doing burpees forever, that might be like bad. I it seems like you you get some kind of muscle imbalance from from doing that. So I guess compensate on the other end by trying to mix it up as much as possible when when when I when I'm when I do have the motivation do other stuff. Warren Parad (01:27:23.57) Mm-hmm. Warren Parad (01:27:43.676) Do you also exercise while you're on vacation? Mark (01:27:46.754) not nearly as much. May maybe a couple of days, but vacation here at least like for for the past couple for me, I I'm like walking so much that I it seems like a reasonable enough substitute. Warren Parad (01:27:58.97) Mm. Warren Parad (01:28:03.411) Yeah, I like to think. Or I'm just like physically exhausted by that the butt point. I don't know if I can. but I also pick like vacation destinations where there's like a lot of like tr temperate climate or something very exhausting and far away from infrastructure. So yeah, I I I totally get you. I like it. I like I like the recommendation. okay. I actually wasn't sure what I wanted to share, but I found for my pick a article. Called I Left Port Twenty Two Open for Fifty Four

Days. it's by Armin Hussein, and there's some interesting conclusions based off of leaving Mark (01:28:43.852) The idea is like anyone could SSH in there. No, no. I I'm I'm just like confirming. That's the idea. Warren Parad (01:28:46.906) Yeah. have you seen the article? yeah, yeah, yeah, for sure. So like SSH server running on port twenty two and you get to see what people are doing. Like what like what what a p what what does the internet try to do with with your port? And what's really interesting about like the conclusions that he finds from it, like I I don't know what you would expect to have happen. so it wasn't a real computer. It was like a virtual machine that was specifically set up to capture and respond to certain commands and events. Mark (01:29:16.938) I guess I've I've never like read this article. I I guess like someone would would like do some like Bitcoin mining program on on it or like ransomware. Warren Parad (01:29:19.78) Yeah. Warren Parad (01:29:25.35) Well yeah. Well that's the thing, is like you try as a as an attacker, you know, you it it's it starts you figure out what the optimal thing is for you to do with a machine that you find that you can SSH into. And that means from a defense standpoint, you may be interested, what are people trying to do with my server so that for my actual servers, if I wanted to configure them in some way to deal with malicious attackers, I can look for a same sort of signatures. So a question is like, well, what did he actually find? And the interesting thing is that most people, most of the attacks that were done, like Bitcoin miners or whatever, were just basically an automated command that was run straight against the machine, like SSHN, trying any password. They get the password, it works, and then they just run a single command, usually CNC based command or C2 infrastructure, basically, it would be a single command which would pull stuff from a third party server that the c attacker owned to run. infrastructure on that machine, like a Bitcoin miner or whatever. And if you run you know, Bitcoin miner.start, the the fake server would just respond, okay, you know, like it's running, right? And and that's it. End of story. And so the attacker, you know, thinks it worked and then they go away and you capture the signature of what these attackers were doing. And so you can capture all of the C2 command and control servers that were in effect that were you know currently being used at this time by proportion and whatnot. And what he found is that like 99% of the attackers, the visitors that went there, never went on beyond like a single command. Basically, they found the open server, they attempted to authenticate with one of like a thousand most common used pass username and passwords. Then they run the result and that's it. They're done. They they disconnect and the server just keeps on running. Some of those actually attacks didn't even run the script in the background. So as soon as they

disconnected, it just stopped. Mark (01:31:17.95) Okay. Warren Parad (01:31:19.449) The attack, like the miner just would stop at that point. Like the command wasn't even sufficient. but like in the top one percent, there was actually some really interesting stuff going on. Like, you could he could tell how sophisticated an attacker was based off of the type of command that they were doing. Like, were they running something that would get saved in the user's bash history for authentication? So, like, if you went in later, you'd be able to see this in the history. Well, there was like some. Mark (01:31:21.666) Mm-hmm. Warren Parad (01:31:47.548) Clearly state sponsored actors, some from the probably Fen French government with French IP addresses that were you know disabling bash history and using direct port protocol to handle messages and whatnot to completely avoid getting anything logged. most of it was crypto related in some way. People trying to run Bitcoin miners or because Bitcoin is worthless, Solana, which is much cheaper. You don't need infrastructure to run Bitcoin miners. I think it's it costs. I think right now it costs more to mine a Bitcoin than Bitcoin is worth. So people who mine Bitcoin are only doing it on with malicious intentions, which I think is absurd. yeah, yeah, for sure, right? you get some cloud credits and you run it. You hijack someone's account and you run it, right? You're not doing it. I mean, if you're really smart, you use Monero instead, because it's untraceable. So no one can track that these are from the same. Mark (01:32:28.734) Mm. Basically. Warren Parad (01:32:46.655) account or potential threat actor. But a lot of it was complete garbage. that wasn't very smart. They weren't doing very s something very interesting. Some people were just curious, like, I found a server. I'm gonna SSH in here and see what I can do and then run some interesting commands. A lot of them were just running LLS to start. Like where am I? What do I have access to in this directory? I don't know. I just I found this article so interesting. Mark (01:33:09.922) Wow, yeah, that that is cool. It it's cool how I guess this the sophistication four follows kind of like a power law. That that's one. two, it's kinda funny. I i it's not clear that anyone like benefited significantly from the successful attack on on on on the server. Warren Parad (01:33:31.069) yeah. no, for for sure not. I mean Mark (01:33:32.566) Maybe someone got enrolled maybe got enrolled into like a botnet or something. That'd probably be the best outcome out of what I can imagine for the attacker. Warren Parad (01:33:40.793) Yeah, well there are there there is a so outside of the Bitcoin or crypto mining, there is a small section where they were basically just recording metrics about the actual machine. So like where it was, its IP address, what it was actually running, which version of Linux or distribution, Ubuntu, Kubuntu, whatever, Gentu, et cetera, for what presumably would be a later attack. Mark (01:33:57.208) Mm. Warren Parad (01:34:07.087) So that later they could come back and be like, you know what, we want a machine in this region with this IP address so that we could use. And so it was just about fingerprinting the actual machine. And a lot of the messages were were from that. Yeah. so like I said, really interesting. It's not very long. It's a like a 20-minute read. pretty interesting article. Mark (01:34:14.446) Mm-hmm. interesting. Mark (01:34:24.632) Cool. Warren Parad (01:34:27.237) Well, thank you, Mark, for joining us today and telling us all about the classification at Facebook and what's next in the semantic layers. I I already hear that semantic layers are are over and done. We're already on the next the next great thing. So always interesting to hear that the latest tech is now obsolete. Mark (01:34:45.964) Right. just until un until the next one. until your next guest tries to yeah, another generation. Warren Parad (01:34:50.945) Yeah, right until until next week where we find out that the replacement for semantic layers is already obsolete. I don't remember what next week's episode is at the moment. but thanks all the listeners for tuning in for this week and I hope to see everyone back next week.