התמלול המלא של SQL is the worst DSL, except for all the other ones

broken. So when we shift to an automated system, we take the hands-on work that someone was doing and translate it to having to do the investigative work and the debugging work. And then if something goes wrong, also doing the original work that they did. Over time, systems become more reliable. So fewer and fewer people have hands on work. Now all the humans left only have the investigative work and the debugging work and never the hands on work. So even if they're able to identify the problems that've been created, they are unable to fix it because we just don't have those people anymore. So by automating stuff, we make the humans' jobs incredibly more challenging to actually solve. The the complexity of doing that job becomes greater. So unless you have a real problem that requires increased scale. Automating it only costs you money. So you should think about before introducing agents to any system, you should always ask the question, could we still do this with humans? Because if the answer is yes, then you shouldn't automate it because that's going to end up in a worse state. And I think this is gonna bring me back to asking about, you know, what do we actually mean by the data platform part and whether or not humans could still be building and maintaining that? I don't speaker-1 (06:37.624) think that that completely goes away. Well if if it goes away there's hopefully some new some some new future. speaker-0 (06:40.812) You hope. speaker-0 (06:47.938) What's the sort of frame of reference here? So if we have a log solution, we're throwing logs in it, obviously those are production logs. but maybe you're also talking about infrastructure logs or business relevant metrics, et cetera. speaker-1 (06:58.552) I love the question and I love the personas that you've introduced. It turns out some logs are business relevant. Some logs are inf only infrastructure. Some logs are both. And sometimes you need to do some things to the logs in order to make them business relevant or make them other things. And the the traditional set of data tools that have been available have created a bit of a spaghetti of okay, let's put all of our logs, for example, going concrete, let's put all of our logs into Splunk. But then the PM wants something, and so they're told, okay, use Splunk for that. They use Splunk for it, and then they need to share it with someone else. There ends up being this notion of, okay, let's teach the business analyst to use Splunk. They're like, I don't the thing that's most interesting about logs is the structure of the log and not necessarily how it's used. Where traditionally we've believed that you have to take the data and move it to different systems because of how it's used. But that's not actually true. The actual thing is that a log is interesting in and of itself. You can compress it with kind of domain specific technologies. There's specific ways that people want to extract things out of logs. There's specific ways people want to look at logs. But you can take that log and make it SQL queryable. You can take that log and make it exposed in other systems. You should be able to have your log, use it from whatever system you want. And it's the same for the agent as well. Like the agent should also be able to access that same data. speaker-0 (08:23.82) You're definitely an optimist. The fact that you think that the logs in these systems are valuable. Right. Because the number one thing that always comes to me is like, our logs are useless. Like w this log doesn't tell us anything that we actually want it to know at this point. speaker-1 (08:36.823) we've seen as we've talked with different people and worked with stuff is especially in the observability world, the useful lifespan of a log is much shorter than in the security world. speaker-0 (08:49.966) I mean, I think that's definitely accurate. You for sure often don't know what will be necessary later, but at the same time people often risk averse. I think that's just a true a true fact. and that includes hoarding logs that are useless because they don't they don't know what they're going to need in the future. With conscious thought you or deliberate action you would know you know, which logs would actually be relevant. The speaker-1 (09:14.552) The logs of people interacting with the site, the logs of people interacting with the online banking, or the logs of people interacting with the e commerce site are useful for the observability team because they use it to figure out, you know, what what caused outages, what was going on, how many people were impacted, what what things happened. You can you look for the logs in order to discover anomalous behavior that you can respond to. The exact same data set that's coming in that ends up powering each of them. It's the access logs from how people are interacting with the website. It's event telemetry from the single page JavaScript application that's being fired back about how the users are working on stuff. That often turns into, well, we have to get this data in, but then we also have to make sure that this subset of this data gets ETL'd over to this other system so that these people can look at this thing. And we have to get this other thing over here so that these people can look at that thing. speaker-0 (10:15.776) Historically, at least in my teams and my organizations, it we always go with the the DevOps mindset. If you build it, you run it, which also means you capture logs and you have to review what's there in case of an incident, you're responsible there. So of course the logs that you're generating have to be meaningful to you. and engineers obviously have a notorious track record of being hard users to please. So The platform that we utilize in any of those moments has always been the thing everyone complains about. they hate historically, they hate sumo logic, they hate Elasticsearch, they hate Grafana, they hate CloudWatch insights. You know, you stick in

something here, logs log z dot io or whatever, however you pronounce it. I mean, the list is goes on and on of all these platforms that I've seen w some team try try. Because every single time I come in, I I see they're using something like we hate it. I'm like, great, do some research. Find a platform you love. And then we'll try that one. They do that and they hate that one too. And the re the reason I I say this is because even if we were able to find something that they liked, there's still this problem of are they collecting the right information? And even if they are, how do other teams get access to that in a way which they're not held accountable for the data that shows up in those logs? And the reason I I bring that up is because historically there's this challenge, right? There is some production data. which another team or someone else in a different role, as you pointed out, like product managers or business analysts, say, you know, hey, we wanna be able to perform this query on our data. And historically they'll they'll use one of their giant data lake SQL back query solutions to do this. And the question is how does the data get from the production system into that other, I'll say third party system? Is there an act an ETL extract transformer load job that runs? If it does, who owns that? Which fields are extracted or whole databases extracted? And it doesn't seem like there's a right answer here because if you extract everything and and another team owns that responsibility, that's just always constantly broken. Like if an engineering team doesn't do it, then the the data changes all the time. someone complains, hey, my query broke. The ETL job breaks down because it all of a sudden security says, Hey, we can't be exposing full access on the production databases to third party tools. speaker-0 (12:33.548) So that that stops working. And even after all of that, someone says, Hey, we're not actually getting the data that we only want in the first place. It doesn't even exist. You know, we're not recording it. We're writing it down, right? We wrote other things. Just like in a lot of production systems, we may forgot to write audit trails, right? You know, you don't always have everything up front. And on the other side of the spectrum, you have engineering teams being forced to der pull hand picked fields. from their production databases and sending them over to your tableaus, your Splunk's, whatever have you. And then the data isn't there anyway. But at least they're responsible for doing it. And you can go to a team and say, Hey, we're not getting this field. Can we add it in some way? That speaker-1 (13:17.862) is a great question. It's a question that business intelligence has fought with for a very long time. It's data governance. you you in in the business intelligence world you end up with these with people who have these UML diagrams of table structure and they're like, thou shalt generate things that follow this structure and put things in and But developers never j like developers are horrible at at like aligning with that. They're like, I need this thing. I'm gonna add this field. I'm not gonna look up what it's supposed to be named based on these people in this other team. And so the there's that. So I don't I don't know exactly how to solve the data governance problem. I don't believe that trying to organize everybody and create structure out of it is actually ever gonna work. Because especially as you expand in an organization, you're just never gonna get everybody to agree. speaker-0 (14:12.344) I I think I I have seen that the most successful mechanism for the data transfer between production or engineering organization and their databases to other people to be able to use it is when the engineering team is responsible and accountable for deciding what their interfaces are going to be and what data they're going to expose in the schema is as if they're maintaining their API format in an open API specification, except the format and the schema that they're maintaining is in some third party or other service running, such as you know, an object store in the cloud somewhere or Snowflake or, you know, what have you. And then the team says, yes, you know, we are we we are accountable. We decide as part of the responsibility of our team that we're going to publish data to this specific repository. And here is the schema for that. And anyone who wants to use it can use it. And that's it. End of story. Because then you don't have a separation between Someone who's trying to define or has expectations on that schema and the team that is basically going to be held accountable for it. I think putting the ability and responsibility to the team that will end up doing this is the right thing. You know, you push down the decision and ownership down to the team that has it, and not expose the whole database. If speaker-1 (15:30.51) If you talk to anyone at a company, even at Yahoo size, and ask them what technologies are your company using, the answer will be everything. Like, are are you using this? Yes, we are. Somewhere. I don't know where. Somewhere we're we're probably using it. That that's just such a common mode of operation in a large enterprise. As much as you might want to force consolidation of technology, a new company is going to be acquired. And that company that you acquired had no idea about any of the things that you tried to enforce until after you acquired them. And now the question is like, is it actually worth it to carry all of that over? And I think all of this is just pointing to this notion of silos and independent decision making. I just saw every technology used in all the places. there were there was one team that was for telemetry and metrics, they were using one system for it. my background is kind of with Apache Druid and stuff. I started the that project and and built stuff and so we used the teams that I worked with we used Apache Druid to

do some advertising analytics and and things like that. And there there were conversations sometimes with the telemetry folks around, you know, we could use this for that. I don't think they were wrong They never really were like, Yes, we're we're gonna adopt this and run with it. And I don't think they're wrong for that choice. They're the ones operating the system. They kind of know what's going on. They're they're getting at it. They have their own service boundaries, so they they get to make their choice. We had other teams, you know, that were though for analytics it was heavy on Hive and on Tez and Hive running on Tez and and all of that stuff. This was back when Hadoop was still a challenger. At large organizations, that you will always have independent teams making independent decisions. And I don't think it's ever possible to fully align them. And so instead of doing that, you have to lean into what is something that gives us the ability to just store the data as is and then connect the dots later. When a a local team makes a decision and they're like, this is how we're going to expose it almost from an open API. speaker-1 (17:48.952) perspective, right? If it's a rest endpoint that they're using to expose it from, how does business analyst use rest? They they don't unless they they can tell SQL to go and issue REST requests, right? speaker-0 (18:02.314) See, I was gonna say correctly, but I I mean I guess I guess incorrectly is also the viable answer that's in the multiple choice drop down there. speaker-1 (18:11.511) Yeah, like they they but they're they're just not going to. They're gonna be like, I don't know how to I don't know how to how to interact with that. speaker-0 (18:19.054) So here here's a counter argument there. organizations that followed the rant that Bezos had set off, and now it's been I think over a decade, is that data analysts would be required to use the rest interfaces to expose or get the data that they want from other teams. So I I totally agree though, in most organizations, you don't have such a strong and specific and maybe mature mindset about how teams will operate on top of technology. And I I feel like if some teams aren't as technologically savvy, then they are limited in their capacity to solve particular business problems. Or if the tools they're using because someone made a bad business decision and bought a 10 year contract with a company to use a particular product that refuses to add in a OAuth to integrated supported interface to or even REST API. You know, maybe it's using WCF or SOAP or something horrific that reads and writes, you know, is it flat Text files on a on a physical machine somewhere, then you are limited by the real world situation there. One one thing that does come up, and I think you sort of hit on this, is that it doesn't matter what the situation is, even if you prescribe the perfect tool for the job, someone someone on the other end can always still mess it up. speaker-1 (19:37.772) Right. Well they'll mess it up. My team has defined this API. So if you want to use it, you need to change what you're doing in order to align with my decision. Getting humans to change their behavior is, I believe, the most difficult thing to do in the world. You're more likely to get people to be like, okay, I don't need it than you are to get them to actually change their behavior. And so this leads me to the what if. So what if You could have your data in a place, and that data could be accessible via the tool of your local team's choice, and also accessible via the tool of that the other team's choice, and also accessible via the tool of some other team you've never heard of's choice. But you have the data in that one place, you can own the definition of it, and it can be accessible from all of the different places that people have. That is, I believe, the Holy Grail. speaker-0 (20:36.75) Yeah, I know what you're talking about. You're about to say it it's going to be give them direct access to the production database so they can read the files on the server directly and use whatever data they want there. speaker-1 (20:49.634) You you have the files up on cloud storage. Yeah. So if you use that as a way to disintermediate your production systems from the actual kind of analytics and and access of it, have this three layer cake where things have decoupled to allow for it. And the the primary thing that allowed for that decoupling is the SQL query language. Where that's fundamentally the API that the visualization layers use and that the business analysts use. And so when you talk about the team should define the API that they're gonna support, the the thing is there's not just one API that you're gonna support. And in the logs world, people want to access logs from SQL. And the language is more belonging to the persona, where the data format itself, the log, can be queried from a number of different languages and different personas prefer different languages because the the language is aligned to the job that they're trying to do. speaker-0 (21:52.002) We have had a similar challenge in in our space when we're defining the, and I hate to call them policies, we call them access records for one of our products. And I mean, we provide like login and access control as a as a SaaS. And customers come in and say, Hey, you know, can you support this? Like, we love this particular format. I'm like, or this format, or this format. And realistically, it's all stored as JSON for us. It I mean you can store it as some other DSLs, like they're The every every domain specific language, DSL, is is the worst. They're they're all literally the worst. they're almost like a programming language, but they promise that they're parsable until people find vulnerabilities with the the language itself and find ways to in inject attacks in. And they're all unique that you have to understand the domain in a lot of ways to actually be able to write one or

write the the form to even do the query in the first place. And those that are amateurs and have come from one world make it very difficult to expose in different way. But it's very easy for us to add in different content types to our API and expose the exact same data via different formats because all of them are constructible. And the one thing I learned, I say, my last 20 years of engineering, and it wasn't that early on, maybe this is a bit of a regret, is that I always tried to avoid SQL when making an API for querying particular data on a production database. It it's the worst, you know, let's use something constructible, something nice. that is can be defined in in JSON and passed over the API and can be validated. But over time, what you find out is you just made another bad domain-specific language. You just made a worse DSL somewhere. And at this point, I'm just like, everything should be able to speak SQL because it has is essentially the thing that humanity has centered upon working correctly. And it's portable from one system to another one. And as long as you can provide that specifically as your query language, then often it's easy to add these other things on top. And it sounds like for your product, you know, for your data pipeline, being able to port from the underlying data store to the to the bad terrible DSL that every single different persona has said, you know, I want it in CQL in and SQL and whatever, whatever have you, and also understand the semantics of that particular language, like the the MySQL of the world versus the speaker-0 (24:11.476) MS SQL, the Microsoft SQL. Like it's not exactly the same, right? It it there are some fundamental differences there based on the data. speaker-1 (24:18.082) Getting the semantics right is definitely a key element of things. So there's some people out there who are like observability, SQL for observability. Everything should be SQL, all observability should be SQL. And while I think that everything should support SQL, I actually think that the the diaspora of different query languages for logs is indicative of a different usage pattern that is actually extremely meaningful for working with logs that is not as interesting for working with business data and like rigid column or structure data. And so I I think that I I do actually think while right now there's tons of different query languages for logs, I do expect that to converge down on a subset of query languages that will remain and that that will exist. Changing human behavior is hard. It instead of telling them that they they need to change what they're doing, if you can just meet them where they are, you can see that's in metrics too for observability. Like people use PromQL so much more than they use SQL for for metrics and telemetry. really PromQL kind of has, I believe, has won as a de facto standard for metrics and time series style things. I'm I'm less aware of different languages in that space. speaker-0 (25:41.484) And

you're totally right. The smart thing to do is to convert whatever interface someone has into the best, most generic solution that can be run on any single platform. So it doesn't matter what DSL someone is using, if you can convert that to an abstract syntax tree and AST, then you can directly execute that AST on whatever run engine you've got. We actually had an episode on how a company is doing that specifically and I will plug that in the description. But I I think this this may this solves a lot of problems, right? You don't have as long as you can come up with a can a transform from that to an AST that you can run and you just run the AST, you've solved the problem. And if you can't convert it to the AST, then you know that there's a feature in the dead DSL that you aren't able to fully utilize. And then the question is why. I do think there's an interesting corollary here, realistically, whereas if you are in a different space, if you can't convert it to the same kind of ASTs that you're used to running, are people thinking about that problem differently? And I'll just use an example. I think very early on I was full in on traces. This is the idea of storing log information that is grouped together during the whole request and maybe across requests. At a single moment. So when you want to query something, you're looking at a single object with lots of additional data in it. An example is like, let's say there's an error in production and you see a log message with that error. Now maybe you have some sort of correlation ID. And so you make your SQL statement, select all the rows where the correlation ID equals the value you know. You're pulling multiple different logs and you're getting that data. For us, that was always really annoying. Especially when you have a whole list of correlation IDs that you want to pull stuff for and you want to join it up with. So very early on, what we did was whenever there was a log message, well, during the whole program execution, we would store a property bag of all the context of all the variables of everything that was happening, and we shove it into the error message. So when it gets logged, we have everything there in one place that we can actually validate and use. speaker-0 (27:37.302) And I think this has caught on recently. I think in the last few years, Honeycomb was a huge proponent of what they were calling observability two point zero. And I think since then a lot of other companies have jumped on the bandwagon here of being like, hey, you know what? Storing line by line stuff isn't necessarily the right answer. The reason I bring this up is because that query sort of depends on the fact that we have data stored in a particularly different way. And so it wouldn't be straightforward to necessarily convert from the query language to an AST that could run on these different kinds of data. So it's not just necessarily about the transforming of the data language that we're using the the DSL to something that we can execute, we also need to understand fundamentally how the data may be relevantly stored for accessing, you know, things like indexes, the what's in a a wide table format or a columnar format, like these things are potentially relevant, whether it's stored as a blob and whatnot. So there's there there are a lot of complexities here that any company would potentially have to deal with. And obviously, you know, some areas that you you've seen particularly. I mean, obviously if it's just giving the consumer a DSL that matches their expectations. You know, you could do a lot of hacks under the hood there. but outside of that, you know, you really start to come into areas where there are challenges because fundamentally the data has to be exposed in different ways. speaker-1 (28:52.622) Absolutely. And and traces is the a great example. Like I've been talking about logs a lot. Interacting with traces and a language for interacting with traces will fundamentally understand the notion of a trace versus a span. But the actual like fundamentals of how is the data stored, how is it persisted, how do you index it, and which query languages are supported is actually a function of the data format itself. And so a trace requires different languages. And same with metrics like PromQL. You could convert PromQL into something that runs against logs. the language believes it's working with time series objects. And if you're reconstructing time series objects at query time, you're wasting a lot of CPU. speaker-0 (29:37.11) Well, I Postgres, I think, would disagree with you because they've managed to just say, yes, we're always going to take all the data and store it in rows and columns, including your you know key value storage and and graph graph databases and vectors and whatever and somehow make it make it work. But and then there's i of course the alternative argument, which is that the relational database is always the fastest, no matter what. which which I've also seen is true. we we've sort of validated that graph databases are so slow that performing Stupid not great queries in in SQL tend to be faster by making lots of extra queries, ten queries at the same time and stitching together the results, rather than using a graph database, which is sort of unfortunate in a lot of ways. But I I mean, you sort of echoed my point, which is that of it's not necessarily the languages which are the problem, but some of the semantics of them, but realistically how the data is being stored. So how do you deal with the fact that the data format is important? speaker-1 (30:31.67) relational database can store everything, yes. And everything that's in a relational database can also be a log. everything that's in a relational database you can also model as traces. I don't know that there's actually a one size fits all. This is the way that all data shall be modeled. speaker-0 (30:48.782) Back to the if you don't have the business problem then there's

then there's no there's no issue technological problem you have to solve. It's always a great one. which is why pushing back is important, right? if you tell someone that, it's gonna be really difficult to do that, they may go back and find out that they don't need to do it after all. speaker-1 (31:08.896) Y yes, yes. speaker-0 (31:10.978) You know, Eric, I think this may be a good moment to switch over to picks for the episode. So maybe you could share with the audience what you've brought. speaker-1 (31:19.24) yeah, I spent a a bunch of time in Japan. really enjoyed learning the language and stuff. And and learning the language to me is I think computer science and all that, it's also language learning in in my mind. I think of it as as similar. But the the big thing is there's a a saying that I love from Japanese. It's keizokwa chikaronari, which literally means persistence leads to strength or Continuous improvement kind of leads to strength. And it's this notion that if you do something five minutes a day, ten minutes a day, after a year, you will have done 3,650 minutes. After two years, after three years, you will have done it so repetitively that you will find that you are now capable of doing it. And and I think looking back on my life on everything, like there's a bunch of things that I did and then stopped doing. And maybe I was okay at it for a little time, but now and I can't do it at all. But it's that notion of like, it's not so much about getting to this end state and being like, okay, it's done, declare victory and we've won. It's about making sure that you're continuously investing and kind of driving towards something, whether that be large time commitments or small time commitments. I I think this notion and this philosophy, I just I I love it is that The things that are really important, if you just continuously work on them, even if it's not a lot, out outcomes will will come along with it. speaker-0 (32:53.442) The non research, the sort of feeling result was that it's about a thousand hours of anything before you become a master. And a five minutes a day would take you about 20 years to master anything. but then you can look at it that way. After 20 years, you can have mastered whatever it is that you want. I think the research actually comes from a book called The Formula, The Science of Success by Barabasi. And it was where they evaluated which in there, there was a lot of good research, but one of them was which Ol Olympic athletes become Olymp like successful, like actually get gold medals. And the ones that get to that level are usually not the ones with raw talent or ability, but the ones who go through significant practice and basically perseverance through through training to get there. So not that you're born with it, but that you actually went and did it. They weren't the best y younger on or when they started, but by the end that that is when they were still able to achieve I mean obviously there are some, you know, naturals that still take it, but by by and large, the most victorious ones are are the ones that just continued at it consistently. speaker-1 (34:00.748) couple it with the notion every time I look in the past, time always went really fast. Where when you look in the future, time like it if things seem much further out. Even that five minutes a day, after three years, looking in the past, you will be able to do something. Maybe you're not a master, you'll be able to do something and it will have taken like no time at all because it will have all gone extremely quickly. speaker-0 (34:24.482) I think especially in the Western world now, there's w it does feel like things are speeding up rather than what has historically been recommended, which is slow down and do things deliberately. And I think maybe maybe AI is to blame for that. Maybe cultural is to blame for that. we'll see. That's not related to my pick. my my pick is and only 'cause I just completely binged it was a television show called The Agency with Michael Fassbender. It has to do with the CIA operating out of London. I mean, obviously, complete fiction. and I don't know what it is. I I I don't I don't have love for stuff that is centered in the in the US. I I always I don't know what it is, but the CIA operating of London, you it's it honestly is really, really good. And I think it's produced by Showtime. speaker-1 (35:13.742) Okay. Nice. speaker-0 (35:16.206) So I don't know if you're into spy dramas. speaker-1 (35:19.584) Yeah, I'm I'm trying I'm trying to remember that name. There was there was a a show a while back about an ex spy based out of Yeah, Burn Notice, yes. speaker-0 (35:27.022) Burn Notice. Yes. Burn Notice, absolutely fantastic. as well. Yeah, it's it's definitely something you can also binge. It's it's great. I mean the nice thing about Burn Notice is that it was produced in a time before w shows got canceled before they were over. Michael Weston, not the actor, the the character. I don't remember the actor's name. he's in a bunch of things, but also very good. So speaker-1 (35:33.526) I lost a ton of that. speaker-0 (35:56.972) Yeah. I d I don't know what it is. and you know what, there's sort of like it's not like such negative dark comedy or or drama. Like the dark the it's not as bad as dark drama. I mean there's still stuff going on, but it doesn't feel like they just do it for the the kick, so speaker-1 (36:16.674) Like one of the nice things about Burn Notice was that it it was very episodic, but there was a little bit of a background thread running between them. And so you could enjoy one episode and and get completion, get some sort of closure, but still there was like the the allure or like what's going on behind the scenes, who's actually doing it. Is is the agency like you gotta watch the whole season to get the gear closure or is it one speaker-0 (36:42.162) Yeah, I think that unfortunately that's still where we're at. Don't worry. the second screen experience is is coming soon and every show that shows up on a streaming service will all be episodic so that people don't have to pay attention anymore. But the agency is still I think it's it's not totally epis not episodic like a single thread, but it you pretty you do need to sort of know what's going on from the previous episode. There still is a major thing there. There's two seasons right now and they are Sort of separated. So you could watch only one and feel like it's complete. Still, yeah, unfortunately, if you're just looking for single stuff, that's not it. I will say that the nice part about Burn Notice was that they also sort of educated you on what is reasonable in Spycraft. I don't know if that was accurate. I mean, I think there are lots of videos on YouTube basically saying, you know, these things they show in shows, you know, aren't real. I think Burn Notice does a better job. And I think some of the things they do in the agency also, like if you're paying attention, really help to clue you in on like how things actually work in practice rather than like today, rather than just doing stuff for the for the heck of it. Okay. So thank you, Eric, for coming on and discussing observability with us and the challenges of managing the database. speaker-1 (37:50.572) Nice. speaker-1 (38:00.123) Yeah, thank you. Thank you for having me. It was great discussion. speaker-0 (38:03.604) And thanks to the audience for tuning in for this week and hopefully we'll see everyone back again next week.