บทถอดความฉบับเต็มของ Elastic with Philipp Krenn

right now which has been coming up for a while. So there is a lot of development there and search is still developing and evolving. So coming back to the original question, initially it was very much about doing all of that or we are still doing all of that at index time, which gives you more power to start the work upfront and then make the search faster afterwards. But it also means you have to do that work upfront and you need to know how your data is structured and how to work with that. By now we also have something called runtime fields where we can extract more of that information later on. So if you say, "Oh we have this data but we have a new question and we need to extract some other thing from the data that we already have." You can do that at runtime now as well, which can be very powerful if you have stored something or you didn't think about the question, you didn't store it the right way or you maybe tried to expect it the wrong way and the data is a little broken and you just want to fix it at search time afterwards, that is possible. And the trade off is of course the search will be a bit slower because you need to do more at query time but you have more flexibility for later on. So we have traditionally been coming from that index time work and now we also have runtime fields which allow you to do more work at period time and you need to pick the right tradeoffs, which is generally where we always say it depends because there are so many ways to do things and you need to make the right trade offs from your scenarios. So if you know the structure of something and you know that you will be searching on that a lot, you should still do that at index time. But if you index wrong or if a new question comes up later on or you have something that you will only very rarely use, you might do that at run time and it might make your life easier. So we want to cater to those different needs. Paul: I think one of the end takeaways that I'm getting from this is bring us back to the basic principle that we're talking about a stack here. It's not one application and folks are coming in, they're like, "Oh do you still do this?" The reaction is not realizing that we're talking about a suite of services and applications that work together. And one thing you said that was interesting is, yeah, you have to choose, are you taking the apple, the orange, and how much of each to accomplish the goal that you need. And just to be clear, when we're talking about the indexes versus the runtime streams, what two layers of the stack? I know you mentioned, I just want to hit it one more time. Philip Krenn: So it is all in Elasticsearch. So Elasticsearch is really the data store and sometimes people are a bit confused and they're what is the database behind it? And there is no more database behind it. Elasticsearch stores that on disc, or actually the library we are using is Apache that is writing the data to disc and doing the actual search. But it is really

all there is. And then you have a rest full API that you query and then you can extract that information and you could seed in Kibana or you could use whatever programming language you want to either call the REST API directly or use one of our clients. So you have all the flexibility to build around that. Paul: Slightly off topic, but you just brought up one of the lower level technologies that powers your organization. So I think one question that I've heard thrown around a lot is, what is the difference between Lucene, Elasticsearch and OpenSearch? Philip Krenn: Yeah, so Lucene is the library written in Java that you can use. It's more than 20 years old by now. And you can use that in a Java program but it's a library that you need to use the right way and you have a lot of flexibility. But what most people want is they want to have a bit of a higher level service on top. So they want to talk, for example, to arrest API because they might not be using Java and might not want to integrate the library and that is what Elasticsearch is doing. So you have the library that writes the data and does the actual searching that is Lucene, and then you have Elasticsearch around that that does the distribution and replication of the data it provides security SL, the REST API. So it is the thing you interact with around it, but it's writing out that data or that the writing is part of Lucene and the core library and you could build your own applications around Lucene but unless you really want to build something very complex or specific that normally is not what you want to do as an end user. And then OpenSearch is a fork from a two year old code base of Elasticsearch. So it is similar in that regard but of course we have added a lot on top of that over the last two years. Paul: Yeah, if you think about how far development's come in the past two years, it's probably their substantial innovation that's happened in the past two years. Philip Krenn: Like runtime fields that we have been speaking about before, those were added since then so you don't have the similar concept there. Paul: What are beats? How do these fit into the stack that we've been poking at so far? Philip Krenn: Right. So maybe people have heard of log test, which is more like the ETL. So extract, transform, load tool, it's a bit a big hammer where you can get from data from lots of sources, you can transform it into various forms and then you can write it out to multiple destinations. The most common one is probably Elasticsearch. Beats are like lightweight agents or shippers. They are written in Go whereas Logstash is Ruby or J Ruby by now. So they are single purpose agents to forward information. So we have file beats, that's basically a thing that tails the lock file forwards it, metric beat to collect metrics, packet beat for network data, audit beat for security events. So we have various shippers. What we are switching to right now is, since we had all of these different beats

and then people had to in install multiple beats and manage those, we have some concept now called Elastic Agent, which is bungling them up together again. So you have one thing that you need to install and run and that thing will then collect all the right data for you. We also have a central management platform if you want, where you can in the elastic stack say, these are the integrations that I want to run on these agents and the pieces of information I want to collect that. There that is called Fleet. So we can manage a fleet of those agents running or you could do it the classic way by providing a Gamel configuration file to those in the so called standalone mode. But yeah, we have these different ways of getting data and we have gone from the big monolithic Logstash instance that is still there and needed for some scenarios but in many cases you don't to beats which are single purpose shippers and now we're at Agent which tries to combine those single purpose shippers and you can just enable the feature that you need and it still runs beats under the hood but you don't see this directly and you don't need to install or manage multiple binaries. Paul: And do those beats shift things straight into Elasticsearch or is Logstash more commonly? Philip Krenn: They can. So there are multiple outputs. So for example, Beats could forward the data directly into Elasticsearch and then there is a concept called ingest in Elasticsearch where we could also do some parsing and transformation. There is even a more extended concept of that in Logstash so the beats could forward to Logstash as well, or you could for example forward the data to Kafka Queue if you have that and want that. So there are multiple outputs that you could use and whatever is the right combination for you. Paul: When you were talking about these beats and you're saying, oh we have this and we have that and you ship it off, it gets transformed and loaded, and you were talking about metrics. Sounds a lot like things that we all use on a day to day. I'm thinking about Datadog and the Splunks of the world and things like this. So what do you think that Elasticsearch is palatable enough for folks who are in a small organization that say listen we really want the capability to do custom metrics, we really want the capability to get any logs but we don't want to pay an arm in a leg newborn child for Datadog. Is this something that you would recommend people to even look into? Because I think there is a scared factor that most developers can have when they're like, New Stack, oh and it's Java based under the hood and people go leave me out of it. Philip Krenn: Yeah, depending on how much you want to see. Since we also provided as a cloud service, you could just get Elasticsearch in Kibana as a cloud service and then you only see an API and you don't even know what it's running under the hood because I doubt people really know what Datadog is running under the hood. From what I know, also

it's zoo of tools that you would not want to manage yourself. So I wouldn't be concerned about that part because if you want to, you can totally run that on-prem but we do take care of that for you so we can provide that as a platform. And Elastic can generally came from that we were providing these tools as a platform and give you maximum flexibility and we are still doing that so you have these options. But over the last few years we have also invested a lot more into those solutions, and the three solutions we focus on is observability, security and enterprise search. Don't be scared by the enterprise and search, it's just billing a higher level search platform around it. But we have these three solutions, we do all of those. They all use the same platform but we have more of a solution or a digestible way to do things that you just activate. I'm not sure it's 100 percent Datadog like yet, but the idea is that it should be very simple to use and build on while still giving you the flexibility of a platform. Whereas Datadog has one way to do things and then you follow the Datadog way. Whereas if you say Oh we have this use case and the other use case and then we build something custom, all of those are covered by the Elasticsearch. Paul: I think one great sway of power that gives you is Datadog or other seams that we'll find out there. They really figure out how to charge. There's a lot of ways that you can cut that cake. And if you're able to have a custom implementation you can really say we want to just do a lot in these custom metrics. I'm sure you can lower that bill and you can still avoid having to run your own infrastructure if you can. This is the happy medium between them two. Philip Krenn: Yes. So our general approach is very different. So we are not agent based like Datadog, we are not volume based like Splunk. We're like, our pricing is basically or how we build stuff is on the Elasticsearch resources mostly. So for example, depending on the pricing, that's what you would pay for both in the cloud service or if you want to run it on on-prem with some of the commercial features or support. So it is how many and how large are those Elasticsearch or Kibana resources that you have. And we don't care about how many metrics series it is, how much volume it is, how many agents you run. It's just like, can this cluster handle all the data that you need to send and search in a reasonable timeframe and that is what you need. So it gives you more flexibility if you say like, oh, I want to have a lot of agents but they don't send that much data. It will not cost an arm and elect because you can just configure what is the right value balance or how long do I want to retain the data. So you have a lot more knobs to turn, which sometimes almost works against us because Datadog is so simple, just enable this and then it just does everything for you but you don't have the flexibility to tune cost or performance, it's the package. And I think sometimes that's almost scary or adds a bit of overhead but gives you more flexibility. So if you want to have that flexibility, I think we are giving you a lot more choices, but of course choice has a little overhead in terms of understanding what are your choices and then implementing them. Paul: Right. You could get into choice paralyzation. Philip Krenn: No I just want to say we try to have that a bit more automated with the solutions. So for example in observability that there is a template and it has a default retention of data and everything and how it moves. But you have the option to tune that down to saying, I want to retain my data only my logs for, I don't know, or traces for 48 hours because after that doesn't matter anymore and I don't need to keep it. Or you could say for compliance reasons, I need to keep this for a year. It's all possible and it will also drastically mean what you need in resources. You have the option but again, option always has its cost I think. Paul: And you mentioned you have an example in a template and stuff and I just want to quickly plug. If people want to go try this out, there's resources for them to do a cookie cutter, hello world, end to end seam setup where you can say, hey turn on a beat, put it in here to this end point, turn on Kibana, you can send this query and you can see the stuff's connected. Philip Krenn: Yeah. If you just want to see the end result, we have demo.elastic.co where it basically is Kibana and you can just see what's happening and you can play around with it. So this is the working setup. If you want to run this yourself, you can just download all of our products and run it or you can start a free trial of the cloud service where you get Elasticsearch, Kibana and both for enterprise search or tracing, you get the server side components as well. So you could just easily run that and then you could install agent for example or Beats to collect the right information then forward it to that cluster to do that. But yeah, we tried to easily show you what is possible and what you can get out of it. Paul: Awesome. And one more time, what was that link? If people want to go visit? Philip Krenn: It's demo.elastic.co. Paul: Perfect. Demo.elastic.co. Got it. Switching gears a little bit Philip if it's all right, at the beginning of the podcast we were talking about how a lot of problems can be boiled down to a search problem. And I love these types of questions because I feel like there's people out there that are saying, listen, if you reframe everything, it's ultimately this game theory or if you think about, and a lot of times it can change the way, even if you're not on board, it can change the way you think about the world a little bit. So I'm curious to pick your brain, why do you think anything can be a text search problem? What about math? That's not a text search problem. Philip Krenn: Right. So for us, search can be very structured or more unstructured, and very

structured could be, it's more a filter that just give me information from these users or these hosts, or it could be just everything within these geo bound or within a radius of geo point. So there is more structured search where it's a hard filter, it's more like a traditional database which is very black and white, yes and no, this data matches doesn't match. And then you can combine that with full text search or free text search problems, which is not so much about an exact match, but it's more about the relevancy. It's like, if you search for a concept then there might be a synonym or it might be singular, plural, or it might just be the adjective instead of the noun. So you're really searching more for a concept that's where search is coming from or where there might be, I don't know if somebody, a taxi system, there might be a permutation so they try to slightly change the input parameters so you're exact match wouldn't catch that. So there is a lot of flexibility in that and for us, we try to frame search as it can be structured and can be more unstructured free flowing, and that combination is very powerful to find what you want. And depending on the type of data, it will be a mix or a spectrum. For example, searching in Wikipedia to a large degree is very free text form, whereas security events for example might be highly structured because then you have the user doing the action and the host or the, I don't know, the Kubernetes name space in observability. So there can be a lot of structured pieces and we can do that hard filter on that, but then we can still go from there and say, it's not just hard filters but it's more the concepts around it or give me anything that is relevant to this problem. Paul: So it's really a method of problem solving we're talking about here is how can we hash up our end goal into a bunch of little questions that we can use as filters to get the information necessary. It's like playing scavenger hunt but you need to figure out what list to set forth with. Philip Krenn: Yeah. I think a lot of modern outages for example are often more a murder mystery where you need to find the suspect like what killed our system today, and we're trying to play into that. Yeah, it's very much about framing it in that way and then with the large amount of data that we have today, it's often very hard to just say just a hard filter or just subset of data. You really need the search capabilities to make all of the data accessible in a reasonable way. Paul: So you're mentioning some security threats a lot. I can really imagine having this wide indexing available. It can be huge for, oh what about this random thing? Is there a correlation here that we can sniff out? Who knows? I didn't tell it to index that but it's widely useful to the organization. So where is Elasticsearch right now foraying into the security and platform analysis world? Philip Krenn: Yeah, so security is a big area of investment since I think for a long time people were just using the classic elk for security analysis and then we saw the need to actually build more of a solution, and that it is from scene to so. We are adding more and more tools to that box to make it easier for a security analyst for example, to figure out what's going on. I always find it very interesting to see what our own team, for example, is doing. So when some widespread exploits hit, we collect a lot of the security data for a long time and then they look back like, did anybody try to exploit on our systems or do we see anything since we have, I don't know, a year or whatever of security events stored so we can then see even if something breaks weeks or month after it actually happened, we can see lock for J, lock for Shell, for example, did anybody run that against our services or did we have that problem in the past? So we see it internally a lot or recently I saw for example, a very clever approach and my colleagues are also using this because there's so many signals and there's so many low quality alerts potentially coming up or low severity alerts that what they have built now is that they are basically having that unique alerts with low quality or low relevancy or severity and bundled together. So some simple thing might not be relevant on its own, but if five signals in combination appear suddenly, then from a security perspective it might be much more relevant. So they basically have built alerting on alerting. So you have low severity alerts, but if five unique ones appear within a very short amount of time then it's the signal that something is happening. So for example, if one specific user account or host for example count those multiple ones of these, it's a much more interesting signal suddenly. And for that we need that wide search base that we can just freely slice and dice a large amount of data to actually find all of that. Paul: That's fascinating. That reminds me of this LSTM neural networks and how they learn on a sequence. Have you seen any use cases of people using Elasticsearch for deep learning? Because these are two disparate areas of focus. Philip Krenn: I think deep learning is sometimes a bit separate. I don't think I have a very good story or use case where people used it. It might be somewhere in the data collection layer and then to feed it and more net story. But I think that's a different application on top of it. And for example, what we are doing in our product is more like we have supervised, unsupervised machine learning to some degree. So unsupervised could be you have anomaly detection, it just learns what is happening, you have, this is the number of users on your system and those type of things or you could bring a custom model to it, but you would need to bring or develop that model elsewhere and you could bring it to our stack then. But we would not do the neural network itself. Paul: But you could. There is a service layer of plugin architecture here where you could bring

in your own indexing capabilities if you so wished. Philip Krenn: Yes. So you can bring models that you have developed outside with common tooling. For example, Python is very big on machine learning. You could develop a model there and then could bring that model to our stack to run it. Yes. Paul: And thinking about the security thing, there was a few years of my life where I actually worked as a backend engineer at Carbon Black and we used Elasticsearch to basically ingest everything that happens and it was incredible. You could feed it into this platform and say, "Hey, I want every single time the temp directory got accessed." We never told it to do that. It just had it somehow and it really opened up the world into at ad hoc security. Right now you see it coming in, you can change your index, you can do the query. It was really remarkable honestly. I think the security field for Elasticsearch is still being explored and it's really cool to see what's happening right now. Philip Krenn: We do hope so. And like we discussed, we are trying to add the higher level services on top, but a lot of people have just built that on their own in the classic elk way, which worked surprisingly well. And then I don't want to say we grudgingly followed to build a tool around it or a solution around it, but we at some point had to agree that, well maybe we can make everybody's lives around that easier and provide more tooling around it. But yeah, it has a lot of potential and we're trying to tap into that more and make it even easier. Paul: Well, I feel like the way these things usually go is, private sector, they see the opportunity, they build four different versions, the companies get sold for a bazillion million dollars and then the open source community slowly catches up and that takes over. We see the cycle happen time and time again. Security might be different though because you get the hand shaky thing going on, but I pray you guys keep doing it because it'd be awesome to have more open source tools and service layer tooling for normal people to use, even for their own stuff. Philip Krenn: Right. I come more from the ops and dev world, so security sometimes can feel a bit foreign because it's very different processes or approaches to some degree. Paul: Right, yeah, mostly from the DevOps side for me too. I know there's a lot of compliance and lawyers and it can get hairy on the security side. On DevOps, there has to be an amazing use case that we can talk about really quick for DevOps in terms of something that another scene couldn't do that maybe you're able to focus in on using Elasticsearch in your own custom beats or whatever it might be. Philip Krenn: In terms of observability, no? Paul: Yeah, In terms of his observability or self-healing systems or up the DevOps type of tree trunk. Philip Krenn: Right. I feel like self-healing is always a complicated topic because it's sometimes oversold strongly and I don't think that the humans will be

fully replaced. It's more like better tooling support will open new doors because you have all of the data and then you can find the relevant one. So one thing that we've been doing lately, and there's this term, AI ops, but for me it always stands for actionable insight because it's not AI solving your problem, but you want to have some insight that you can actually react on and to fix your systems. So for me, AI op stands for actionable insight and that's what we're trying to do. So for example, we have these higher level services built on top of the data that we have collected where you say there is the latency spiking or the errors are spiking. And then you could say, show me why. And then we basically look at the data and then we say, there is this strong correlation. It's like this one endpoint is slow or this one endpoint is throwing arrows or it's just users from this country or this specific browser, I don't know, a react application or whatever. So we basically take the data and then we look at one specific subset and say it strongly correlates with this problem. And that creates very good actionable insights because you suddenly have an automatic way to see what is the correlation of where the problem is coming from, and you can then figure out what is going on there. It doesn't replace the human, it's not necessarily self feeling. Maybe you could script something on top of that, but it has a good way and we have a calculation that says, there's a 95% correlation of this error spike with this specific endpoint or this one service or this user sub segment. And that allows you to then quickly figure out what is going on. And for me that's the actionable insight point that it's not AI that magically fixes what's going on, but it's a human support to show you what is relevant and then you can take the right steps. Paul: You're drinking from a fire hose when you're typically trying to fix something, you're in a panic mode. And at the end of the day, a lot of times you do need a human to sort of like, do you even want the system to do it yourself? That's another whole philosophical question we could get into. Philip Krenn: Yeah. I think there might be scenarios where this just works and, oh, you reboot the service and the restart fixed everything. Paul: That makes me uncomfortable. Nobody likes having that happen. You restart it just works and you're like, mm-hmm. Philip Krenn: Yeah, but at least if it gets you out of the thick for now. There could be something where you bump the service twice and if that doesn't resolve it then you wake up a human in the middle of the night. Maybe that works for some scenarios, but it might make it also much worse. For example, if you have data and you write it inconsistently or whatever and then bumping the service or it cannot keep up with writing the data and then you might lose data that's solely in memory or whatever. I'm sure there are scenarios where you could make it a lot worse by any automated

action, which doesn't mean that humans don't also sometimes make it worse with their actions but that's a different problem then. Yeah, I think where the space, for example, in observability is generally heading is that collecting the information, I don't want to say it's a solved problem, but it's like that is the baseline and just getting through the data is there and good, but where the real value or innovation is on these higher level service is to say, I see this correlation and to tell a human look at this correlation, this looks super weird or this is where you should start focusing on. Especially when you're woke up in the middle of the night, you don't want to look through a thousand events by hand and then figure out what might be the reason here, but you might want to get these starting points or this looks interesting or relevant, start here. And I think that's where generally the innovation in a lot of spaces, both observability but also security is coming from that you try to have these higher level toolings or support for humans there without replacing the human because I don't think that's really realistic. Paul: We're trying to increase productivity here. Philip Krenn: Right. Paul: At the end of the day. Philip Krenn: Yeah, I always say it's the support for humans because I don't think we'll be able to replace everybody by little bots anytime soon. Paul: Yeah, I don't think that's an uncommon or hot take. There's a lot of, I don't know, unique thought that needs to go into these actual insights like you're saying. I meet plenty of people that say we're going to get to a system that's going to heal itself, it's going to do it all. Philip Krenn: I don't know. Paul: I tend to lean more your boat as well. We need to learn from ourselves. We're an evolving body of developers and creators and what is actual is also going to change as time goes on. Philip Krenn: Yeah. For me it's like, every time I have to talk to a chat bot I'm like, no, this is not going to work out. Paul: Yeah, this is unacceptable. Philip Krenn: Yeah. This is just like, even if you use the right keywords it might still go off totally the wrong way and I'm like, for a chatbot, maybe if it goes south that then I'm just unhappy. But if you try to automatically fix the service, then you might make it much worse. Paul: If you're enjoying the podcast and you want to hear more about Elasticsearch or other open source creators, tune to Pod Rocket, we're available on Apple Podcasts, Spotify, and a bunch of other places and we hope to see you around. Let's move on to if people wanted to get involved with the community or if they wanted to see what was coming up next or what you specifically are working on Philip, what's coming? Philip Krenn: We have the history for us, everything is GitHub. So you can see the pool requests, you can see the GitHub issues where we discuss what is coming up next. We're always a bit cautious about roadmap because priorities shift or we find new opportunities or for a specific reason, depends on something else or could be done a better way and then we don't want to release something in a poor state. So we are very cautious about not promising specific features or dates too much because we just want to do the right thing and not be held to our promise. Also, I think our mentality to that is you win very little by promising something, you just get a lot of anger if you don't stand by your promises though you might have good reasons for not fulfilling those. So we're trying to avoid that, and we're very focused on shipping the right stuff as quickly as possible, which I think is one of the interesting shifts that we as a company have done that initially we were very focused on major releases and then a lot of features would come in a major release. But now for us, major releases are more like, these are the breaking changes the cleanup, but releasing features is very decoupled from that. So it might almost be hard to keep up with all our releases just because we are shipping so many features in each minor release. And for example, right now we're at 8.4, 8.5 will come out pretty soon, but even in the 7.X release, which was going on for two years, we shipped the ton of features over the years. So the roadmap is very dynamic and we're just trying to ship as fast as possible without being held to a roadmap that we set maybe a year in advance and then for specific reasons we might not follow through with that. Paul: Totally understandable. I feel like that's one of the best developer roadmaps you could give somebody, you know what? You gave honesty and you said, we're going to listen to the community. What's a better roadmap than that at the end of the day? Philip Krenn: Yeah, and it's very important to listen to the community. I just think that you also need to be a bit careful to not only follow what the community asks for because if you ask people around, I don't know, 1880 or something like that, what do they want? They would have probably said they want a faster horse. And if you just follow through with that, you would never have invented the car because you need to take another step. Or it's sometimes you need an unexpected change or something that maybe is not the next logical step. If you just try to follow the logical steps, you might miss out on these jumps of innovation. So sometimes there might be something that the community asks for, but we are looking at it and then say, well yeah, I can see that, but we see a chance for this jump and then we want to make this jump. I think there's a balance of following the logical path of what people are asking for and also being the driver of innovation to see what's moving you ahead further. Paul: Right on. Well I also notice we're already over time but Philip, thank you for coming on the podcast and hopefully educating some people about Elasticsearch who are maybe confused about, is it a stack? Is it a service? How do I get into it? Before we end, if people wanted to hear more from you. Philip Krenn: Yeah. Paul: Are you on Twitter? Are you on Medium? Where can we find more content? Philip Krenn: I am on Twitter. So if you take my last name K-R-E-N-N, and you rotate the letters by 13, wrote 13 for the ones who are very old, you end up at my Twitter handle, which is XERAA. Paul: That is the best Twitter handle explanation I've ever heard. Philip Krenn: Which is pretty unique. And I very intentionally, I'm not on medium. I have car.net as my domain because I'm very cautious about platforms coming and going and I want to end my content, but also the user tracking and everything myself. But yeah, I'm generally xeraa on GitHub, Twitter, the web and everywhere else. Paul: Awesome. Well yeah, thanks again and hope to have you back in the future where we can talk about some of the more exciting things that you and the team have been developing. Philip Krenn: Thanks a lot for having me. And yeah, if there are any questions, happy to get pinged on whatever medium you prefer. Yeah, thanks a lot for hosting.