WEBVTT
00:00:00.160 --> 00:00:06.320
Welcome to the Talos Takes Podcast, where we discuss Talos' latest research and security news.
00:00:06.559 --> 00:00:10.640
This podcast is for everyone, from the C-suite to the front lines.
00:00:13.679 --> 00:00:20.000
You're hunting through a data set, looking for the telltale signs of an adversary, but the logs feel hollow.
00:00:20.239 --> 00:00:25.120
The DNS query doesn't line up with the connection, and the process tree is missing its parent.
00:00:25.440 --> 00:00:27.760
It's the classic synthetic data problem.
00:00:28.079 --> 00:00:36.079
It looks like telemetry, but it doesn't behave like a real network, leaving your detection rules untested and your analysts frustrated.
00:00:36.719 --> 00:00:38.159
Welcome to Talestakes.
00:00:38.479 --> 00:00:50.000
I'm Amy Simonisi, and today David Bianco is joining me to talk about why current synthetic data like LANL and OpTC are failing our security teams.
00:00:50.560 --> 00:00:54.799
We discuss how his new open source project, EvidenceForge, is changing the game.
00:00:55.119 --> 00:01:05.040
EvidenceForge creates logs that both look real and tell a coherent causal story that gives your analysts a better foundation to sharpen their skills.
00:01:05.760 --> 00:01:07.359
David, welcome to the podcast.
00:01:07.920 --> 00:01:09.120
Hi, thanks for having me.
00:01:09.840 --> 00:01:10.400
Absolutely.
00:01:10.560 --> 00:01:11.519
Happy to have you back.
00:01:12.239 --> 00:01:26.799
David, I'm sure a lot of practitioners have been there, you know, trying to test a new detection rule and realizing that the only data that they have is either stale or super anonymized or just wrong.
00:01:27.040 --> 00:01:32.079
Um, was there a specific moment that led you to build EvidenceForge?
00:01:32.640 --> 00:01:33.920
Yeah, there actually was.
00:01:34.000 --> 00:01:35.519
And it's it's kind of funny.
00:01:35.599 --> 00:01:44.879
So I I have uh another project, which is also an open source project that we released recently, a little bit earlier this year, called the Peak Threat Hunting Assistant.
00:01:45.040 --> 00:01:53.760
The Peak Threat Hunting Assistant is an agentic experiment to help threat hunters research and plan for their threat hunts.
00:01:54.000 --> 00:01:54.319
Yes.
00:01:54.480 --> 00:02:01.200
If this is sounding familiar to you listeners, we did an episode with David a few weeks back about the assistant.
00:02:01.439 --> 00:02:05.280
I will put a link to that and the blog in the show notes as well.
00:02:05.760 --> 00:02:19.919
Part of the planning in the Peak framework is you have to kind of understand what data that you have available and you check it to make sure that the types of things you're going to hunt for will actually show up in that data.
00:02:20.159 --> 00:02:24.960
And I was using some big data sets that had been published.
00:02:25.120 --> 00:02:29.840
You mentioned the OpTC, the operationally transparent cyber data.
00:02:29.919 --> 00:02:33.599
I forget what it stands for, but it's a fairly well-known data set.
00:02:33.840 --> 00:02:35.039
And it's large.
00:02:35.120 --> 00:02:38.719
I mean, it's like four terabytes of data or something like that.
00:02:38.879 --> 00:02:46.400
And so I had ingested it into our Splunk instance when I was dealing with testing enterprise scale data.
00:02:46.560 --> 00:02:48.800
Does this thing work against the enterprise scale data?
00:02:48.960 --> 00:02:56.000
And it did work, but for development purposes, I needed smaller data sets that I could iterate through more quickly.
00:02:56.159 --> 00:03:04.319
And I realized that there were not very many good actual publicly released data sets that would fit exactly what I wanted.
00:03:04.479 --> 00:03:14.639
So I just decided to look into creating, or I actually I decided to look into the state of synthetic data to see if there were good synthetic data generators around.
00:03:14.800 --> 00:03:24.080
There are a number of them that are pretty good that only produce one kind of log, and they produce mostly random formats or random fields.
00:03:24.240 --> 00:03:31.120
So it looks good if you look at a particular event, but it doesn't really tell a story.
00:03:31.199 --> 00:03:40.560
They're more for, hey, can I ingest all this data at scale, like, you know, just testing to see if my SIM pipelines will keep up or something like that.
00:03:40.800 --> 00:04:06.960
But they didn't really do what I needed for threat hunting, which was have the same kind of activity show up in different logs and supporting pivoting between the logs, and specifically, not even most of them didn't really have the ability to have both malicious uh you know activity going on as well as realistically generated user background noise.
00:04:07.199 --> 00:04:11.759
And so I realized that I had my next project in front of me.
00:04:12.240 --> 00:04:12.639
Yeah.
00:04:12.960 --> 00:04:23.120
You talk about in your log, which I'll put a link to in the episode notes, that most synthetic log generators are a collection of independent emitters.
00:04:23.279 --> 00:04:28.319
And this is kind of what you were just referring to, where each one knows how to produce its own format.
00:04:28.399 --> 00:04:32.480
Yeah, it doesn't share state with others, but evidence forge is different.
00:04:32.639 --> 00:04:35.680
So can you explain the model that it uses?
00:04:35.839 --> 00:04:42.160
You know, how does that change the way that logs relate to each other and how does it make a difference for the person on the other end?
00:04:42.800 --> 00:05:01.680
I think the biggest thing is the design started with knowing that we needed to have both the simulated malicious activity and the simulated user background noise, and that we were going to be generating multiple types of uh evidence.
00:05:01.839 --> 00:05:11.680
So you could get Windows event logs for uh several different event IDs, but also Windows Sysmon for also several event IDs.
00:05:11.920 --> 00:05:20.160
Um, not all not every single one, of course, but uh the most common that you wouldn't use for threat hunting various things.
00:05:20.240 --> 00:05:30.720
Uh but also like uh Zeek sensor logs, Snort IDS, uh Cisco ASA firewalls, Linux syslogs, all you know, it's a bunch of different kinds of ones.
00:05:30.879 --> 00:05:47.040
Uh and then given that I knew that I needed to make all those, I realized that it was going to be difficult if I approached it as a log generator, like, okay, I'm gonna do the syslogs now, and then I'm gonna do the the Windows event logs and just as all separate.
00:05:47.199 --> 00:06:15.839
So we we built this world model where you have an event that happens, and the event that happens is kind of like an abstract view of that event, and then it attaches individual contexts to that event, depending on which log sources would naturally see that, and then each of the contexts shares its like core of information with that event.
00:06:16.000 --> 00:06:40.000
So if you have, I don't know, like a network connection, you will see it in Zeek connection logs, but you probably will also see it in our simulated EDR, and it will have both this, you know, all the same uh ports and IP addresses and protocols on both sides because they're all built from the same record.
00:06:40.240 --> 00:06:49.839
No matter which log source it appears in, it'll be format correct for that log source, but the details of the fields will match up with the other appearances.
00:06:50.160 --> 00:06:51.040
Got it.
00:06:51.279 --> 00:06:57.360
One of the most interesting parts of this blog is your focus on causality.
00:06:57.519 --> 00:07:05.040
Um, you know, the idea that a logon shouldn't just exist, but it should have the Kerberos exchange that preceded it.
00:07:05.199 --> 00:07:09.759
Can you speak to that a little bit, maybe with regard to creating the kill chain?
00:07:10.399 --> 00:07:16.000
It's half of the evidence forge promise that you're gonna get some coherent story.
00:07:16.160 --> 00:07:25.040
That's the half that that fulfills is if the same event shows up in multiple logs, like different viewpoints, they will agree with each other.
00:07:25.279 --> 00:07:30.720
And the other part of it, though, is that's that's one event, but now look at all the events.
00:07:31.439 --> 00:07:36.160
An attack or some kind of malicious activity isn't just one event, right?
00:07:36.319 --> 00:07:42.079
It's usually a life cycle, like a whole cyber kill chain kind of thing.
00:07:42.240 --> 00:07:50.639
And so the other part of it that EvidenceForge provides is this AI-assisted creation of the scenario.
00:07:50.959 --> 00:08:11.839
The scenario being what the environment that you're simulating is, like what kind of organization, what is the technology stack, what are the computers and the users and the files or data involved, what is the simulated internet environment that it's in, where are the sensors?
00:08:11.920 --> 00:08:13.120
There's a whole bunch of things.
00:08:13.279 --> 00:08:18.480
You don't have to really remember all these because the AI assistant kind of guides you through it.
00:08:18.639 --> 00:08:32.240
It'll give you like an interview, like we're doing actually, and it'll ask you questions like, hey, what would you like to do with respect to, you know, what kind of sensors do you want, or what visibility would you like them to have, or or anything like that.
00:08:32.320 --> 00:08:35.200
Uh it will it can make things up if you want it to.
00:08:35.279 --> 00:08:38.720
So you don't have to be specifying every single thing.
00:08:38.879 --> 00:08:51.600
You could literally say, hey, I would like to model a phishing attack in a branch office of a financial institution, and it will do the right thing and ask you, like, here's what I'm have in mind.
00:08:51.759 --> 00:08:53.120
What would you like to correct?
00:08:53.279 --> 00:08:59.919
Or should we go ahead and write out this scenario file, which is uh just a config file in like YAML format?
00:09:00.159 --> 00:09:10.720
The nice thing about that is you have a file, the AI helps you create the config file, but when it comes to generating all the data, it's a deterministic Python script.
00:09:10.799 --> 00:09:15.759
So actually, the same config file will give you the same output every time you run it.
00:09:16.000 --> 00:09:22.720
So it's it's random, but it's seeded, and the seed is in the config file, so it's reproducible.
00:09:22.879 --> 00:09:26.320
Like I could give you the config file and you could reproduce the same data set.
00:09:26.639 --> 00:09:27.440
That's so sick.
00:09:27.600 --> 00:09:31.039
Um, so why keep the AI out of the generation phase?
00:09:31.200 --> 00:09:37.120
Um, like what are what are the are there any risks letting an LLM write the logs directly?
00:09:37.519 --> 00:09:46.240
The main two really are first of all, there's no way the the LLM is going to do the job correctly.
00:09:46.399 --> 00:09:47.919
It's just too large.
00:09:48.240 --> 00:09:57.519
Um like we can generate liter gigabytes, several gigabytes of of log data, and that would just overfill your your context window.
00:09:57.679 --> 00:10:06.879
And whatever you said at the beginning of the scenario, it would have totally forgotten by the time you got like 10 minutes into the scenario, let alone by the end of it.
00:10:07.120 --> 00:10:10.399
And it wouldn't look coherent, wouldn't look like a good story.
00:10:10.559 --> 00:10:13.279
But even if you could do that, it's just an inefficient use.
00:10:13.360 --> 00:10:17.679
It's slower, it costs a lot of money uh because of all the tokens.
00:10:17.840 --> 00:10:18.159
Yes.
00:10:18.320 --> 00:10:19.600
So it's just terrible.
00:10:19.679 --> 00:10:34.159
So, but we targeted AI for the piece that AI is good at, and that is helping you create the scenario so it can either know or even do research on like how does this typically work?
00:10:34.320 --> 00:10:42.159
Like if you have this kind of scenario, this malicious attack that you want to model, what are the typical phases of that?
00:10:42.399 --> 00:10:43.759
You don't have to know.
00:10:44.000 --> 00:10:50.879
If you do know, great, you can tell it, but it'll figure it out and ask you, hey, did I get this right?
00:10:51.039 --> 00:10:54.240
So the AI is really good at that and it keeps you on track.
00:10:54.399 --> 00:11:12.720
One of the key safeguards that we have is built-in automatic validation to make sure the AI produced a good scenario file and with good information about if it's not a valid scenario file, what was wrong with it, so that the AI can self-correct itself.
00:11:12.960 --> 00:11:24.399
But you can generate the scenario for like pennies and it'll take minutes rather than thousands of dollars of tokens, at which at which point just pay a red team.
00:11:24.639 --> 00:11:24.960
Yeah.
00:11:25.120 --> 00:11:25.440
Yeah.
00:11:26.000 --> 00:11:32.000
So this synthetic data is purposefully not perfect.
00:11:32.240 --> 00:11:40.080
With EvidenceForge, you built in a way to simulate gaps in network visibility, like missing spanner tap data.
00:11:40.320 --> 00:11:43.759
Why is it important for this data to be imperfect?
00:11:44.000 --> 00:11:48.080
I I will back up for just a second and say there's two kinds of imperfection.
00:11:48.320 --> 00:11:54.240
One is it's a simulation, so we can never simulate ever a real network perfectly, right?
00:11:54.480 --> 00:11:54.639
Right.
00:11:54.799 --> 00:11:56.480
Like there will be mistakes.
00:11:56.720 --> 00:11:59.279
We are explicitly not trying to make perfect data.
00:11:59.360 --> 00:12:07.600
We're trying to make data that's good enough for the purpose, but an experienced analyst looking closely at it would probably still tell that it was synthetic data.
00:12:07.759 --> 00:12:14.399
But uh good enough for training, good enough for detection engineering, and and maybe a little bit more than just good enough, right?
00:12:14.559 --> 00:12:21.120
But the the part I think you're talking about is I have this built-in evaluation loop that helps me.
00:12:21.279 --> 00:12:35.919
I on the development side, I also have the AI help me evaluate the quality of the generated data because it's very hard to do uh when you're generating several gigabytes of data on every iteration to see if you fix the problem or whatever.
00:12:36.080 --> 00:12:42.720
So I have the AI look at each iteration of the data from four lenses.
00:12:42.799 --> 00:12:44.960
So, like four different agents, really.
00:12:45.120 --> 00:12:53.600
Um, one is a network, one is a host forensics, one is a detection engineer, and one is an experienced threat hunter.
00:12:54.000 --> 00:13:01.039
One thing that was surprising to me at the beginning of this, I set these up and they worked for a while.
00:13:01.120 --> 00:13:10.480
And every time it goes through, it would point out like the things that it found were the strongest tells that it was synthetic data, and then I we could fix them.
00:13:10.720 --> 00:13:22.000
But uh, it eventually got to the point where one of the strongest tells coming over and over again from most of the reviewers was you have every event.
00:13:25.360 --> 00:13:26.639
And that's unrealistic.
00:13:26.879 --> 00:13:28.960
You're not, yeah, you're not missing any.
00:13:29.120 --> 00:13:32.399
And I was like, well, yeah, we're simulating uh something.
00:13:32.559 --> 00:13:48.879
I would like to get all the data, but they kept coming back and saying this is clearly synthetic because in the real world you have things that drop pipelines that don't work, they work 99.9%, maybe, or uh, but not a hundred percent.
00:13:49.120 --> 00:13:57.279
And so, yeah, I had a a model in there where you could optionally tell it, yeah, I I want to be smart about how I drop.
00:13:57.679 --> 00:14:08.480
I don't necessarily just drop random events, although you could, but you know, you could drop like a few minutes of data out of one of the logs or something like that.
00:14:08.559 --> 00:14:26.720
Uh it you basically it's configurable via profiles uh and and it ships with uh default not doing anything like that, but you can turn it on like the the typical enterprise uh protocol and it would it would fix that too perfect data a little bit.
00:14:26.960 --> 00:14:27.600
Yeah.
00:14:28.000 --> 00:14:29.200
I was really shocked.
00:14:29.279 --> 00:14:30.879
I was like, at first it was a compliment.
00:14:30.960 --> 00:14:36.080
It's like, oh, the the AI says my uh my program's data is too good.
00:14:36.320 --> 00:14:38.320
Yeah, I don't know how to take that.
00:14:39.039 --> 00:14:42.879
Yeah, I mean, I guess take it as a compliment, but that's really funny.
00:14:43.200 --> 00:14:50.879
Okay, um, so I'm going to get even more into the nitty-gritty because I'm a nerd and I found this really interesting.
00:14:51.039 --> 00:14:55.360
Uh, in the blog, you talk about Hawk's processes for background noise.
00:14:55.519 --> 00:15:05.039
Um, for people who aren't data scientists, how does that make background noise feel more human compared to just random traffic?
00:15:05.360 --> 00:15:07.679
Yeah, uh, I'm not data scientists either.
00:15:07.840 --> 00:15:10.399
I just pretend to be one as a threat hunter.
00:15:10.720 --> 00:15:13.360
There's I always tell people there's a big overlap.
00:15:13.919 --> 00:15:15.840
Data science and threat hunting.
00:15:16.000 --> 00:15:19.600
Threat hunting is almost like just data science applied to cybersecurity.
00:15:19.759 --> 00:15:21.679
So yeah, there's a there's a lot of overlap.
00:15:21.840 --> 00:15:28.799
The it's not that important that you know like exactly what the the the Hawks algorithm is or how it works or whatever.
00:15:28.960 --> 00:15:33.840
But the the basic idea is we have certain things that we're simulating.
00:15:34.000 --> 00:15:43.360
On the computers that we're simulating, they usually have things that are driven only by the computer, like automation, schedule tasks, or whatever.
00:15:43.519 --> 00:15:44.960
They recur on a schedule.
00:15:45.120 --> 00:15:49.679
So, you know, five minutes after the hour, every hour, or at 8 a.m.
00:15:49.840 --> 00:15:51.919
on Monday mornings or whatever it is, right?
00:15:52.240 --> 00:15:56.080
And they're kind of metronomic unless you lose events.
00:15:56.399 --> 00:15:58.720
But the users are not like that.
00:15:58.879 --> 00:15:59.120
Right.
00:15:59.360 --> 00:16:18.559
They come in in the morning, they let's let's just say you're typing commands at the like the Linux command line, or maybe you're clicking on commands or clicking on icons in the in the Windows environment, but you're doing some things, launching applications, sending emails, browsing the web or whatever.
00:16:18.720 --> 00:16:20.639
You don't do that like a metronome.
00:16:20.799 --> 00:16:21.039
Right.
00:16:21.279 --> 00:16:22.480
It's in burst, basically.
00:16:22.559 --> 00:16:31.840
I'm gonna type this command, I'm gonna type this command, I'm gonna type this command, but in the between I have to review the outputs and see what I need to do next to make a judgment.
00:16:31.919 --> 00:16:35.440
It takes a little bit of time, and it's not always the same amount of time.
00:16:35.759 --> 00:16:43.200
Um, but also it's bursty because I might be doing things and then get distracted, get some coffee for a little while.
00:16:43.279 --> 00:16:47.039
So the the Hawks algorithm is just trying to model that.
00:16:47.200 --> 00:16:57.600
It's the short form of it is every action that a user takes makes it slightly more likely that it will take another action shortly after.
00:16:58.320 --> 00:16:59.919
So it's making a burst.
00:17:00.080 --> 00:17:00.559
That's it.
00:17:00.720 --> 00:17:08.559
I mean, it doesn't snowball to be like, oh, eventually the user has to take all the actions, you know, without a break.
00:17:08.880 --> 00:17:13.839
But it's just a way of trying to to model that burstiness.
00:17:14.079 --> 00:17:14.960
It's not perfect.
00:17:15.119 --> 00:17:25.119
Again, we you can't like model uh the whole thing perfectly, but it's good enough uh that it doesn't obviously look synthetic just because of the timing.
00:17:25.440 --> 00:17:34.240
Yeah, and it's I it kind of reminds me of um a blog article that we put out several months back about good enough fuzzing.
00:17:34.480 --> 00:17:42.160
Um and just like, you know, like you need to know what your intention is when you're creating these things or when you're using them.
00:17:42.319 --> 00:17:49.920
Like it it doesn't have to be perfect for your use case in order for it to be a successful element of what you're doing.
00:17:50.400 --> 00:18:01.920
Um, so to close out, if someone is listening to this and wants to start using EvidenceForge today or tomorrow, what do you think would be a good place for their first experiment?
00:18:02.079 --> 00:18:03.119
Where should they start it?
00:18:03.440 --> 00:18:07.359
It's all available on GitHub, so they could just check out the GitHub repo.
00:18:07.680 --> 00:18:17.119
And in fact, there is a quick start uh in the README file there that we have published a scenario that is in the repo.
00:18:17.279 --> 00:18:31.680
And you could just ask your favorite AI, Claude, or Codex, it works with either of them, to say, hey, please generate the this example scenario and just see like what the process is like.
00:18:32.160 --> 00:18:39.920
Or you can literally just say, I want to generate an evidence forge scenario, and the AI will lead you through it.
00:18:40.160 --> 00:18:45.519
So it's really easy to get started making the scenarios.
00:18:45.680 --> 00:18:49.279
There's hopefully very little learning curve if I did my job right.
00:18:49.440 --> 00:18:55.279
There might you might have to install the skills and and whatever, but I tried to make it really easy.
00:18:55.599 --> 00:18:58.160
I really encourage everyone to go and check it out.
00:18:58.319 --> 00:18:59.839
David, thanks so much for joining.
00:19:00.079 --> 00:19:01.279
Hey, thanks for having me.
00:19:01.759 --> 00:19:04.559
It's always fun to talk about my projects.
00:19:05.519 --> 00:19:07.359
I'm a typical researcher.
00:19:07.759 --> 00:19:09.200
Same, same, same.
00:19:09.279 --> 00:19:11.200
Yeah, me with my knitting all the time.
00:19:11.279 --> 00:19:13.519
Just like, let me show it off to you.
00:19:13.920 --> 00:19:16.079
Thanks for tuning in to Talos Takes.
00:19:16.240 --> 00:19:18.480
We'll see you in two weeks with a new episode.
00:19:18.559 --> 00:19:20.960
And until then, stay safe out there.
00:00:00.160 --> 00:00:06.320
Welcome to the Talos Takes Podcast, where we discuss Talos' latest research and security news.
00:00:06.559 --> 00:00:10.640
This podcast is for everyone, from the C-suite to the front lines.
00:00:13.679 --> 00:00:20.000
You're hunting through a data set, looking for the telltale signs of an adversary, but the logs feel hollow.
00:00:20.239 --> 00:00:25.120
The DNS query doesn't line up with the connection, and the process tree is missing its parent.
00:00:25.440 --> 00:00:27.760
It's the classic synthetic data problem.
00:00:28.079 --> 00:00:36.079
It looks like telemetry, but it doesn't behave like a real network, leaving your detection rules untested and your analysts frustrated.
00:00:36.719 --> 00:00:38.159
Welcome to Talestakes.
00:00:38.479 --> 00:00:50.000
I'm Amy Simonisi, and today David Bianco is joining me to talk about why current synthetic data like LANL and OpTC are failing our security teams.
00:00:50.560 --> 00:00:54.799
We discuss how his new open source project, EvidenceForge, is changing the game.
00:00:55.119 --> 00:01:05.040
EvidenceForge creates logs that both look real and tell a coherent causal story that gives your analysts a better foundation to sharpen their skills.
00:01:05.760 --> 00:01:07.359
David, welcome to the podcast.
00:01:07.920 --> 00:01:09.120
Hi, thanks for having me.
00:01:09.840 --> 00:01:10.400
Absolutely.
00:01:10.560 --> 00:01:11.519
Happy to have you back.
00:01:12.239 --> 00:01:26.799
David, I'm sure a lot of practitioners have been there, you know, trying to test a new detection rule and realizing that the only data that they have is either stale or super anonymized or just wrong.
00:01:27.040 --> 00:01:32.079
Um, was there a specific moment that led you to build EvidenceForge?
00:01:32.640 --> 00:01:33.920
Yeah, there actually was.
00:01:34.000 --> 00:01:35.519
And it's it's kind of funny.
00:01:35.599 --> 00:01:44.879
So I I have uh another project, which is also an open source project that we released recently, a little bit earlier this year, called the Peak Threat Hunting Assistant.
00:01:45.040 --> 00:01:53.760
The Peak Threat Hunting Assistant is an agentic experiment to help threat hunters research and plan for their threat hunts.
00:01:54.000 --> 00:01:54.319
Yes.
00:01:54.480 --> 00:02:01.200
If this is sounding familiar to you listeners, we did an episode with David a few weeks back about the assistant.
00:02:01.439 --> 00:02:05.280
I will put a link to that and the blog in the show notes as well.
00:02:05.760 --> 00:02:19.919
Part of the planning in the Peak framework is you have to kind of understand what data that you have available and you check it to make sure that the types of things you're going to hunt for will actually show up in that data.
00:02:20.159 --> 00:02:24.960
And I was using some big data sets that had been published.
00:02:25.120 --> 00:02:29.840
You mentioned the OpTC, the operationally transparent cyber data.
00:02:29.919 --> 00:02:33.599
I forget what it stands for, but it's a fairly well-known data set.
00:02:33.840 --> 00:02:35.039
And it's large.
00:02:35.120 --> 00:02:38.719
I mean, it's like four terabytes of data or something like that.
00:02:38.879 --> 00:02:46.400
And so I had ingested it into our Splunk instance when I was dealing with testing enterprise scale data.
00:02:46.560 --> 00:02:48.800
Does this thing work against the enterprise scale data?
00:02:48.960 --> 00:02:56.000
And it did work, but for development purposes, I needed smaller data sets that I could iterate through more quickly.
00:02:56.159 --> 00:03:04.319
And I realized that there were not very many good actual publicly released data sets that would fit exactly what I wanted.
00:03:04.479 --> 00:03:14.639
So I just decided to look into creating, or I actually I decided to look into the state of synthetic data to see if there were good synthetic data generators around.
00:03:14.800 --> 00:03:24.080
There are a number of them that are pretty good that only produce one kind of log, and they produce mostly random formats or random fields.
00:03:24.240 --> 00:03:31.120
So it looks good if you look at a particular event, but it doesn't really tell a story.
00:03:31.199 --> 00:03:40.560
They're more for, hey, can I ingest all this data at scale, like, you know, just testing to see if my SIM pipelines will keep up or something like that.
00:03:40.800 --> 00:04:06.960
But they didn't really do what I needed for threat hunting, which was have the same kind of activity show up in different logs and supporting pivoting between the logs, and specifically, not even most of them didn't really have the ability to have both malicious uh you know activity going on as well as realistically generated user background noise.
00:04:07.199 --> 00:04:11.759
And so I realized that I had my next project in front of me.
00:04:12.240 --> 00:04:12.639
Yeah.
00:04:12.960 --> 00:04:23.120
You talk about in your log, which I'll put a link to in the episode notes, that most synthetic log generators are a collection of independent emitters.
00:04:23.279 --> 00:04:28.319
And this is kind of what you were just referring to, where each one knows how to produce its own format.
00:04:28.399 --> 00:04:32.480
Yeah, it doesn't share state with others, but evidence forge is different.
00:04:32.639 --> 00:04:35.680
So can you explain the model that it uses?
00:04:35.839 --> 00:04:42.160
You know, how does that change the way that logs relate to each other and how does it make a difference for the person on the other end?
00:04:42.800 --> 00:05:01.680
I think the biggest thing is the design started with knowing that we needed to have both the simulated malicious activity and the simulated user background noise, and that we were going to be generating multiple types of uh evidence.
00:05:01.839 --> 00:05:11.680
So you could get Windows event logs for uh several different event IDs, but also Windows Sysmon for also several event IDs.
00:05:11.920 --> 00:05:20.160
Um, not all not every single one, of course, but uh the most common that you wouldn't use for threat hunting various things.
00:05:20.240 --> 00:05:30.720
Uh but also like uh Zeek sensor logs, Snort IDS, uh Cisco ASA firewalls, Linux syslogs, all you know, it's a bunch of different kinds of ones.
00:05:30.879 --> 00:05:47.040
Uh and then given that I knew that I needed to make all those, I realized that it was going to be difficult if I approached it as a log generator, like, okay, I'm gonna do the syslogs now, and then I'm gonna do the the Windows event logs and just as all separate.
00:05:47.199 --> 00:06:15.839
So we we built this world model where you have an event that happens, and the event that happens is kind of like an abstract view of that event, and then it attaches individual contexts to that event, depending on which log sources would naturally see that, and then each of the contexts shares its like core of information with that event.
00:06:16.000 --> 00:06:40.000
So if you have, I don't know, like a network connection, you will see it in Zeek connection logs, but you probably will also see it in our simulated EDR, and it will have both this, you know, all the same uh ports and IP addresses and protocols on both sides because they're all built from the same record.
00:06:40.240 --> 00:06:49.839
No matter which log source it appears in, it'll be format correct for that log source, but the details of the fields will match up with the other appearances.
00:06:50.160 --> 00:06:51.040
Got it.
00:06:51.279 --> 00:06:57.360
One of the most interesting parts of this blog is your focus on causality.
00:06:57.519 --> 00:07:05.040
Um, you know, the idea that a logon shouldn't just exist, but it should have the Kerberos exchange that preceded it.
00:07:05.199 --> 00:07:09.759
Can you speak to that a little bit, maybe with regard to creating the kill chain?
00:07:10.399 --> 00:07:16.000
It's half of the evidence forge promise that you're gonna get some coherent story.
00:07:16.160 --> 00:07:25.040
That's the half that that fulfills is if the same event shows up in multiple logs, like different viewpoints, they will agree with each other.
00:07:25.279 --> 00:07:30.720
And the other part of it, though, is that's that's one event, but now look at all the events.
00:07:31.439 --> 00:07:36.160
An attack or some kind of malicious activity isn't just one event, right?
00:07:36.319 --> 00:07:42.079
It's usually a life cycle, like a whole cyber kill chain kind of thing.
00:07:42.240 --> 00:07:50.639
And so the other part of it that EvidenceForge provides is this AI-assisted creation of the scenario.
00:07:50.959 --> 00:08:11.839
The scenario being what the environment that you're simulating is, like what kind of organization, what is the technology stack, what are the computers and the users and the files or data involved, what is the simulated internet environment that it's in, where are the sensors?
00:08:11.920 --> 00:08:13.120
There's a whole bunch of things.
00:08:13.279 --> 00:08:18.480
You don't have to really remember all these because the AI assistant kind of guides you through it.
00:08:18.639 --> 00:08:32.240
It'll give you like an interview, like we're doing actually, and it'll ask you questions like, hey, what would you like to do with respect to, you know, what kind of sensors do you want, or what visibility would you like them to have, or or anything like that.
00:08:32.320 --> 00:08:35.200
Uh it will it can make things up if you want it to.
00:08:35.279 --> 00:08:38.720
So you don't have to be specifying every single thing.
00:08:38.879 --> 00:08:51.600
You could literally say, hey, I would like to model a phishing attack in a branch office of a financial institution, and it will do the right thing and ask you, like, here's what I'm have in mind.
00:08:51.759 --> 00:08:53.120
What would you like to correct?
00:08:53.279 --> 00:08:59.919
Or should we go ahead and write out this scenario file, which is uh just a config file in like YAML format?
00:09:00.159 --> 00:09:10.720
The nice thing about that is you have a file, the AI helps you create the config file, but when it comes to generating all the data, it's a deterministic Python script.
00:09:10.799 --> 00:09:15.759
So actually, the same config file will give you the same output every time you run it.
00:09:16.000 --> 00:09:22.720
So it's it's random, but it's seeded, and the seed is in the config file, so it's reproducible.
00:09:22.879 --> 00:09:26.320
Like I could give you the config file and you could reproduce the same data set.
00:09:26.639 --> 00:09:27.440
That's so sick.
00:09:27.600 --> 00:09:31.039
Um, so why keep the AI out of the generation phase?
00:09:31.200 --> 00:09:37.120
Um, like what are what are the are there any risks letting an LLM write the logs directly?
00:09:37.519 --> 00:09:46.240
The main two really are first of all, there's no way the the LLM is going to do the job correctly.
00:09:46.399 --> 00:09:47.919
It's just too large.
00:09:48.240 --> 00:09:57.519
Um like we can generate liter gigabytes, several gigabytes of of log data, and that would just overfill your your context window.
00:09:57.679 --> 00:10:06.879
And whatever you said at the beginning of the scenario, it would have totally forgotten by the time you got like 10 minutes into the scenario, let alone by the end of it.
00:10:07.120 --> 00:10:10.399
And it wouldn't look coherent, wouldn't look like a good story.
00:10:10.559 --> 00:10:13.279
But even if you could do that, it's just an inefficient use.
00:10:13.360 --> 00:10:17.679
It's slower, it costs a lot of money uh because of all the tokens.
00:10:17.840 --> 00:10:18.159
Yes.
00:10:18.320 --> 00:10:19.600
So it's just terrible.
00:10:19.679 --> 00:10:34.159
So, but we targeted AI for the piece that AI is good at, and that is helping you create the scenario so it can either know or even do research on like how does this typically work?
00:10:34.320 --> 00:10:42.159
Like if you have this kind of scenario, this malicious attack that you want to model, what are the typical phases of that?
00:10:42.399 --> 00:10:43.759
You don't have to know.
00:10:44.000 --> 00:10:50.879
If you do know, great, you can tell it, but it'll figure it out and ask you, hey, did I get this right?
00:10:51.039 --> 00:10:54.240
So the AI is really good at that and it keeps you on track.
00:10:54.399 --> 00:11:12.720
One of the key safeguards that we have is built-in automatic validation to make sure the AI produced a good scenario file and with good information about if it's not a valid scenario file, what was wrong with it, so that the AI can self-correct itself.
00:11:12.960 --> 00:11:24.399
But you can generate the scenario for like pennies and it'll take minutes rather than thousands of dollars of tokens, at which at which point just pay a red team.
00:11:24.639 --> 00:11:24.960
Yeah.
00:11:25.120 --> 00:11:25.440
Yeah.
00:11:26.000 --> 00:11:32.000
So this synthetic data is purposefully not perfect.
00:11:32.240 --> 00:11:40.080
With EvidenceForge, you built in a way to simulate gaps in network visibility, like missing spanner tap data.
00:11:40.320 --> 00:11:43.759
Why is it important for this data to be imperfect?
00:11:44.000 --> 00:11:48.080
I I will back up for just a second and say there's two kinds of imperfection.
00:11:48.320 --> 00:11:54.240
One is it's a simulation, so we can never simulate ever a real network perfectly, right?
00:11:54.480 --> 00:11:54.639
Right.
00:11:54.799 --> 00:11:56.480
Like there will be mistakes.
00:11:56.720 --> 00:11:59.279
We are explicitly not trying to make perfect data.
00:11:59.360 --> 00:12:07.600
We're trying to make data that's good enough for the purpose, but an experienced analyst looking closely at it would probably still tell that it was synthetic data.
00:12:07.759 --> 00:12:14.399
But uh good enough for training, good enough for detection engineering, and and maybe a little bit more than just good enough, right?
00:12:14.559 --> 00:12:21.120
But the the part I think you're talking about is I have this built-in evaluation loop that helps me.
00:12:21.279 --> 00:12:35.919
I on the development side, I also have the AI help me evaluate the quality of the generated data because it's very hard to do uh when you're generating several gigabytes of data on every iteration to see if you fix the problem or whatever.
00:12:36.080 --> 00:12:42.720
So I have the AI look at each iteration of the data from four lenses.
00:12:42.799 --> 00:12:44.960
So, like four different agents, really.
00:12:45.120 --> 00:12:53.600
Um, one is a network, one is a host forensics, one is a detection engineer, and one is an experienced threat hunter.
00:12:54.000 --> 00:13:01.039
One thing that was surprising to me at the beginning of this, I set these up and they worked for a while.
00:13:01.120 --> 00:13:10.480
And every time it goes through, it would point out like the things that it found were the strongest tells that it was synthetic data, and then I we could fix them.
00:13:10.720 --> 00:13:22.000
But uh, it eventually got to the point where one of the strongest tells coming over and over again from most of the reviewers was you have every event.
00:13:25.360 --> 00:13:26.639
And that's unrealistic.
00:13:26.879 --> 00:13:28.960
You're not, yeah, you're not missing any.
00:13:29.120 --> 00:13:32.399
And I was like, well, yeah, we're simulating uh something.
00:13:32.559 --> 00:13:48.879
I would like to get all the data, but they kept coming back and saying this is clearly synthetic because in the real world you have things that drop pipelines that don't work, they work 99.9%, maybe, or uh, but not a hundred percent.
00:13:49.120 --> 00:13:57.279
And so, yeah, I had a a model in there where you could optionally tell it, yeah, I I want to be smart about how I drop.
00:13:57.679 --> 00:14:08.480
I don't necessarily just drop random events, although you could, but you know, you could drop like a few minutes of data out of one of the logs or something like that.
00:14:08.559 --> 00:14:26.720
Uh it you basically it's configurable via profiles uh and and it ships with uh default not doing anything like that, but you can turn it on like the the typical enterprise uh protocol and it would it would fix that too perfect data a little bit.
00:14:26.960 --> 00:14:27.600
Yeah.
00:14:28.000 --> 00:14:29.200
I was really shocked.
00:14:29.279 --> 00:14:30.879
I was like, at first it was a compliment.
00:14:30.960 --> 00:14:36.080
It's like, oh, the the AI says my uh my program's data is too good.
00:14:36.320 --> 00:14:38.320
Yeah, I don't know how to take that.
00:14:39.039 --> 00:14:42.879
Yeah, I mean, I guess take it as a compliment, but that's really funny.
00:14:43.200 --> 00:14:50.879
Okay, um, so I'm going to get even more into the nitty-gritty because I'm a nerd and I found this really interesting.
00:14:51.039 --> 00:14:55.360
Uh, in the blog, you talk about Hawk's processes for background noise.
00:14:55.519 --> 00:15:05.039
Um, for people who aren't data scientists, how does that make background noise feel more human compared to just random traffic?
00:15:05.360 --> 00:15:07.679
Yeah, uh, I'm not data scientists either.
00:15:07.840 --> 00:15:10.399
I just pretend to be one as a threat hunter.
00:15:10.720 --> 00:15:13.360
There's I always tell people there's a big overlap.
00:15:13.919 --> 00:15:15.840
Data science and threat hunting.
00:15:16.000 --> 00:15:19.600
Threat hunting is almost like just data science applied to cybersecurity.
00:15:19.759 --> 00:15:21.679
So yeah, there's a there's a lot of overlap.
00:15:21.840 --> 00:15:28.799
The it's not that important that you know like exactly what the the the Hawks algorithm is or how it works or whatever.
00:15:28.960 --> 00:15:33.840
But the the basic idea is we have certain things that we're simulating.
00:15:34.000 --> 00:15:43.360
On the computers that we're simulating, they usually have things that are driven only by the computer, like automation, schedule tasks, or whatever.
00:15:43.519 --> 00:15:44.960
They recur on a schedule.
00:15:45.120 --> 00:15:49.679
So, you know, five minutes after the hour, every hour, or at 8 a.m.
00:15:49.840 --> 00:15:51.919
on Monday mornings or whatever it is, right?
00:15:52.240 --> 00:15:56.080
And they're kind of metronomic unless you lose events.
00:15:56.399 --> 00:15:58.720
But the users are not like that.
00:15:58.879 --> 00:15:59.120
Right.
00:15:59.360 --> 00:16:18.559
They come in in the morning, they let's let's just say you're typing commands at the like the Linux command line, or maybe you're clicking on commands or clicking on icons in the in the Windows environment, but you're doing some things, launching applications, sending emails, browsing the web or whatever.
00:16:18.720 --> 00:16:20.639
You don't do that like a metronome.
00:16:20.799 --> 00:16:21.039
Right.
00:16:21.279 --> 00:16:22.480
It's in burst, basically.
00:16:22.559 --> 00:16:31.840
I'm gonna type this command, I'm gonna type this command, I'm gonna type this command, but in the between I have to review the outputs and see what I need to do next to make a judgment.
00:16:31.919 --> 00:16:35.440
It takes a little bit of time, and it's not always the same amount of time.
00:16:35.759 --> 00:16:43.200
Um, but also it's bursty because I might be doing things and then get distracted, get some coffee for a little while.
00:16:43.279 --> 00:16:47.039
So the the Hawks algorithm is just trying to model that.
00:16:47.200 --> 00:16:57.600
It's the short form of it is every action that a user takes makes it slightly more likely that it will take another action shortly after.
00:16:58.320 --> 00:16:59.919
So it's making a burst.
00:17:00.080 --> 00:17:00.559
That's it.
00:17:00.720 --> 00:17:08.559
I mean, it doesn't snowball to be like, oh, eventually the user has to take all the actions, you know, without a break.
00:17:08.880 --> 00:17:13.839
But it's just a way of trying to to model that burstiness.
00:17:14.079 --> 00:17:14.960
It's not perfect.
00:17:15.119 --> 00:17:25.119
Again, we you can't like model uh the whole thing perfectly, but it's good enough uh that it doesn't obviously look synthetic just because of the timing.
00:17:25.440 --> 00:17:34.240
Yeah, and it's I it kind of reminds me of um a blog article that we put out several months back about good enough fuzzing.
00:17:34.480 --> 00:17:42.160
Um and just like, you know, like you need to know what your intention is when you're creating these things or when you're using them.
00:17:42.319 --> 00:17:49.920
Like it it doesn't have to be perfect for your use case in order for it to be a successful element of what you're doing.
00:17:50.400 --> 00:18:01.920
Um, so to close out, if someone is listening to this and wants to start using EvidenceForge today or tomorrow, what do you think would be a good place for their first experiment?
00:18:02.079 --> 00:18:03.119
Where should they start it?
00:18:03.440 --> 00:18:07.359
It's all available on GitHub, so they could just check out the GitHub repo.
00:18:07.680 --> 00:18:17.119
And in fact, there is a quick start uh in the README file there that we have published a scenario that is in the repo.
00:18:17.279 --> 00:18:31.680
And you could just ask your favorite AI, Claude, or Codex, it works with either of them, to say, hey, please generate the this example scenario and just see like what the process is like.
00:18:32.160 --> 00:18:39.920
Or you can literally just say, I want to generate an evidence forge scenario, and the AI will lead you through it.
00:18:40.160 --> 00:18:45.519
So it's really easy to get started making the scenarios.
00:18:45.680 --> 00:18:49.279
There's hopefully very little learning curve if I did my job right.
00:18:49.440 --> 00:18:55.279
There might you might have to install the skills and and whatever, but I tried to make it really easy.
00:18:55.599 --> 00:18:58.160
I really encourage everyone to go and check it out.
00:18:58.319 --> 00:18:59.839
David, thanks so much for joining.
00:19:00.079 --> 00:19:01.279
Hey, thanks for having me.
00:19:01.759 --> 00:19:04.559
It's always fun to talk about my projects.
00:19:05.519 --> 00:19:07.359
I'm a typical researcher.
00:19:07.759 --> 00:19:09.200
Same, same, same.
00:19:09.279 --> 00:19:11.200
Yeah, me with my knitting all the time.
00:19:11.279 --> 00:19:13.519
Just like, let me show it off to you.
00:19:13.920 --> 00:19:16.079
Thanks for tuning in to Talos Takes.
00:19:16.240 --> 00:19:18.480
We'll see you in two weeks with a new episode.
00:19:18.559 --> 00:19:20.960
And until then, stay safe out there.
