605: Goodbye World

1
00:00:00,005 --> 00:00:04,085
Enhanced BPF is another hotness of Linux for the last couple of years.
2
00:00:04,945 --> 00:00:09,105
And when the patches were first added to Linux, the lead developer,
3
00:00:09,205 --> 00:00:13,265
Alexei Storitov, said this allows you to do crazy things.
4
00:00:13,765 --> 00:00:17,485
This is normally the words you don't tell Linus when you want him to accept
5
00:00:17,485 --> 00:00:20,905
patches into the kernel, but fortunately the patches were accepted.
6
00:00:21,585 --> 00:00:25,645
Enhanced BPF puts a virtual machine in the kernel that we can program from user space.
7
00:00:36,845 --> 00:00:41,605
Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris.
8
00:00:41,745 --> 00:00:42,565
My name is Wes.
9
00:00:42,745 --> 00:00:43,465
And my name is Brent.
10
00:00:44,125 --> 00:00:48,425
Well, hello, gentlemen. Today, we're digging into a superpower that is inside
11
00:00:48,425 --> 00:00:50,085
all of our Linux kernels.
12
00:00:50,305 --> 00:00:54,945
We're going to talk about how eBPF works and how anyone can take advantage of it.
13
00:00:55,065 --> 00:00:57,365
Then we're going to round out the show with some great feedback,
14
00:00:57,365 --> 00:01:02,145
some picks, and a lot more. It is a special out-of-time episode.
15
00:01:02,145 --> 00:01:06,525
As you listen to this right now, we are at Planet Nix and Scale.
16
00:01:06,765 --> 00:01:08,505
So this is an episode in between.
17
00:01:08,965 --> 00:01:12,245
You're going to get all our Planet Nix and Scale coverage coming soon,
18
00:01:12,245 --> 00:01:17,105
but we wanted to take a moment in between episodes and do something kind of
19
00:01:17,105 --> 00:01:19,265
fun and really dig in and get technical.
20
00:01:20,085 --> 00:01:23,305
But first, I want to say a big good morning to our friends at TailScale.
21
00:01:23,945 --> 00:01:28,505
TailScale.com slash unplugged. They are the easiest way to connect your devices
22
00:01:28,505 --> 00:01:31,325
and services to each other wherever they are.
23
00:01:31,525 --> 00:01:35,705
It is modern networking. It's a flat mesh network that is protected by...
24
00:01:35,705 --> 00:01:36,565
Waggaard.
25
00:01:36,765 --> 00:01:42,725
That's right. And it is so fast, so quick to get going, and it gives you superpowers.
26
00:01:43,025 --> 00:01:46,145
Not only do you get a flat mesh network across complex networks,
27
00:01:46,245 --> 00:01:49,465
so maybe you have multiple data centers or VPSs, or you've got mobile devices,
28
00:01:49,525 --> 00:01:53,745
or you've got double carrier grade NAT, it'll smooth all of that out.
29
00:01:54,025 --> 00:01:57,945
That's fantastic. But then there's also a whole suite of tools that make it
30
00:01:57,945 --> 00:01:58,965
really convenient to use.
31
00:01:59,105 --> 00:02:02,505
Sort of like AirDrop for your entire Tailnet, including like your Android device
32
00:02:02,505 --> 00:02:04,265
and your Linux device, so you can send files around.
33
00:02:04,465 --> 00:02:07,485
They will manage your SSH keys through your Tailnet for you,
34
00:02:07,485 --> 00:02:10,285
so you can just log into all your individual devices. You don't have to manually
35
00:02:10,285 --> 00:02:11,925
copy keys around like an animal.
36
00:02:12,405 --> 00:02:16,585
And they also offer more advanced features, so you can set up ACLs and really
37
00:02:16,585 --> 00:02:19,525
manage the system and lock down only certain things to certain people.
38
00:02:19,765 --> 00:02:23,265
And when you try it now, when you go to tailscale.com slash unplugged,
39
00:02:23,365 --> 00:02:26,125
you get it for free up to 100 devices and three users.
40
00:02:26,765 --> 00:02:30,405
No credit card required. I mean, you can really cook with 100 devices,
41
00:02:30,405 --> 00:02:33,545
and then maybe you'll discover it's great to bring to work too.
42
00:02:33,665 --> 00:02:36,185
Thousands of companies like Instacart, Hugging Face, Duolingo,
43
00:02:36,325 --> 00:02:40,385
Jupyter Broadcasting, and many others use Tailscale, and we love it.
44
00:02:40,565 --> 00:02:43,545
Try one for yourself. Go get yourself a little Tailscale right now.
45
00:02:43,725 --> 00:02:45,985
You're going to love the way it tastes, and you're going to love how easy it
46
00:02:45,985 --> 00:02:49,545
is to get going. If you've got five minutes, you'll probably get it running on three devices.
47
00:02:49,785 --> 00:02:55,465
I have no inbound ports on any of my firewalls. Talescale.com slash unplugged.
48
00:02:58,187 --> 00:03:02,207
Well, like I mentioned, we are at Planet Nix and Scale right now.
49
00:03:02,907 --> 00:03:09,627
But we did think it was sort of a perfect out-of-time episode because we really
50
00:03:09,627 --> 00:03:15,147
first got excited about eBPF at Scale back in 2019?
51
00:03:15,707 --> 00:03:17,207
Yeah, a million years ago.
52
00:03:17,627 --> 00:03:25,327
And it was a great presentation that really just brought home to us how powerful this was going to be.
53
00:03:25,587 --> 00:03:28,947
Yeah, you heard from the man himself, Brendan Gregg. observability,
54
00:03:29,187 --> 00:03:33,367
performance, tracing, guru, eBPF was still kind of new then.
55
00:03:33,607 --> 00:03:38,087
He's kind of well-known. You've maybe seen his famous video online using D-Trace
56
00:03:38,087 --> 00:03:40,667
to show how hard drives don't like it when you yell at them.
57
00:03:41,567 --> 00:03:46,207
So, you know, has deep insight into this area and was, even in 2019 and earlier,
58
00:03:46,447 --> 00:03:48,947
was already getting excited about eBPF.
59
00:03:49,067 --> 00:03:55,407
So it's kind of neat to look back now as like a whole giant marketplace of eBPF-based,
60
00:03:56,147 --> 00:03:57,387
observability tools now exist.
61
00:03:57,387 --> 00:04:01,367
And the name is sort of a misnomer, right? Because it sounds like a packet filter.
62
00:04:01,927 --> 00:04:04,927
And so you think, well, how? Okay, what is this a firewall, guys?
63
00:04:05,167 --> 00:04:08,527
No, no, no. That's why we wanted to play that intro clip. It is so much more.
64
00:04:08,967 --> 00:04:13,887
It is really a VM inside the kernel that can run simple code that you can create
65
00:04:13,887 --> 00:04:15,047
and craft that is protected.
66
00:04:15,867 --> 00:04:18,747
And so since this is a prerecord, we're not going to have your boost this week,
67
00:04:18,807 --> 00:04:21,947
but we do want to know if you like this type of deep dive into this particular topic.
68
00:04:22,207 --> 00:04:24,607
So boost those in and we'll bank them for when we come back.
69
00:04:25,127 --> 00:04:28,067
But let's talk about the extended Berkeley packet filter.
70
00:04:28,427 --> 00:04:32,207
Yeah. So it does do some networking stuff still to this day.
71
00:04:32,527 --> 00:04:39,167
But it did, as you say, start out as a packet filter introduced in 1992 to efficiently
72
00:04:39,167 --> 00:04:42,927
filter network packets in BSD operating systems. and.
73
00:04:42,927 --> 00:04:46,387
Pf sense and other firewall products like open sense of you know they've been
74
00:04:46,387 --> 00:04:51,367
using bpf as part of their core product i that's one of the reasons i was an early pf sense user.
75
00:04:51,367 --> 00:04:54,927
And you know before too long within the same decade it made its way over to
76
00:04:54,927 --> 00:04:58,407
linux in the form of tcp dump and already you're seeing this thing right where
77
00:04:58,407 --> 00:05:03,307
you can kind of use user space to help better observe what's going on on your system,
78
00:05:04,185 --> 00:05:07,545
And so if you've ever written sort of an expression to filter things or look
79
00:05:07,545 --> 00:05:11,205
at packets using TCP dump, well, you're using a language that then gets compiled
80
00:05:11,205 --> 00:05:14,205
down to BPF bytecode and executed.
81
00:05:14,565 --> 00:05:17,645
That bytecode right there is kind of the magic, right? Because it turns out
82
00:05:17,645 --> 00:05:21,025
that this thing is essentially capable of running this bytecode.
83
00:05:21,085 --> 00:05:22,105
So it's not just a packet filter.
84
00:05:22,565 --> 00:05:25,645
Yeah, and that's like how the implementation works. As it started out,
85
00:05:25,745 --> 00:05:27,265
it was a very simple virtual machine.
86
00:05:27,445 --> 00:05:32,045
And don't think virtual machine like QEMU necessarily are like simulating a full computer.
87
00:05:32,045 --> 00:05:35,665
the point is it's like a very limited restricted bytecode
88
00:05:35,665 --> 00:05:39,245
that can only do certain things relevant to at first filtering packets
89
00:05:39,245 --> 00:05:42,245
and that lets you make sure that like it's not going to do anything crazy it
90
00:05:42,245 --> 00:05:45,165
can't go into infinite loops all kinds of other nice things and optimize
91
00:05:45,165 --> 00:05:47,965
it and then be able to load it
92
00:05:47,965 --> 00:05:50,885
in and have you know you run the program it supplies
93
00:05:50,885 --> 00:05:53,785
the packet and anything else that you need is input to it
94
00:05:53,785 --> 00:05:56,645
and then the machine executes and ultimately that's how you tell like
95
00:05:56,645 --> 00:06:00,145
do i accept the packet or do i drop the packet uh but
96
00:06:00,145 --> 00:06:03,665
at first it was a you know a very limited
97
00:06:03,665 --> 00:06:06,845
i think i had like a two two registers to use super limited thing
98
00:06:06,845 --> 00:06:14,685
but ebpf extended bpf was introduced in linux 318 this was like 2014 so bpf
99
00:06:14,685 --> 00:06:18,905
had been around for a while there'd been various developments but ebpf really
100
00:06:18,905 --> 00:06:23,265
kicked things off in the linux side of things bpf hadn't you know caught up
101
00:06:23,265 --> 00:06:26,625
with the times in some ways it was still 32 bit it had some of those register limitations.
102
00:06:27,185 --> 00:06:30,145
So they upgraded to 64 bit registers added more
103
00:06:30,145 --> 00:06:33,165
instructions they added the verifier to
104
00:06:33,165 --> 00:06:36,285
the kernel which is a big part of it that lets you analyze bpf programs
105
00:06:36,285 --> 00:06:40,145
to make sure they're safe no infinite loops they don't do invalid memory access
106
00:06:40,145 --> 00:06:43,785
there's a checker process right because it's like you're loading something from
107
00:06:43,785 --> 00:06:48,085
user space into the kernel that's a big security concern so you you want to
108
00:06:48,085 --> 00:06:51,505
make sure that you have and just be on security operationally too right you
109
00:06:51,505 --> 00:06:53,225
don't want it to be able to crash the kernel so.
110
00:06:53,225 --> 00:06:57,285
We actually started to see this stuff land you know progressively in linux 3
111
00:06:57,285 --> 00:07:02,525
18 and beyond so this is actually like you said 2014 that's a this has been landing for a while.
112
00:07:02,525 --> 00:07:07,405
Yeah definitely and then so that was kind of like the raw stuff and then 2018
113
00:07:07,405 --> 00:07:12,605
and beyond people started adding more tools like the bcc but the bpf compiler
114
00:07:12,605 --> 00:07:18,225
collection bpf trace which we'll talk about too uh the company psyllium and
115
00:07:18,225 --> 00:07:20,845
their their products have a bunch of like ebpf stuff for,
116
00:07:22,105 --> 00:07:26,405
Kubernetes offerings plus we got better like compiler support there's something
117
00:07:26,405 --> 00:07:31,265
called Kori or compile once run everywhere so like compilers have better support
118
00:07:31,265 --> 00:07:34,645
to be able to make you know you can compile your BPF program and not have to
119
00:07:34,645 --> 00:07:37,585
worry about as much necessarily depending on what you're doing with it about
120
00:07:37,585 --> 00:07:41,765
how compatible it'll be with the kernel where you compiled it versus where you're running it so.
121
00:07:41,765 --> 00:07:46,365
The idea being like it should be compatible across multiple versions of Linux
122
00:07:46,365 --> 00:07:48,585
as long as they have this correct implementation.
123
00:07:48,585 --> 00:07:56,305
Right okay This has been so successful that newer versions of Dtrace on Linux,
124
00:07:58,081 --> 00:08:02,061
They're basically just some extra user space stuff that uses eBPF and other
125
00:08:02,061 --> 00:08:03,321
kernel primitives under the hood.
126
00:08:03,501 --> 00:08:04,841
Really? Yeah. Huh.
127
00:08:05,081 --> 00:08:07,481
And eBPF was also ported to Windows.
128
00:08:07,821 --> 00:08:11,501
Yeah, I heard about this. They're getting a lot of good stuff over there.
129
00:08:11,681 --> 00:08:16,401
One important thing with the extended part is they were also able to make the
130
00:08:16,401 --> 00:08:21,081
instruction set be more sympathetic to modern hardware, and they implemented
131
00:08:21,081 --> 00:08:22,681
just-in-time compilation as well.
132
00:08:22,721 --> 00:08:25,901
So that meant eBPF can be really fast.
133
00:08:26,461 --> 00:08:29,281
They've also made it so that there's relatively stable APIs we've kind of been
134
00:08:29,281 --> 00:08:33,921
talking about, so that as kernels change that, you can use eBPF to hook into internals.
135
00:08:34,361 --> 00:08:37,501
If you do that, there's no guarantees, right? But that's kind of on the tin.
136
00:08:37,841 --> 00:08:41,461
But there is some stable interfaces that you can use, which is nice.
137
00:08:42,901 --> 00:08:46,641
So the other important point to know is there's kind of various things it can do.
138
00:08:47,461 --> 00:08:52,221
There's XDP, which is Express Data Path, which intercepts packets basically
139
00:08:52,221 --> 00:08:54,781
at like the earliest point that you can.
140
00:08:54,781 --> 00:08:59,761
you have limitations on what you can do but it can be super low level and performant
141
00:08:59,761 --> 00:09:03,041
and so you can see this sometimes maybe unlike responding to ddos attacks where
142
00:09:03,041 --> 00:09:06,801
you're getting flooded with traffic that you have you know maybe you can identify
143
00:09:06,801 --> 00:09:10,461
various ways that you can program in here so you could essentially.
144
00:09:10,461 --> 00:09:14,401
Ideally catch it before it begins to overwhelm the system because you're catching
145
00:09:14,401 --> 00:09:16,281
it much further in like the driver stack.
146
00:09:16,281 --> 00:09:19,881
Yeah that's the idea and it doesn't do as much as like the very general and
147
00:09:19,881 --> 00:09:23,901
powerful but you know full-featured linux kernel networking stack you can kind
148
00:09:23,901 --> 00:09:26,381
of just be like, well, no, if it looks like this at all, just cut it off immediately.
149
00:09:26,561 --> 00:09:29,581
Yeah, if I'm getting DDoSed, I don't want the network stack trying to figure
150
00:09:29,581 --> 00:09:31,861
out what to do with all this traffic because that's what's going to take me down.
151
00:09:31,981 --> 00:09:34,701
Yeah, one way to say it is limited context but maximum performance.
152
00:09:35,641 --> 00:09:36,421
I like that.
153
00:09:37,081 --> 00:09:39,981
Okay, so then you can also do various types of K-probes.
154
00:09:40,221 --> 00:09:40,601
I'm sorry?
155
00:09:40,961 --> 00:09:46,281
Yeah, K-probes. And this is dynamic instrumentation. That's why it sounds quite probing.
156
00:09:46,601 --> 00:09:49,041
You can hook almost any kernel function.
157
00:09:50,861 --> 00:09:54,421
but you don't there's no like clear definition of what you're going to get you
158
00:09:54,421 --> 00:09:57,081
have to go look at the function you're hooking into it's all going to be dependent
159
00:09:57,081 --> 00:10:01,381
on that but no kernel internals any kernel function almost i'm sure there are
160
00:10:01,381 --> 00:10:03,561
some limitations that's pretty a lot of them yeah i.
161
00:10:03,561 --> 00:10:08,501
Mean that could be like from you know keyboard input to network traffic to disk
162
00:10:08,501 --> 00:10:10,381
io i mean that's there's all kinds of things.
163
00:10:10,381 --> 00:10:13,721
There so that's where a lot of some of the power comes from yeah but that may
164
00:10:13,721 --> 00:10:16,381
or may not be stable there's no guarantee about it being stable across kernel
165
00:10:16,381 --> 00:10:18,761
versions, can change at any time. People can update the signatures.
166
00:10:19,241 --> 00:10:22,541
That's one of the things that's been happening in the Rust discussion is the
167
00:10:22,541 --> 00:10:26,361
kernel, many developers expect to be able to make a change like that and because
168
00:10:26,361 --> 00:10:29,761
they have one big code base, they can update it everywhere and be able to do
169
00:10:29,761 --> 00:10:32,721
a refactoring like that. And, but the other one,
170
00:10:33,548 --> 00:10:39,408
Trace points. This is an important one. Trace points. Predefined stable instrumentation.
171
00:10:39,688 --> 00:10:39,868
Ah.
172
00:10:40,048 --> 00:10:44,968
Low overhead, but only available where explicitly added by kernel developers.
173
00:10:45,248 --> 00:10:49,968
So, okay. So a trace point would be like a spot you hook in and start getting
174
00:10:49,968 --> 00:10:51,588
metrics or information out of.
175
00:10:51,888 --> 00:10:56,108
But the developer of the subsystem has to implicitly support that. Yeah.
176
00:10:56,168 --> 00:10:58,748
You have to add that in versus totally dynamic with a K-probe.
177
00:10:58,848 --> 00:10:58,988
Okay.
178
00:10:59,288 --> 00:11:02,988
But the upside is you get a structured data specific to each trace point,
179
00:11:03,048 --> 00:11:05,008
right? So the trace point tells you what it is.
180
00:11:05,168 --> 00:11:05,968
Yeah, yeah, okay.
181
00:11:06,128 --> 00:11:10,368
And they're maintained across kernel versions, so you can kind of rely on them for more long-term use.
182
00:11:10,748 --> 00:11:13,708
I have a feeling this might be relevant later. Okay, good to know.
183
00:11:14,128 --> 00:11:18,488
Yeah, I think that's probably like the quick, high-level version of what eBPF
184
00:11:18,488 --> 00:11:19,808
is and kind of what it can do.
185
00:11:19,948 --> 00:11:26,488
And it is so, so, so simple, but yet so powerful is what I love about it.
186
00:11:26,568 --> 00:11:31,588
And we have a couple of examples in the show, and I think some hands-on stuff
187
00:11:31,588 --> 00:11:32,508
that people could take away.
188
00:11:33,128 --> 00:11:35,948
And maybe we should start with BCC tools themselves.
189
00:11:36,388 --> 00:11:39,188
Yeah, because we've both had a chance to play with those.
190
00:11:39,328 --> 00:11:42,648
Yeah, I was, you know, I was joking around with Wes and be like,
191
00:11:42,708 --> 00:11:48,168
it'd be kind of great to know where in the system when I open up this directory
192
00:11:48,168 --> 00:11:49,468
that's a fuse mounted directory,
193
00:11:49,688 --> 00:11:54,568
like where is the actual delay happening and what part of the system is sitting around waiting.
194
00:11:54,568 --> 00:11:59,108
And through this process of trying to get to that, I came across tools that
195
00:11:59,108 --> 00:12:03,508
let me look at my disk IO analysis or let me look at the network traffic and
196
00:12:03,508 --> 00:12:06,688
really kind of using these different tools, putting them together.
197
00:12:06,688 --> 00:12:11,728
You can actually start to get a really good picture of where the delay was happening in the system.
198
00:12:12,523 --> 00:12:15,623
And, you know, for me, it was like, this is going to be amazing.
199
00:12:16,043 --> 00:12:21,603
I had discovered by running it on my desktop, just as when I was experimenting
200
00:12:21,603 --> 00:12:25,263
with this stuff, that, oh, yeah, I've still got this errands app that we talked
201
00:12:25,263 --> 00:12:27,243
about on the show that I'm not currently using like today.
202
00:12:27,283 --> 00:12:31,123
But I guess when I close it, it doesn't actually close. It doesn't leave an icon.
203
00:12:31,303 --> 00:12:36,863
I had no idea it was running. But using these tools, I started seeing this application
204
00:12:36,863 --> 00:12:40,483
that was hitting my disk every so often. And I'm like, I don't recognize this.
205
00:12:40,483 --> 00:12:42,963
and I discovered I actually had that running the entire time.
206
00:12:43,103 --> 00:12:44,063
And it was kind of useful.
207
00:12:44,303 --> 00:12:48,983
Yeah, it kind of skips through a lot of the boundaries and other limitations
208
00:12:48,983 --> 00:12:51,723
that can sometimes pop up when you're trying to look at your system.
209
00:12:51,983 --> 00:12:53,743
So it can be surprisingly insightful.
210
00:12:54,403 --> 00:12:57,843
They do have a nice tutorial, which we'll link. They also, I like that they
211
00:12:57,843 --> 00:12:59,303
have a set of things to run first.
212
00:12:59,443 --> 00:13:02,883
Like before you go to these tools, make sure you check all like the regular
213
00:13:02,883 --> 00:13:06,403
system monitoring tools first because we'll see as a theme, like,
214
00:13:06,903 --> 00:13:09,823
you know, your H-tops and B-tops and all kinds of things.
215
00:13:10,483 --> 00:13:14,483
kind of get a broad look and you could see some specifics whereas you could
216
00:13:14,483 --> 00:13:18,683
you can make broad eppf programs but by default they're going to be a lot more
217
00:13:18,683 --> 00:13:21,043
specific when you're looking at like one thing.
218
00:13:21,043 --> 00:13:21,743
Like disk.
219
00:13:21,743 --> 00:13:23,743
Latency or something specific to the file system.
220
00:13:23,743 --> 00:13:29,003
Or networking yeah yeah that's a good point um all right so we could talk about
221
00:13:29,003 --> 00:13:33,663
some of the commands we tried i know that uh file top and tcp top we played
222
00:13:33,663 --> 00:13:39,363
around with um i did not get a chance to play around with ext4 slower and XFS
223
00:13:39,363 --> 00:13:41,643
slower, but this is an interesting way you can use it too.
224
00:13:41,883 --> 00:13:45,223
Uh-huh, yeah. It'll tell you when, you know, there's things happening in your
225
00:13:45,223 --> 00:13:49,143
file system that are causing latency more than, like, a programmable amount
226
00:13:49,143 --> 00:13:50,603
you can pass on the command line.
227
00:13:51,363 --> 00:13:56,503
Or they've got ZFS dist, which traces ZFS reads, writes, opens,
228
00:13:56,683 --> 00:14:00,383
fsyncs, and then summarizes the latency of all that in a histogram for you.
229
00:14:00,503 --> 00:14:01,363
That's really cool.
230
00:14:01,823 --> 00:14:07,943
Yeah, right? And then two really useful basic ones are exec snoop and open snoop.
231
00:14:08,799 --> 00:14:14,099
Yeah, OpenSnoop, isn't that kind of like, oh, there's like a Mac app that monitors
232
00:14:14,099 --> 00:14:16,679
traffic? Is that what it was? OpenSnoop something else?
233
00:14:16,859 --> 00:14:19,319
Yeah, no. So OpenSnoop, you're thinking of, there's a Linux one,
234
00:14:19,399 --> 00:14:21,459
I think, right? OpenSnitch? Yeah, just one thing.
235
00:14:22,039 --> 00:14:23,379
Snitch. Snitch. That's, yes.
236
00:14:24,759 --> 00:14:26,479
OpenSnoop tracks file opens.
237
00:14:26,679 --> 00:14:26,879
Okay.
238
00:14:27,059 --> 00:14:32,079
So you can run this and it hooks into the kernel via BPF. And then anything
239
00:14:32,079 --> 00:14:35,259
that's running on your system that opens files, it'll just print out in your
240
00:14:35,259 --> 00:14:36,499
console right in front of you.
241
00:14:36,519 --> 00:14:38,199
I did play with this one. I did play with this OpenSnoop one.
242
00:14:38,279 --> 00:14:41,359
You're right. Yeah, that's really cool because I remember I launched,
243
00:14:41,479 --> 00:14:44,659
you could launch particular things and then just watch all the crap that happens on your system.
244
00:14:44,759 --> 00:14:47,719
Right. And especially, you know, you can filter it, you can have it look at
245
00:14:47,719 --> 00:14:50,419
specific processes or you could, you know, grab output or whatever.
246
00:14:50,859 --> 00:14:54,499
So you can filter on a busy system, but I think it's especially useful on a
247
00:14:54,499 --> 00:14:58,139
system you think should be, you know, relatively quiet and just as a way to
248
00:14:58,139 --> 00:15:00,659
see, like, what is actually happening in the background?
249
00:15:01,879 --> 00:15:07,179
And for short-lived processes that might be hard to see in a summary-type program
250
00:15:07,179 --> 00:15:09,999
that are still doing things, it'll still print in OpenSnoop.
251
00:15:10,499 --> 00:15:13,719
That's where ExecSnoop does a similar thing, but for process opens.
252
00:15:14,299 --> 00:15:17,279
So again, for things where it's hard to see in a general tool,
253
00:15:17,379 --> 00:15:21,099
but you really want to see at a nitty-gritty level what is happening in being
254
00:15:21,099 --> 00:15:24,459
Exec on this box, ExecSnoop is handy for that.
255
00:15:25,059 --> 00:15:30,899
I have a sense Exec Snoop has come up actually in real life for you as a handy tool.
256
00:15:31,059 --> 00:15:31,139
Yeah.
257
00:15:31,219 --> 00:15:32,719
Do you want to share a war story with us?
258
00:15:32,719 --> 00:15:38,579
I do. So a few years ago, I was adminning a VPS box. I wasn't totally in charge
259
00:15:38,579 --> 00:15:42,759
of, but had some, you know, administration over. And...
260
00:15:44,289 --> 00:15:48,289
Things were seemingly broken. I'd noticed some messed up statistics first,
261
00:15:48,409 --> 00:15:49,629
right? The metrics were a little odd.
262
00:15:49,869 --> 00:15:53,889
And I went on and I started just kind of doing regular updates and poking around at the system.
263
00:15:54,249 --> 00:15:57,749
And I noticed that I wasn't able to actually update the initRAMFS,
264
00:15:58,069 --> 00:16:01,529
right? I was doing all the, it was an Ubuntu box. I had everything going.
265
00:16:01,689 --> 00:16:08,689
But there was some like dynamic library linking problem that was happening after
266
00:16:08,689 --> 00:16:11,529
I started digging into the output and tried rebuilding it way too many times.
267
00:16:11,529 --> 00:16:13,529
And I'm like, what, why? Why can't I get it?
268
00:16:13,529 --> 00:16:15,549
Yeah, well, I can't get my initRAMFS updated.
269
00:16:15,789 --> 00:16:18,149
The kernel's here. Everything else in the update's fine.
270
00:16:18,229 --> 00:16:20,589
But you can't boot in that new kernel if you can't get that updated.
271
00:16:20,809 --> 00:16:26,349
Yeah. So this was maybe closer to when I really started playing with these tools
272
00:16:26,349 --> 00:16:28,889
a little bit more seriously and having them installed on more boxes.
273
00:16:29,189 --> 00:16:30,589
And so they were already available.
274
00:16:31,689 --> 00:16:37,049
And I used OpenSnoop because I wanted to see exactly what was being opened and
275
00:16:37,049 --> 00:16:39,649
touched by the update initRAMFS process.
276
00:16:39,889 --> 00:16:40,149
Right.
277
00:16:40,149 --> 00:16:43,769
And then that led me to look at some weird file paths.
278
00:16:44,209 --> 00:16:51,609
And I also started, then I did exec snoop, and that showed similar file paths running on the system.
279
00:16:51,829 --> 00:16:53,549
Oh, uh-oh.
280
00:16:53,729 --> 00:16:57,809
And then I was able to figure out that there was, in fact, a crypto miner running on the box.
281
00:16:57,909 --> 00:16:58,329
Oh, no.
282
00:16:58,529 --> 00:17:05,649
But it had loaded a custom kernel module to hide itself from tools like PS and top and htop.
283
00:17:05,749 --> 00:17:06,429
That's pretty slick.
284
00:17:06,549 --> 00:17:07,689
But not from exec snoop.
285
00:17:08,209 --> 00:17:11,809
Clever. So that's why it was breaking when you were trying to update in that in an FS.
286
00:17:12,029 --> 00:17:15,789
Yes. Something it did to the system. I never quite figured out exactly what
287
00:17:15,789 --> 00:17:19,529
it had changed, but it had messed with some of the files on the system in ways
288
00:17:19,529 --> 00:17:21,769
that meant that the linking was no longer cleanly happening.
289
00:17:21,929 --> 00:17:21,989
Yeah.
290
00:17:22,844 --> 00:17:26,724
So that's a good way to stack those two tools.
291
00:17:26,884 --> 00:17:31,004
You know, OpenSnoop and ExecSnoop are really a couple of nice tools you can stack together.
292
00:17:31,404 --> 00:17:36,044
And there's a whole bunch of the, this is part of BCC, the BPF compiler collection,
293
00:17:36,784 --> 00:17:38,264
which includes a whole bunch of other stuff.
294
00:17:38,344 --> 00:17:42,784
But these are the tools that they ship by default, which leverages the framework
295
00:17:42,784 --> 00:17:45,984
they've built to implement these via eBPF.
296
00:17:46,084 --> 00:17:49,044
And so there's like, yeah, all kinds of stuff, file system specific stuff.
297
00:17:49,224 --> 00:17:51,804
They've got things for, you know, looking at network connections,
298
00:17:51,964 --> 00:17:55,244
TCP connections. They've got stuff for monitoring databases. It's broad.
299
00:17:55,664 --> 00:17:59,524
I don't know if it's necessarily best practice, but you could say that with
300
00:17:59,524 --> 00:18:03,704
these tools, you could be fairly confident that you had cleaned up the crypto
301
00:18:03,704 --> 00:18:04,844
miner and the infection.
302
00:18:05,624 --> 00:18:09,984
You know, because you can actually watch at a much more intimate level what's
303
00:18:09,984 --> 00:18:10,804
happening at the system.
304
00:18:10,944 --> 00:18:11,904
Definitely useful. Yeah.
305
00:18:12,124 --> 00:18:14,884
I'm not saying it's probably a best practice, you know, but probably just wipe
306
00:18:14,884 --> 00:18:18,804
the box. Yeah, but it is sort of nice if for some reason you don't have that
307
00:18:18,804 --> 00:18:22,744
option, you can use these tools to kind of confirm that stuff isn't coming back after a reboot.
308
00:18:23,004 --> 00:18:24,684
Right. And the more, you know, depending
309
00:18:24,684 --> 00:18:28,704
on, but you can see exactly where they're hooking and modify the,
310
00:18:28,844 --> 00:18:33,104
because a lot of this is implemented via a combo of C because that's the part
311
00:18:33,104 --> 00:18:36,944
that gets compiled and like it's like a limited subset of C that does the BPF
312
00:18:36,944 --> 00:18:40,144
stuff that gets converted into a loadable thing for the kernel.
313
00:18:40,144 --> 00:18:45,784
but there's a bunch of python utilities around it to wrap it so you could fork
314
00:18:45,784 --> 00:18:50,724
that copy it modify it if you need it even more or to try to observe specific
315
00:18:50,724 --> 00:18:54,724
things once you identified like a particular problem or threat now.
316
00:18:54,724 --> 00:18:55,584
That's really going deep,
317
00:18:59,061 --> 00:19:06,241
onepassword.com slash unplugged. That's the number one password.com slash unplugged, all lowercase.
318
00:19:06,421 --> 00:19:10,241
And you're going to want to go there because this is something that if I had,
319
00:19:10,321 --> 00:19:14,621
when I still worked in IT, I think it would have sustained me for many, many more years.
320
00:19:14,781 --> 00:19:19,421
The reality is your end users don't, and I mean, without exception,
321
00:19:19,781 --> 00:19:22,601
work on only company-owned devices, applications, and services.
322
00:19:22,981 --> 00:19:27,441
Maybe you get lucky and they mostly do, but I don't find that to be the reality today.
323
00:19:27,861 --> 00:19:31,941
And so the next question becomes, how do you actually keep your company's data
324
00:19:31,941 --> 00:19:35,641
safe when it's sitting on all of these unmanaged apps and devices?
325
00:19:36,681 --> 00:19:40,241
Well, that's where 1Password has an answer. It's extended access management.
326
00:19:40,461 --> 00:19:45,101
1Password extended access management helps you secure every sign-on for every
327
00:19:45,101 --> 00:19:49,961
app on every device because it solves the problems that your traditional IAMs
328
00:19:49,961 --> 00:19:51,901
and MDMs just can't touch.
329
00:19:52,241 --> 00:19:57,721
And it's the first security solution that brings unmanaged devices and applications
330
00:19:57,721 --> 00:20:01,701
and identities, like even vendor, under your control.
331
00:20:02,161 --> 00:20:05,661
It ensures that every credential is strong and protected, every device is known
332
00:20:05,661 --> 00:20:08,261
and healthy, and every app is visible.
333
00:20:08,921 --> 00:20:12,821
This is some powerful stuff, and it's available for companies that use Okta
334
00:20:12,821 --> 00:20:16,241
or Microsoft Entra, and it's in beta for Google Workspace customers too.
335
00:20:17,041 --> 00:20:20,561
OnePassword changed the game for password management, and now they're taking
336
00:20:20,561 --> 00:20:24,921
everything they've learned there and expanding it to the login level and the application level.
337
00:20:25,161 --> 00:20:28,741
You know what a difference it makes when people have proper password management.
338
00:20:28,921 --> 00:20:32,241
Now let's have proper login management and authorization.
339
00:20:32,921 --> 00:20:38,001
1Password also has regular third-party audits and the industry's largest bug
340
00:20:38,001 --> 00:20:39,981
bounty. They exceed the standards set by others.
341
00:20:40,321 --> 00:20:44,501
They really do. So go secure every app, every device, and every identity.
342
00:20:44,681 --> 00:20:49,521
Even the unmanaged ones. You just go to 1Password.com slash unplugged.
343
00:20:49,641 --> 00:20:52,321
All over case. that's onepassword.com.
344
00:20:52,321 --> 00:20:54,061
Slash unplugged,
345
00:20:56,501 --> 00:21:02,261
now Wes it sounds like BCC is an abstraction to EBPF what's going on under the hood here.
346
00:21:02,261 --> 00:21:05,061
Yeah so there are tools that you
347
00:21:05,061 --> 00:21:09,221
can just run like we've been talking about but how did those tools come to be
348
00:21:09,221 --> 00:21:13,921
well that's where some of those abstractions and a framework comes in that allows
349
00:21:13,921 --> 00:21:19,561
basically embedding C code directly within Python scripts and then And BZC's
350
00:21:19,561 --> 00:21:23,441
tools also sort of handle making sure you've got the whole tool chain available.
351
00:21:23,441 --> 00:21:28,181
So it uses LLVM and Clang to compile things. It handles verification of stuff.
352
00:21:28,361 --> 00:21:31,501
It handles loading it into the kernel for you and attaching it to the right
353
00:21:31,501 --> 00:21:35,341
hooks, basically through like Python method calls instead of you having to run
354
00:21:35,341 --> 00:21:37,381
the right commands in the bash shell.
355
00:21:37,981 --> 00:21:41,121
So you kind of get like a unified approach where you can write kernel level
356
00:21:41,121 --> 00:21:44,381
programs without necessarily having to deal with like all the other stuff.
357
00:21:44,381 --> 00:21:48,101
You know I was thinking gosh this really seems like going way beyond what my
358
00:21:48,101 --> 00:21:50,381
skill set would be able to manage but then I thought,
359
00:21:51,299 --> 00:21:56,959
python code i bet an llm would get me 90 of the way there nice these days and
360
00:21:56,959 --> 00:21:58,239
then i could probably just finish it off.
361
00:21:58,239 --> 00:22:01,059
And there's a fair amount of like existing tools that you
362
00:22:01,059 --> 00:22:03,759
can either feed into an llm and ask about or
363
00:22:03,759 --> 00:22:07,979
modify or try and you know hack on yourself yeah and so here's a simple example
364
00:22:07,979 --> 00:22:12,879
that we can play with it uses xdp okay and so there's a simple c program that
365
00:22:12,879 --> 00:22:17,459
has a function called xdp drop all okay uh and that's going to attach to the
366
00:22:17,459 --> 00:22:21,479
xdp hook and so we get like a little data summary of the packet info,
367
00:22:21,639 --> 00:22:22,619
which we're not going to care about.
368
00:22:22,859 --> 00:22:26,839
All we're going to do is return XDP drop, which is a magic value that tells
369
00:22:26,839 --> 00:22:28,839
the kernel, hey, just drop this.
370
00:22:29,079 --> 00:22:29,479
Just drop all.
371
00:22:29,479 --> 00:22:33,019
So it's going to drop everything. And then in the Python, that's it C-wise. That's all we do.
372
00:22:33,299 --> 00:22:36,039
And then in the Python world, there's a little setup to make,
373
00:22:36,119 --> 00:22:37,579
hey, I want a new BPF thing or whatever.
374
00:22:37,939 --> 00:22:41,159
And then we tell it to load in our little blob of C.
375
00:22:41,679 --> 00:22:44,679
And then we attach it to the interface we want. That's two lines of Python.
376
00:22:45,959 --> 00:22:49,279
And then it prints a nice little message and starts working.
377
00:22:49,839 --> 00:22:53,139
And what it should do is drop all network traffic?
378
00:22:53,359 --> 00:22:55,559
Yeah, so we specified a specific interface.
379
00:22:56,079 --> 00:22:58,079
Yeah, so you say on this interface, just drop everything.
380
00:22:58,159 --> 00:23:00,779
And that's because we attached it to that specific interface.
381
00:23:01,199 --> 00:23:04,899
And this is like a quick kill switch. So what we're going to do as a test here
382
00:23:04,899 --> 00:23:08,459
is I've set up a ping with an audible sound so we can see when we're getting a result.
383
00:23:09,279 --> 00:23:13,699
So we can see we're pinging the box right now. And then at some point, Wes is going to kill.
384
00:23:13,939 --> 00:23:17,719
He's actually going to run that kill script, and it's going to drop all traffic.
385
00:23:17,899 --> 00:23:19,619
and we'll hear it drop off. Whenever you're ready, Mr. Payne.
386
00:23:20,259 --> 00:23:22,479
Three, two, one.
387
00:23:24,039 --> 00:23:30,019
There it goes. Yeah, it takes almost no time at all. It happens almost immediately.
388
00:23:30,399 --> 00:23:35,159
Now, and of course, in my regular SSH session, I can't stop it,
389
00:23:35,239 --> 00:23:39,299
but I do have a sneaky console here, so we should be able to take a peek.
390
00:23:39,299 --> 00:23:40,719
Wes always has a sneaky console.
391
00:23:41,219 --> 00:23:43,159
All right, I'm leaving the ping going, so if it resumes.
392
00:23:43,359 --> 00:23:45,459
We should hear it. I'm hitting Control-C now.
393
00:23:45,839 --> 00:23:48,399
Okay, out there it goes. Look at that.
394
00:23:49,039 --> 00:23:53,279
Yeah. So like that wasn't that much work. This is just an Ubuntu 24.04 box.
395
00:23:53,459 --> 00:23:56,679
So like, you know, all the stuff you needed was in the repos already.
396
00:23:58,385 --> 00:24:02,425
I'm thinking, you know, like that's just a it's it's not a total practical example,
397
00:24:02,545 --> 00:24:08,185
but it's a quick example of the power you have there and you're executing that from user space.
398
00:24:08,405 --> 00:24:12,725
Yeah, you do need, you know, root permissions to be able to load it into the kernel.
399
00:24:12,905 --> 00:24:16,185
But then you're executing things inside the kernel that just immediately cutting
400
00:24:16,185 --> 00:24:17,625
off traffic to that interface.
401
00:24:17,805 --> 00:24:22,145
Yes. And I didn't have to build a per kernel kernel module with the right headers
402
00:24:22,145 --> 00:24:26,425
and like have to worry about messing up my implementation in a way that's going to crash the box.
403
00:24:26,425 --> 00:24:29,945
You didn't even have to create like a IP tables like that rejects traffic because
404
00:24:29,945 --> 00:24:33,085
that would actually be much further down in the processing stack if you're using
405
00:24:33,085 --> 00:24:33,925
something like IP tables.
406
00:24:34,045 --> 00:24:37,225
Yeah, that's cool. That's fun.
407
00:24:37,545 --> 00:24:40,485
So in reality, you'd want, right, you would do some sort of filtering where
408
00:24:40,485 --> 00:24:44,445
you would look at the data structure you get from XDP to figure out like,
409
00:24:44,745 --> 00:24:47,205
oh, is this one I want to block or let it keep going through processing?
410
00:24:47,525 --> 00:24:50,965
But yeah, this is your basic. It's not hello world. It's goodbye world.
411
00:24:51,105 --> 00:24:52,985
Yeah. It's goodbye world. Exactly.
412
00:24:52,985 --> 00:24:57,945
But the main point to show that was that this BCC framework is one way if you
413
00:24:57,945 --> 00:24:59,245
want to start developing custom tools.
414
00:24:59,365 --> 00:25:01,705
But you can get even more ad hoc than that.
415
00:25:01,785 --> 00:25:02,125
All right.
416
00:25:02,285 --> 00:25:03,425
Because there's BPF trace.
417
00:25:03,745 --> 00:25:05,565
Yeah. I want to talk about this.
418
00:25:05,745 --> 00:25:09,845
And this is basically sort of like the closest thing to D-Trace for Linux if
419
00:25:09,845 --> 00:25:13,065
you don't want to use the Oracle tool, which I believe Gentoo now has.
420
00:25:13,565 --> 00:25:19,245
Oh, all right. So BPF trace is sort of what I know of D-Trace is like this ultimate
421
00:25:19,245 --> 00:25:22,465
tool when you're debugging or trying to figure out where your system has gone sideways.
422
00:25:22,465 --> 00:25:25,325
again like i was talking about like why is this one fuse directory taking so
423
00:25:25,325 --> 00:25:28,565
long to open and so i imagine this is a similar type of.
424
00:25:28,565 --> 00:25:32,085
Yeah one way to think about is you basically get like a nice targeted little
425
00:25:32,085 --> 00:25:39,045
scripting language to write tracing programs oh that use the trace points in the kernel.
426
00:25:39,045 --> 00:25:41,925
Wow okay all right.
427
00:25:41,925 --> 00:25:44,525
And there are some other options there's a program called ply that i
428
00:25:44,525 --> 00:25:47,305
haven't tried but looks also nice um but bpf trace
429
00:25:47,305 --> 00:25:50,085
is quite popular uh so we talked about right
430
00:25:50,085 --> 00:25:53,185
there's different there's xdp there's the k prob stuff these are the trace points
431
00:25:53,185 --> 00:25:56,885
which is the the easy stable ones because they're defined and they give you
432
00:25:56,885 --> 00:26:03,385
a known data structure so you can do bpf trace dash l or the bcc tools has tplist
433
00:26:03,385 --> 00:26:06,905
which is also just lists all the trace points it also shows you can do other
434
00:26:06,905 --> 00:26:09,845
user space stuff but we're not going to talk about that today so.
435
00:26:09,845 --> 00:26:13,405
There is so tplist i guess that makes it a lot easier if tplist is going to
436
00:26:13,405 --> 00:26:17,985
give you all of the kernel trace points so you know what you have to work with that's really useful.
437
00:26:17,985 --> 00:26:21,065
Right And it tells you the shape of them, too. You get the sort of data structure.
438
00:26:21,185 --> 00:26:21,425
Okay.
439
00:26:21,985 --> 00:26:22,265
Um...
440
00:26:23,624 --> 00:26:25,164
Let me pull it up here.
441
00:26:26,464 --> 00:26:27,484
He's got it there on the machine.
442
00:26:28,004 --> 00:26:32,784
Yeah, so here's sort of like, here's some stuff about block dirty buffer.
443
00:26:32,984 --> 00:26:36,104
And it tells you you get a dev device, you get a sector and a size.
444
00:26:36,284 --> 00:26:39,224
And so you know those are the things you can work with. And then it has a name
445
00:26:39,224 --> 00:26:41,884
and you just tell it that you want to trace that thing.
446
00:26:42,884 --> 00:26:47,404
So this came up because, you know, we talked a little bit about BCC having file system specific tools.
447
00:26:47,584 --> 00:26:51,464
Like, oh, I want to see watch for slow operations from ButterFS, say, right?
448
00:26:51,564 --> 00:26:51,684
Right.
449
00:26:52,044 --> 00:26:56,624
Well, they haven't yet. I suppose I should step up or someone should step up
450
00:26:56,624 --> 00:27:00,204
and add these, but there isn't yet a BcacheFS version of these tools, right?
451
00:27:00,324 --> 00:27:01,264
There we go. That'd be cool.
452
00:27:01,424 --> 00:27:05,064
Obviously, probably what you want to do is just, you know, fork the existing
453
00:27:05,064 --> 00:27:08,404
ButterFS one and modify it and make it work right for BcacheFS.
454
00:27:08,624 --> 00:27:08,744
Sure.
455
00:27:08,944 --> 00:27:13,284
But if you want to be ad hoc about it, BPF Trace can handle this too.
456
00:27:13,324 --> 00:27:18,204
So I did TP list and then grepped that for things that said BcacheFS.
457
00:27:18,204 --> 00:27:22,244
So you got all of the kernel interfaces regarding trace points,
458
00:27:22,444 --> 00:27:24,584
as I should say, trace points regarding BcacheFS.
459
00:27:25,144 --> 00:27:28,544
So here's just like a list of some of what the trace points look like.
460
00:27:29,004 --> 00:27:33,364
I see. Okay. So now, okay, so what I'm seeing here on Wes's screen is like an
461
00:27:33,364 --> 00:27:38,624
output, BcacheFS colon, and I can get data update, rebalance, extent.
462
00:27:38,884 --> 00:27:41,944
There's a lot of different options here. Essentially, just what is the file system up to?
463
00:27:41,944 --> 00:27:46,644
And so these would all be things that Kent or team have put in explicitly so
464
00:27:46,644 --> 00:27:51,744
that there's a way to like easily and with low overhead watch these things.
465
00:27:51,964 --> 00:27:54,784
So since these trace points already exist, you can use...
466
00:27:56,807 --> 00:28:00,107
bcc to write your own kind of monitoring.
467
00:28:00,107 --> 00:28:02,367
Yeah or bpf trace which is right.
468
00:28:02,367 --> 00:28:03,187
Yeah thank you.
469
00:28:03,187 --> 00:28:06,307
Um there's a lot of terms there's a lot of bpf terms
470
00:28:06,307 --> 00:28:09,707
so here's one that stood out uh copy gc
471
00:28:09,707 --> 00:28:12,467
wait and bcashfs is a copy on write
472
00:28:12,467 --> 00:28:15,327
file system and it does this bucket based allocation and so
473
00:28:15,327 --> 00:28:18,047
it has what's called a copying garbage collector so as
474
00:28:18,047 --> 00:28:20,867
you're copying files it'll handle moving things around so
475
00:28:20,867 --> 00:28:24,007
that it can then like get good bucket allocation and
476
00:28:24,007 --> 00:28:28,167
defragment things on the fly that kind of thing so copy
477
00:28:28,167 --> 00:28:30,927
gc wait is a trace point that tells you when
478
00:28:30,927 --> 00:28:33,967
you're when bcachefs is waiting for copy gc
479
00:28:33,967 --> 00:28:36,787
to complete so it can be a reason that your
480
00:28:36,787 --> 00:28:39,707
file system is being slow or not responding especially if you're moving or copying
481
00:28:39,707 --> 00:28:46,227
files around so you can see on your screen uh just a little tiny script uh sudo
482
00:28:46,227 --> 00:28:51,187
bpf trace dash e and then you pass it a little string and we tell it we want
483
00:28:51,187 --> 00:28:56,267
to use the trace point bcachefs copy gc wait and then we have a little script here that,
484
00:28:56,947 --> 00:29:01,107
one of the inputs is the device and then it has little stuff that extracts the
485
00:29:01,107 --> 00:29:04,567
major and minor number from that you don't have to do that and then all it does
486
00:29:04,567 --> 00:29:09,987
is print what device we're waiting on and how much we're waiting yeah and so
487
00:29:09,987 --> 00:29:12,047
it's you know i don't know 10 lines of code.
488
00:29:12,047 --> 00:29:16,747
And you get a nice kind of structured output it's easy to human readable it
489
00:29:16,747 --> 00:29:19,967
tells you the total flushes the total buffers flushed you get you know the total
490
00:29:19,967 --> 00:29:22,227
output whatever whatever the stat is you're watching.
491
00:29:22,227 --> 00:29:26,727
And then so on my system i'm just doing it on you know one rudifest disc so
492
00:29:26,727 --> 00:29:29,407
i knew which disc it would be it was just kind of interesting to see like i
493
00:29:29,407 --> 00:29:32,807
went around and i dd'd some big files and copied them around and i could see
494
00:29:32,807 --> 00:29:36,347
like oh yeah right the operation completes and then immediately the print tells
495
00:29:36,347 --> 00:29:38,347
me that oh we waited that much and.
496
00:29:38,347 --> 00:29:41,487
The practical use here is you know in a rate array,
497
00:29:42,398 --> 00:29:47,298
It might be useful to discover that one of your disks is significantly underperforming the others.
498
00:29:47,458 --> 00:29:49,578
It could be indicative of a larger problem. It could be indicative of why you're
499
00:29:49,578 --> 00:29:50,498
having performance issues.
500
00:29:51,238 --> 00:29:54,518
So, like, it turns out that like this, there's actually probably a fair amount
501
00:29:54,518 --> 00:29:57,938
of situations where single trace points, just on their own, might be something
502
00:29:57,938 --> 00:29:58,838
you want to look at, right?
503
00:29:58,918 --> 00:30:03,098
So other ones for BcacheFS, write buffer flush. That's an important event.
504
00:30:03,358 --> 00:30:07,118
Or journal writes. You might be wanting to know stuff about how the journal works.
505
00:30:07,778 --> 00:30:10,678
but you can also use the scripting language and the
506
00:30:10,678 --> 00:30:13,778
fact that eBPF supports basic data structures like maps
507
00:30:13,778 --> 00:30:17,078
and histograms to make more
508
00:30:17,078 --> 00:30:20,658
complicated combined programs so i had an llm i
509
00:30:20,658 --> 00:30:24,038
fed it the output of that tplist stuff that told it the available trace all
510
00:30:24,038 --> 00:30:27,118
the traces you had and what the structures looked like so you had to write the
511
00:30:27,118 --> 00:30:34,158
right code that's clever and uh bpf trace makes it easy to have stuff that runs
512
00:30:34,158 --> 00:30:38,138
at an interval right so you can set up all the traces And then every five seconds,
513
00:30:38,478 --> 00:30:40,138
it can print out a little summary that it's done.
514
00:30:40,238 --> 00:30:44,978
And so each time it traces, it can update a variable, and then it can calculate latencies or deltas.
515
00:30:45,838 --> 00:30:50,018
So this is a quick one that it did that looks at that right buffer flush trace
516
00:30:50,018 --> 00:30:54,938
point, and it samples it over every five seconds, and then it tells you how many flushes happened,
517
00:30:55,538 --> 00:30:59,478
how many buffers were flushed, how many were skipped, how many were fast path
518
00:30:59,478 --> 00:31:02,778
flushes, and then the total amount of data that was flushed.
519
00:31:02,778 --> 00:31:06,898
So just like a quick way to get, you know, an accurate little like every five
520
00:31:06,898 --> 00:31:07,818
second little printout.
521
00:31:07,978 --> 00:31:09,158
And what's great is like, as
522
00:31:09,158 --> 00:31:13,078
far as I know, this is really putting no measurable strain on the system.
523
00:31:13,178 --> 00:31:14,918
It's one of the more performant ways to do it. Yeah.
524
00:31:14,998 --> 00:31:18,798
So you can really get deep insights without some of the overhead you sometimes
525
00:31:18,798 --> 00:31:19,838
get by that kind of monitoring.
526
00:31:20,677 --> 00:31:24,537
So then to kind of further stress just how far this whole having an AI help
527
00:31:24,537 --> 00:31:27,137
me out would go, I had an idea.
528
00:31:27,457 --> 00:31:30,257
I did ultimately tone it back a bit, but basically like, what if I wanted to
529
00:31:30,257 --> 00:31:33,097
monitor some of the mesh network traffic that was going on?
530
00:31:33,157 --> 00:31:35,877
There's a lot of options for that, but can this do this too?
531
00:31:36,337 --> 00:31:40,717
And so it built me a little script that you give it an interface name,
532
00:31:40,857 --> 00:31:42,377
like tailscale0, say, right?
533
00:31:42,377 --> 00:31:46,317
and then it'll do a similar thing where every five seconds it'll just look at
534
00:31:46,317 --> 00:31:51,097
that interface and it'll count sent packets sent bytes receive packets receive
535
00:31:51,097 --> 00:31:57,277
bytes and it'll do a send latency histogram for you on the sent packets yeah.
536
00:31:57,277 --> 00:32:01,477
So you get this you actually i mean it's not a gui but it's it's it's a bar
537
00:32:01,477 --> 00:32:03,037
graph on the command line.
538
00:32:03,037 --> 00:32:05,997
Yeah right and so the combination of the built-ins ebpf and
539
00:32:05,997 --> 00:32:08,877
then the built-ins into bpf trace which
540
00:32:08,877 --> 00:32:11,937
has some of this stuff to do nice histogram display and stuff from
541
00:32:11,937 --> 00:32:16,397
their print command um it's you know this is a little bit longer some of it's
542
00:32:16,397 --> 00:32:19,677
kind of because there's a print statement for each thing we're printing it's
543
00:32:19,677 --> 00:32:22,957
like five different things we're measuring right so it takes up some space on
544
00:32:22,957 --> 00:32:28,337
the screen but like it's a single page of code here so it's not a it's not crazy
545
00:32:28,337 --> 00:32:29,757
to start trying to understand it.
546
00:32:29,757 --> 00:32:33,397
Right i've never even seen some of this stuff but i under i could like the trace
547
00:32:33,397 --> 00:32:37,857
points they all make sense they're all it's just real plain english really it's
548
00:32:37,857 --> 00:32:40,357
really easy and then you have the print and the time interval.
549
00:32:40,357 --> 00:32:45,697
And so under the hood, it is tracing something called NetDevStartTransmit.
550
00:32:45,837 --> 00:32:46,097
Okay.
551
00:32:46,317 --> 00:32:50,357
So then it filters, it gets data, it filters on interface name from that data,
552
00:32:50,397 --> 00:32:53,457
and then it has a little counter it's keeping, so it increments the counter.
553
00:32:53,697 --> 00:32:58,997
And then it has a thread ID that you can use, and so it gets the current timestamp
554
00:32:58,997 --> 00:33:03,857
in nanoseconds and stores that in a map based on its thread ID.
555
00:33:04,437 --> 00:33:08,797
And then it also does a trace point for sent packets with NetDevTransmit,
556
00:33:08,797 --> 00:33:11,777
and then it grabs the start time if it exists for
557
00:33:11,777 --> 00:33:14,757
its thread id and then it can compute how
558
00:33:14,757 --> 00:33:17,557
long it took to send from that and then that's where it can use
559
00:33:17,557 --> 00:33:20,837
the histogram stuff to print out a nice little thing now that it's tracking
560
00:33:20,837 --> 00:33:27,637
latency and then it has another uh trace point to use uh net if receive skb
561
00:33:27,637 --> 00:33:31,557
which tracks incoming packets and then it just has a little bit of code here
562
00:33:31,557 --> 00:33:34,417
to kind of tie it together every five seconds and do the printout and then it
563
00:33:34,417 --> 00:33:37,117
clears all the counters so that it can do the next cycle.
564
00:33:39,294 --> 00:33:43,394
That's a neat little magic, like, superpower, a pocket of power that's in the
565
00:33:43,394 --> 00:33:44,474
Linux kernel there for this.
566
00:33:44,614 --> 00:33:48,314
Now, it does mean, right, like, you can't use it if you don't have trace points you care about.
567
00:33:48,374 --> 00:33:51,554
And so you have to start understanding some kernel internals and,
568
00:33:51,634 --> 00:33:53,494
like, what trace points might matter and what they mean.
569
00:33:54,134 --> 00:33:59,234
But if you have a specific problem or a mystery on your system that you're trying
570
00:33:59,234 --> 00:34:02,594
to look into and you're willing to try to chase down hypotheses using these
571
00:34:02,594 --> 00:34:04,134
tools, I think you could get pretty far.
572
00:34:04,274 --> 00:34:08,014
Yeah, that's for sure. And then, you know, there are actual complete products
573
00:34:08,014 --> 00:34:09,574
out there that are dipping into this stuff.
574
00:34:10,654 --> 00:34:15,674
I was reading online that there's actually several Kubernetes products that
575
00:34:15,674 --> 00:34:18,734
are tapping into eBPF to do things under the hood.
576
00:34:19,234 --> 00:34:24,634
Also, Falco, which is a real-time security monitoring application you can run on Linux.
577
00:34:24,714 --> 00:34:28,354
It says, Falco uses eBPF to monitor system calls and network events directly
578
00:34:28,354 --> 00:34:32,714
in the kernel, enabling rapid detection of anomalous behavior with low latency.
579
00:34:33,174 --> 00:34:36,854
This kernel-level approach is faster and less resource-intensive than the user
580
00:34:36,854 --> 00:34:37,814
space monitoring tools.
581
00:34:38,094 --> 00:34:41,734
There's also user space tracing you can do. So if you set it up right,
582
00:34:41,814 --> 00:34:45,754
you can trace JVM events or Ruby things or Python stuff.
583
00:34:46,394 --> 00:34:50,034
And so you have products now, too, that will use things like open telemetry
584
00:34:50,034 --> 00:34:51,274
or other tracing standards,
585
00:34:51,274 --> 00:34:55,194
and you can have a trace then that can have both the kernel side via the ePPF
586
00:34:55,194 --> 00:35:00,694
stuff and the user space stuff without necessarily having to do as much custom
587
00:35:00,694 --> 00:35:05,774
observability implementation in the code base because they can kind of do more dynamic.
588
00:35:07,114 --> 00:35:09,854
And you might be able to then see stuff, right, where like, oh,
589
00:35:10,054 --> 00:35:12,654
there was a problem on the kernel layer, and then that's why we started seeing
590
00:35:12,654 --> 00:35:14,574
increased latency in the application layer.
591
00:35:16,174 --> 00:35:20,594
But you can use that. There's a lot of open source products or open core things,
592
00:35:20,634 --> 00:35:24,154
and there's just a lot of sort of primitives in one layer above primitives that
593
00:35:24,154 --> 00:35:27,214
are probably already on your kernel.
594
00:35:27,534 --> 00:35:31,234
I wonder if it's possible people out there listening have already been using
595
00:35:31,234 --> 00:35:33,694
eBPF for a while and this is all old news to them.
596
00:35:34,094 --> 00:35:35,574
Who didn't tell us? Like, have
597
00:35:35,574 --> 00:35:38,234
you used it in what applications and what utilities has it been for you?
598
00:35:38,274 --> 00:35:42,794
I mean, we've gone through a few examples here, but like Wes's story with the
599
00:35:42,794 --> 00:35:48,354
VPS and my story with tracking down like a background desktop application,
600
00:35:48,814 --> 00:35:52,074
there's just these also very practical day-to-day uses for this stuff.
601
00:35:52,074 --> 00:35:55,014
that's nothing more than just using some of the pre-existing tools that you
602
00:35:55,014 --> 00:35:57,934
just run and tap into this, and you don't have to write any kind of scripting code.
603
00:35:58,394 --> 00:36:03,094
Yeah, I mean, it's been around for long enough now that it's well-packaged in most distros.
604
00:36:03,634 --> 00:36:06,634
Can you use eBPF to figure out why I have so many tabs open?
605
00:36:08,774 --> 00:36:11,554
Well, maybe we can add a trace point to Firefox to keep track at least.
606
00:36:15,247 --> 00:36:20,067
Jupiterbroadcasting.com slash river river is the most trusted spot in the u.s
607
00:36:20,067 --> 00:36:26,487
for individuals and businesses to buy send receive bitcoin and they make it
608
00:36:26,487 --> 00:36:30,427
easy in three simple steps jupiterbroadcasting.com slash river hey.
609
00:36:30,427 --> 00:36:33,587
I just got set up with them and it was in fact very easy.
610
00:36:33,587 --> 00:36:36,427
I think their best low-key feature actually
611
00:36:36,427 --> 00:36:39,507
has to do with cash just as an aside so bitcoin is
612
00:36:39,507 --> 00:36:42,247
dipping as we record and if oh man this is
613
00:36:42,247 --> 00:36:45,467
this was the moment so they have a 3.8 percent
614
00:36:45,467 --> 00:36:48,307
interest cash savings account and they pay the interest
615
00:36:48,307 --> 00:36:51,587
out in sats so you put your cash in there and it's
616
00:36:51,587 --> 00:36:55,407
fdic insured 3.8 percent interest and
617
00:36:55,407 --> 00:36:58,407
then when the when the price dips on bitcoin you can use that balance
618
00:36:58,407 --> 00:37:03,187
to smash buy and so you can you can essentially dca with that 3.8 percent interest
619
00:37:03,187 --> 00:37:08,187
and then you can smash buy when the price dips it's just brilliant and river
620
00:37:08,187 --> 00:37:13,227
is a really great company i had a call with their community management person
621
00:37:13,227 --> 00:37:15,607
and was really impressed with what they said.
622
00:37:15,927 --> 00:37:19,367
And if you're in Canada and you're looking for another trusted source to get
623
00:37:19,367 --> 00:37:22,127
sats, bitcoinwell.com slash Jupiter.
624
00:37:22,367 --> 00:37:24,907
There you go. That's our tip.
625
00:37:28,187 --> 00:37:32,907
Well, since this week we are hoarding our boost for next week's episode,
626
00:37:32,907 --> 00:37:36,407
we decided to do a little dive into the old mailbag.
627
00:37:36,787 --> 00:37:40,267
Zach sent us a little note here. I'm behind on listening to episodes.
628
00:37:40,447 --> 00:37:45,287
But the NixOS episode from last month, Linux Unplugged 5.9.8,
629
00:37:45,567 --> 00:37:47,567
was really interesting.
630
00:37:47,747 --> 00:37:51,047
I wanted to throw out some thoughts responding to a comment that was sent in
631
00:37:51,047 --> 00:37:57,907
about wanting a NixOS-like system, but being able to use the old traditional system admin tools.
632
00:37:58,487 --> 00:38:02,347
For context, I've tried out NixOS, but quickly got to the point where I would
633
00:38:02,347 --> 00:38:04,887
need to dive in and figure out those flakes.
634
00:38:05,147 --> 00:38:08,327
At that point, it just became one more thing on that list to do.
635
00:38:08,327 --> 00:38:11,147
Since then, I've gotten into bootable containers.
636
00:38:11,447 --> 00:38:15,107
I've specifically been using the Universal Blue ecosystem with Bootsy,
637
00:38:15,347 --> 00:38:20,207
but have wanted to try out Vanilla OS and Common Arch that also use OCI images
638
00:38:20,207 --> 00:38:21,787
for delivery and customization.
639
00:38:22,407 --> 00:38:27,347
Listening to that NixOS episode, I felt like many of those talking points that
640
00:38:27,347 --> 00:38:32,267
were given in favor of NixOS were also very in line with the benefits that I've
641
00:38:32,267 --> 00:38:36,527
seen in using OCI images to build out customized OS images.
642
00:38:37,818 --> 00:38:41,298
I don't use any of the headliner distros like aurora or
643
00:38:41,298 --> 00:38:44,618
bluefin although i have tried a few and overall they were quite
644
00:38:44,618 --> 00:38:47,378
well put together but i've taken their more
645
00:38:47,378 --> 00:38:50,718
low-level image builds and layered my own customizations
646
00:38:50,718 --> 00:38:55,878
on top of them it's been a fantastic way to manage everything as when i need
647
00:38:55,878 --> 00:38:59,518
a new server or desktop all i have to do is install my image and everything's
648
00:38:59,518 --> 00:39:05,098
ready to go i will add the caveat that there may be some learning if you haven't
649
00:39:05,098 --> 00:39:07,938
already been familiar with building OCI containers,
650
00:39:08,238 --> 00:39:13,378
but I live in a world where professionally and for hobbies, that's what I'm doing.
651
00:39:13,578 --> 00:39:15,918
So that wasn't a huge burden for me to learn.
652
00:39:16,458 --> 00:39:20,158
Thanks for the show. Really have been enjoying it and wanted to send this little
653
00:39:20,158 --> 00:39:23,938
message in just in case someone else might find some use.
654
00:39:24,478 --> 00:39:27,738
Well, cool. Thank you for sharing. We love success stories like this.
655
00:39:27,838 --> 00:39:29,998
I wonder if you have any of it on GitHub or anything too.
656
00:39:30,138 --> 00:39:33,038
If people are curious. Good question. I really
657
00:39:33,038 --> 00:39:35,738
feel like this nails something that we've been feeling and chatting a
658
00:39:35,738 --> 00:39:40,978
lot about behind the scenes is uh the really
659
00:39:40,978 --> 00:39:43,978
really brilliant thing that the universal blue folks
660
00:39:43,978 --> 00:39:50,818
have tapped into is this existing knowledge set around how oci images work and
661
00:39:50,818 --> 00:39:55,478
how you can layer things and really tapped into the whole devops workflow that
662
00:39:55,478 --> 00:39:59,678
people live every day and now you can use it that skill set that you've learned
663
00:39:59,678 --> 00:40:03,638
and you can use it to customize your Linux desktop. I mean, that is just really powerful.
664
00:40:04,438 --> 00:40:07,538
It's a different approach than Nix, right? Whereas Nix, you're building it from
665
00:40:07,538 --> 00:40:12,418
the ground up. You would use Nix to maybe generate those OCI images or something like that.
666
00:40:12,598 --> 00:40:16,318
But both, like he says, are essentially accomplishing similar goals.
667
00:40:16,598 --> 00:40:22,378
Yeah, and a lot of the immutability side of things and the image-based approach, right?
668
00:40:22,498 --> 00:40:27,438
Like starting to think of your system as a cohesive whole that you deliver in
669
00:40:27,438 --> 00:40:31,238
those coherent units rather than ad hoc imperative updates.
670
00:40:31,378 --> 00:40:35,258
Now, admittedly, I don't use them as much, but Brent, when I think of an image-based
671
00:40:35,258 --> 00:40:37,458
system, I don't necessarily think of flexibility.
672
00:40:38,118 --> 00:40:42,178
Yeah, I was kind of wondering about this. I've had a few people tell me,
673
00:40:42,258 --> 00:40:43,878
oh, yeah, images is totally the way to go.
674
00:40:44,358 --> 00:40:47,558
Of course, we've been playing with Nix and NixOS at the same time,
675
00:40:47,558 --> 00:40:50,598
and the thing I keep tripping on with images is...
676
00:40:51,779 --> 00:40:55,419
You build them once and then you use them many times. But the way,
677
00:40:55,599 --> 00:41:00,379
Chris, I think you and I have been using at least NixOS on the desktop is your
678
00:41:00,379 --> 00:41:02,599
Nix config is like constantly evolving.
679
00:41:03,059 --> 00:41:06,379
Therefore, the next time you go to install it somewhere else,
680
00:41:06,519 --> 00:41:08,699
it's always that new updated copy.
681
00:41:08,819 --> 00:41:12,279
And I would imagine one of the downsides with building an image is,
682
00:41:12,679 --> 00:41:14,159
well, you have that intermediate step.
683
00:41:14,279 --> 00:41:18,039
You have to go ahead and build your image every time you want that newest, freshest version.
684
00:41:18,179 --> 00:41:20,979
Am I understanding that correctly? Is that a proper downside here?
685
00:41:20,979 --> 00:41:25,439
I know some of the silver blue types, right, they're still using a lot of RPM
686
00:41:25,439 --> 00:41:28,079
OS tree, which has ways to apply things live.
687
00:41:28,519 --> 00:41:31,399
So it might depend a little bit on how exactly what you're putting in those
688
00:41:31,399 --> 00:41:34,499
images, or maybe you have like a dynamic switch image thing.
689
00:41:34,659 --> 00:41:35,719
But yeah, I do wonder about that.
690
00:41:35,839 --> 00:41:37,479
And then you also got to remember, it depends on what you're doing.
691
00:41:37,699 --> 00:41:40,419
You know, it might just be maybe a lot of your applications are flat packs and
692
00:41:40,419 --> 00:41:42,279
you're installing those and updating those separately anyways.
693
00:41:43,119 --> 00:41:47,919
Or you're using, you know, various containers or other things for your dynamic environments.
694
00:41:48,479 --> 00:41:51,619
I'll take Liam's email here. He says, good day, gents. In the most recent LUP,
695
00:41:51,719 --> 00:41:54,419
you mentioned not having received much feedback about multi-monitor setup.
696
00:41:54,499 --> 00:41:55,539
So he sent us a picture of his.
697
00:41:55,779 --> 00:42:00,199
I've got a laptop screen at 1080 at 144 hertz. That's my primary display,
698
00:42:00,359 --> 00:42:01,239
you know, for things like Skype.
699
00:42:01,599 --> 00:42:04,639
Then I have two externals, a 32-inch 2K.
700
00:42:05,399 --> 00:42:08,979
I do think that's the sweet spot, he says, at 60 hertz. And then my rightmost
701
00:42:08,979 --> 00:42:10,159
monitor is in portrait mode.
702
00:42:10,519 --> 00:42:15,139
That's where I can see more lines of code. All of this works without issue on XFCE.
703
00:42:15,539 --> 00:42:19,079
In case you do navigate to the image, yeah, that's a treadmill desk.
704
00:42:19,079 --> 00:42:22,119
I'm on my second one since the start of the pandemic this.
705
00:42:22,119 --> 00:42:23,099
Is a nice setup.
706
00:42:23,099 --> 00:42:27,759
Listeners since Sidebyte, Lunduk and Last Days and Early Tech Snap member since
707
00:42:27,759 --> 00:42:33,539
2022 annual member since December that's awesome thank you very much Liam and
708
00:42:33,539 --> 00:42:37,799
I like your setup I am as you boys know big fan of the vertical monitor,
709
00:42:38,962 --> 00:42:44,422
For our show docs or, you know, like reviewing a config file or just having a terminal up there.
710
00:42:45,382 --> 00:42:46,102
Journal D.
711
00:42:46,822 --> 00:42:50,322
Yep. The vertical monitor is a productivity hack.
712
00:42:50,362 --> 00:42:54,762
If you can afford the luxury of, because you're not, it's not great.
713
00:42:54,762 --> 00:42:58,922
But one of the other things I will use the vertical monitor for is I stack two chats.
714
00:42:59,242 --> 00:43:01,682
Like if I have two different work chats or something, I'll stack them on the
715
00:43:01,682 --> 00:43:03,382
vertical monitor. Works really great.
716
00:43:03,542 --> 00:43:07,902
Right. Yeah. It doesn't fit every activity. So it's sort of a less generally useful screen at times.
717
00:43:08,062 --> 00:43:12,502
Yeah. Yeah. It is more limited, so it's more of a luxury. I do have to admit that.
718
00:43:12,682 --> 00:43:18,502
I have a bit of a question here for us. Did we ever divulge what our current setups are?
719
00:43:18,602 --> 00:43:22,042
Chris, I know you've gone into details, but I don't think Wes and I have actually shared that.
720
00:43:22,422 --> 00:43:25,242
Wes, I bet you kind of move around depending on, because you're mostly using laptops.
721
00:43:25,622 --> 00:43:31,662
I do a lot of laptop. I've got a dual monitor set up in the sort of main office-y
722
00:43:31,662 --> 00:43:38,442
bit, and then I have just a single where I do laptop with one screen in my living room sort of desk.
723
00:43:38,442 --> 00:43:40,482
You got a monitor at the living? That's nice.
724
00:43:40,762 --> 00:43:43,842
That's for the like casual work where I want to do stuff, but with the TV on.
725
00:43:43,942 --> 00:43:46,162
Yeah. Okay, Brent, what's your setup?
726
00:43:46,622 --> 00:43:49,882
Well, I don't actually do dual monitor anymore.
727
00:43:51,062 --> 00:43:55,222
I am now doing the tri-monitor thing. Chris, I know you're like a quad monitor
728
00:43:55,222 --> 00:44:01,222
guy, but I did steal your little tip. You've been trying to get me to use this for years.
729
00:44:01,282 --> 00:44:06,362
I have same as Liam here on the right-hand side, a vertical monitor.
730
00:44:06,522 --> 00:44:08,802
It's just an old monitor I've had forever. it doesn't,
731
00:44:10,018 --> 00:44:12,958
matter though because it's just chats or like currently
732
00:44:12,958 --> 00:44:15,778
i have our episode recorder there i
733
00:44:15,778 --> 00:44:18,678
have the window on which we're connected together
734
00:44:18,678 --> 00:44:21,458
and then i also have the jb chat even though we're not doing
735
00:44:21,458 --> 00:44:24,098
it live for some reason it just makes me feel better and it's
736
00:44:24,098 --> 00:44:28,258
perfect for that and so i've got three monitors running usually for me off a
737
00:44:28,258 --> 00:44:34,478
laptop and got this big massive 20 well i say massive although liam beat me
738
00:44:34,478 --> 00:44:39,258
on this one but it's this 27 inch like 4k dell monitor right as my main laptop
739
00:44:39,258 --> 00:44:42,798
as my main monitor in front of me and that's been a pretty sweet setup.
740
00:44:42,798 --> 00:44:44,538
I'm on to him.
741
00:44:44,538 --> 00:44:45,178
He asked.
742
00:44:45,178 --> 00:44:46,478
This question because he wanted to.
743
00:44:46,478 --> 00:44:49,298
Answer exactly and i'm here for yeah
744
00:44:49,298 --> 00:44:51,978
i love it um you know i've been i know i mentioned this
745
00:44:51,978 --> 00:44:54,798
recently on the show but i have i definitely have always
746
00:44:54,798 --> 00:44:57,718
been a multi-monitor person for many many many years now but i
747
00:44:57,718 --> 00:45:01,338
have been enjoying the single extra extra
748
00:45:01,338 --> 00:45:04,418
ridiculously wide screen it's actually a curved wide
749
00:45:04,418 --> 00:45:10,258
screen at home and uh and i use gnome there as well and i i really like i mean
750
00:45:10,258 --> 00:45:14,718
you can you can lay out three full windows side by side on the screen it's just
751
00:45:14,718 --> 00:45:17,638
really you get a lot of room and when you start getting that kind of room you
752
00:45:17,638 --> 00:45:23,938
don't really need a second screen as much and i will say it is a much easier way to do desktop linux.
753
00:45:23,938 --> 00:45:27,178
I do think that'll be in one of my future monitor purchases for sure.
754
00:45:27,178 --> 00:45:30,698
If you go with like an intel or an amd gpu right now maybe one day in the video
755
00:45:30,698 --> 00:45:35,378
But Intel and AMD and a single monitor, I promise you, your life will be easier
756
00:45:35,378 --> 00:45:38,538
than if you try to be a quad monitor guy. It's not an easy path.
757
00:45:38,718 --> 00:45:41,318
But what if I want to chain DisplayPort together?
758
00:45:41,638 --> 00:45:47,758
Yeah. Okay. Yeah. Oh, man. And, like, I don't know what my deal is because I've
759
00:45:47,758 --> 00:45:50,298
only been using computers for, like, 30-plus years.
760
00:45:50,698 --> 00:45:56,638
But, like, I still have major strugs trying to make sure that the correct monitor
761
00:45:56,638 --> 00:45:59,158
is the default, like, bioscreen monitor.
762
00:45:59,178 --> 00:45:59,718
Oh, gosh.
763
00:46:00,738 --> 00:46:03,478
And I don't know maybe this isn't how it works,
764
00:46:05,158 --> 00:46:10,418
hand to the mixer if it doesn't change i get it just set up right and then like
765
00:46:10,418 --> 00:46:13,958
a couple of days go by and the next time i turn the machine on like the different
766
00:46:13,958 --> 00:46:18,238
monitors lighting up and i've gone through and i've i i thought i did the proper
767
00:46:18,238 --> 00:46:22,418
ordering you know in the plugs and it changes i swear that happens i don't know
768
00:46:22,418 --> 00:46:23,518
maybe maybe i'm making it up.
769
00:46:23,518 --> 00:46:26,638
You need to like use uuids for your monitor setup.
770
00:46:26,638 --> 00:46:31,518
Yeah well it's at the bios level like you know props to plasma by the time i
771
00:46:31,518 --> 00:46:32,838
get to Plasma, it's always fine.
772
00:46:34,018 --> 00:46:37,198
And I've been actually having really good success with locking my screens.
773
00:46:37,198 --> 00:46:41,838
They go to sleep. They wake back up. Everything's fine. All the orientations are correct.
774
00:46:42,458 --> 00:46:48,378
And when it boots, it always nails it. So Plasma has been just doing great with my multi-moder setup.
775
00:46:48,638 --> 00:46:52,078
But my BIOS, right? Or like when the system's booting, the console where you're
776
00:46:52,078 --> 00:46:57,858
just seeing the output, like that is a moving target for some reason.
777
00:46:57,898 --> 00:47:02,918
And I don't think that's technically how this works but maybe it's me you know
778
00:47:02,918 --> 00:47:05,058
moving monitors around i don't know you.
779
00:47:05,058 --> 00:47:06,438
Could take the next year off.
780
00:47:06,438 --> 00:47:06,938
And just.
781
00:47:06,938 --> 00:47:08,258
Rewrite your bios.
782
00:47:08,258 --> 00:47:12,778
Oh i thought maybe i'd do a study where i just empirically like write every
783
00:47:12,778 --> 00:47:17,238
move i make down okay this one's in this plug boot yeah you write that step one i.
784
00:47:17,238 --> 00:47:20,278
Think maybe you should study the cameras that you have in the studio because
785
00:47:20,278 --> 00:47:23,678
every time i show up i tend to shuffle all your cables around for your monitors.
786
00:47:23,678 --> 00:47:26,518
That is true that is true,
787
00:47:28,852 --> 00:47:33,372
Okay, well, the pick this week has to be something that gets you up and running
788
00:47:33,372 --> 00:47:35,692
with BPF right away, right?
789
00:47:35,832 --> 00:47:38,332
Like, we don't want to sit here and talk about eBPF this whole episode and not
790
00:47:38,332 --> 00:47:39,792
give you a great tool to check out.
791
00:47:40,052 --> 00:47:46,072
And this week, it's Network Top. It helps you monitor traffic from your network using BPF.
792
00:47:46,232 --> 00:47:49,832
Yeah, we should be clear. This one is classic BPF, but it's still handy.
793
00:47:49,832 --> 00:47:52,012
And that means it is broadly useful, too.
794
00:47:52,172 --> 00:47:52,352
Yeah.
795
00:47:52,692 --> 00:47:53,792
It's built in Rust.
796
00:47:54,192 --> 00:47:54,872
Oh, it is.
797
00:47:54,952 --> 00:47:58,652
Yeah, so it's super performant. It's easy to use.
798
00:47:58,812 --> 00:48:05,292
And it means you get to use like the TCP dump style syntax in a nice little Tui.
799
00:48:05,512 --> 00:48:08,692
And so I sent you an example. What do you think of it?
800
00:48:09,012 --> 00:48:12,052
Well, first of all, it's very readable, right? Because it is,
801
00:48:12,132 --> 00:48:15,492
like you said, you're getting charts in a terminal UI.
802
00:48:16,492 --> 00:48:20,832
And I mean, it's immediately understandable. I'm trying to figure out what you're
803
00:48:20,832 --> 00:48:22,452
doing here because I see ginormous,
804
00:48:22,612 --> 00:48:28,112
like you have almost no traffic and then it jumps up to almost 26.3 megabytes
805
00:48:28,112 --> 00:48:34,032
a second and it hovers there for about 40 seconds and then it drops down to absolutely nothing.
806
00:48:34,032 --> 00:48:37,652
Yeah so you'll you see how there's okay there's an input section at the top
807
00:48:37,652 --> 00:48:41,852
and then under that there's rules yeah you'll see what it says all on the left
808
00:48:41,852 --> 00:48:45,132
and then to the right it says host and then a specific ip address.
809
00:48:45,132 --> 00:48:47,872
So you're you're looking at all traffic from just this host.
810
00:48:47,872 --> 00:48:52,172
Yes um and so those are two separate rules so basically it's it's looking at
811
00:48:52,172 --> 00:48:57,032
one interface and just watching it totally by default and then you can add bpf
812
00:48:57,032 --> 00:49:00,252
rules that it'll add as additional things that it'll monitor at the same time
813
00:49:00,252 --> 00:49:05,832
and then you can use the arrow keys to toggle between the views for different uh rules okay and,
814
00:49:07,244 --> 00:49:11,704
Um, there was no traffic because, uh, I SSH to another machine here at the studio.
815
00:49:12,044 --> 00:49:14,784
We hadn't really been talking, right? I exchanged a few packets for that, but that was all.
816
00:49:14,984 --> 00:49:20,724
Um, and then I did a net cap back to my laptop, just reading from random.
817
00:49:21,324 --> 00:49:24,904
Uh, and so that was the big traffic spike. And then I killed it just to watch it all drop off.
818
00:49:25,344 --> 00:49:29,144
But you know, like it could, you can do, you can do like specific ports that you want to watch.
819
00:49:29,264 --> 00:49:33,444
You can do specific IPs, host names, TCP states, like whatever you can do with
820
00:49:33,444 --> 00:49:35,924
TCP dump. Basically you can put those rules in here.
821
00:49:35,924 --> 00:49:40,444
And you're getting a command line TUI, a terminal user interface.
822
00:49:40,724 --> 00:49:44,024
So it's really easy to understand. Like, these rules Wes are talking about are
823
00:49:44,024 --> 00:49:47,624
clearly delineated in the UI. It's really simple syntax.
824
00:49:47,904 --> 00:49:49,804
So maybe there's something you're trying to hunt for. You find it here,
825
00:49:49,824 --> 00:49:53,404
and then you can go do the TCP dump to capture the traffic and do some more analysis.
826
00:49:53,604 --> 00:49:57,824
And it is MIT licensed, so it is free to use.
827
00:49:58,244 --> 00:50:03,344
And then there is a bonus pick, BPF Tune.
828
00:50:03,504 --> 00:50:06,944
Yeah, well, you kind of hinted at this a bit. And I happen to know you turned
829
00:50:06,944 --> 00:50:08,744
it on for at least one of your systems.
830
00:50:08,744 --> 00:50:10,324
I have it running on my home workstation.
831
00:50:10,444 --> 00:50:15,764
Despite the fact that this is indeed a GPL 2.0 with Linux Syscall Note licensed
832
00:50:15,764 --> 00:50:17,584
Oracle open source project.
833
00:50:18,484 --> 00:50:24,304
BPF Tune aims to provide lightweight, always-on auto-tuning of system behavior.
834
00:50:24,564 --> 00:50:28,484
The key benefits it provides is by using BPF observability features,
835
00:50:28,484 --> 00:50:32,784
it continuously monitors and adjusts system behavior. because we can observe
836
00:50:32,784 --> 00:50:34,344
the system behavior at a fine grain.
837
00:50:35,084 --> 00:50:40,324
We can then tune at a finer grain too. So like individual socket policies,
838
00:50:40,804 --> 00:50:42,164
individual device policies.
839
00:50:42,444 --> 00:50:46,224
Yeah, right. So think of all those things like CTL can tweak.
840
00:50:46,764 --> 00:50:51,564
It's using BPF hooks to watch your system and then automatically go tweak those.
841
00:50:51,764 --> 00:50:55,244
Yeah. Now I haven't really noticed a difference, but I've only had it on for a couple of days.
842
00:50:55,404 --> 00:50:58,744
But the idea is brilliant. It's just so brilliant.
843
00:50:58,884 --> 00:51:01,604
Like here's the system. I'm monitoring. OK, now I'll just go make adjustments
844
00:51:01,604 --> 00:51:05,444
here to make essentially maybe like ease pressure on the system or something like that.
845
00:51:06,062 --> 00:51:08,742
I don't know if anybody has experience with this, because I just started,
846
00:51:08,942 --> 00:51:10,862
but it seems like a fantastic idea.
847
00:51:11,602 --> 00:51:14,442
BPF tune. Chris will have a link to these in the show notes.
848
00:51:16,042 --> 00:51:20,342
And in Nix, it's like just one, it's like, you know, service enable BPF auto
849
00:51:20,342 --> 00:51:21,642
tune. And then you just, you're good.
850
00:51:23,502 --> 00:51:26,702
There's more you can do, but that's, it's really simple. So I was like,
851
00:51:26,742 --> 00:51:27,802
all right, I'm just going to turn this on.
852
00:51:27,942 --> 00:51:31,622
And I just love the concept of the system auto tuning itself.
853
00:51:31,622 --> 00:51:35,482
Yeah, and they kind of make the case here too, if you're doing the cattle not
854
00:51:35,482 --> 00:51:40,282
pets approach and how many of your systems ever really even have a human person
855
00:51:40,282 --> 00:51:41,922
who might be on it who could tune it.
856
00:51:42,302 --> 00:51:44,902
Okay, maybe you can hand-tune the database, but you're not going to hand-tune
857
00:51:44,902 --> 00:51:46,622
all your dynamic web workers.
858
00:51:46,802 --> 00:51:48,162
This could be great for my Odroid.
859
00:51:48,322 --> 00:51:52,182
But if some of them are longer lived, this king can make sure they're running all right.
860
00:51:52,322 --> 00:51:55,782
I should put this on my Odroid. I really should. Little Odroid's doing a lot
861
00:51:55,782 --> 00:51:59,282
of work and I never really check in on it. Just sitting there being a little yeoman about it.
862
00:51:59,702 --> 00:52:03,862
Well, we will have links to this and everything else. There's a lot of links this week.
863
00:52:04,882 --> 00:52:11,042
Linuxunplugged.com slash 605. And remember, we want to know if you enjoy these deep dives.
864
00:52:11,202 --> 00:52:13,942
We can get into the weeds here a bit too much, but if this is the kind of stuff
865
00:52:13,942 --> 00:52:18,182
you like, let us know because there's, it's honestly for us as creators,
866
00:52:18,342 --> 00:52:22,062
it's a little, creators, it's a little scary to do topics like this.
867
00:52:22,162 --> 00:52:24,422
I know that sounds stupid, but it is. It's just a little scary.
868
00:52:24,722 --> 00:52:26,462
So we always like to know what your thoughts are.
869
00:52:27,202 --> 00:52:29,222
We'll be back at our regular live time.
870
00:52:30,022 --> 00:52:33,802
Tuesdays as in Sunday Sunday at 12 p.m. Pacific, 3 p.m. Eastern.
871
00:52:38,642 --> 00:52:43,962
Now if you want more show remember our members get the full bootleg which is
872
00:52:43,962 --> 00:52:48,342
clocking in at like an hour and 20 minutes or something right now and this is a short one,
873
00:52:49,162 --> 00:52:54,142
and of course you can get details of that at linuxunplugged.com Thank you so
874
00:52:54,142 --> 00:52:57,842
much for joining us on this week's episode of the Unplugged program we just
875
00:52:57,842 --> 00:53:00,922
really appreciate your time for listening if you want to share it with somebody
876
00:53:00,922 --> 00:53:04,242
we always like that too word of mouth is the best advertising for a podcast
877
00:53:04,242 --> 00:53:05,402
we always appreciate that,
878
00:53:05,862 --> 00:53:09,722
thank you so much for being here and we'll see you right back here next Tuesday
879
00:53:09,722 --> 00:53:12,722
as in Sunday which isn't Tuesday at all.