Episode 31 - Container Security with Josh Duffney

WEBVTT
00:00:07.471 --> 00:00:14.326
Hello, welcome to Microsoft Community Insights Podcast, where we share insights from community experts to stay up to date in Microsoft.
00:00:14.326 --> 00:00:16.106
I am Nicholas and I'll be your host today.
00:00:16.106 --> 00:00:27.868
In this podcast, we will dive into the world of container security, but before we get started, I want to remind you to follow us on social media so you never miss an episode, and it will help us reach more amazing people like yourself.
00:00:27.868 --> 00:00:31.190
Today we have a special guest called Josh Duffy.
00:00:31.190 --> 00:00:33.125
Could you please introduce yourself, please?
00:00:33.366 --> 00:00:33.829
Absolutely.
00:00:33.829 --> 00:00:34.762
First, the intro.
00:00:34.762 --> 00:00:37.609
That's the longest five seconds I've ever experienced.
00:00:37.609 --> 00:00:39.232
Very exciting, I'm just kidding.
00:00:39.232 --> 00:00:40.526
Yeah, so my name is Josh Duffy.
00:00:40.526 --> 00:00:42.868
I'm a senior cloud advocate at Microsoft.
00:00:45.259 --> 00:00:47.847
My background we were chatting a little bit about this before we went live.
00:00:47.847 --> 00:00:50.240
Let's see here I'll give this a succinct version, if I can.
00:00:50.240 --> 00:00:51.143
We can dive into more.
00:00:51.143 --> 00:01:00.322
So I wanted to be a video game designer because I played a lot of video games in high school, decided I didn't want to have $100,000 of student loan debt right starting life.
00:01:00.322 --> 00:01:09.341
So I decided to go to a community college and the only thing computer related they had was this computer networking degree, an associate's degree, and it turned out to be kind of an introduction to everything.
00:01:09.341 --> 00:01:15.286
I had a networking class, a full semester on printers, which was very painful I still dislike printers to this day.
00:01:15.286 --> 00:01:29.632
But anyway I landed a help desk job from there, got really into like system administration, worked my way up to like a one man IT shop running a small bank and racking servers, and then I kind of got into PowerShell and so a lot of people have seen my name before.
00:01:29.632 --> 00:01:31.986
It's probably because of the stuff that I used to do with PowerShell.
00:01:31.986 --> 00:01:35.025
But anyway, that led me kind of on the DevOps path.
00:01:35.246 --> 00:01:48.995
Previous to Microsoft I was an SRE at Stack Overflow and then I got recruited in by Mike Robbins, who's familiar in the PowerShell community too, to be a docs writer or a content developer, I think is the technical term for that, and I focused on Ansible and Terraform.
00:01:48.995 --> 00:01:54.129
I had written a book on Ansible and successfully implemented it in a couple enterprises and stuff previously.
00:01:54.129 --> 00:02:09.447
I did the docs writing for about 18 months and then I got the opportunity to join the cloud native team here at Microsoft in the DevRel organization for advocacy work, which was a good split for me between some of the content creation and speaking which I enjoy doing, and I'd done that a lot in my past.
00:02:09.447 --> 00:02:16.240
I ran a PowerShell user group and other user groups before, spoken at many things, different conferences, so it seemed like a good fit.
00:02:16.240 --> 00:02:22.770
And then I landed somehow in container security for probably like the past two years or so.
00:02:22.770 --> 00:02:27.701
So that's like I don't know how long have I been working Like 12 years, I don't know.
00:02:27.701 --> 00:02:28.986
12-year snapshot right there.
00:02:29.520 --> 00:02:33.448
So before you land your job at Microsoft and you are in Stack Overflow.
00:02:33.448 --> 00:02:41.152
Did you have experience in Cloud Native or just be using PowerShell to create like meetups?
00:02:43.604 --> 00:02:47.968
I wouldn't have put it under like cloud native, but I definitely had a lot of cloud experience.
00:02:47.968 --> 00:02:50.219
What's the introduction to Azure?
00:02:50.219 --> 00:02:57.308
So before Stack Overflow, the company I worked for had AWS and Azure, had to work in both clouds and we had on-prem so we had to do all that.
00:02:57.308 --> 00:03:14.532
So I'd been working in the cloud for a number of years, probably before the term cloud native came about and just I don't know, it seemed like the things that I was naturally doing in the cloud and then having them change, I suppose, from on-prem and stuff, and then it just became a term.
00:03:14.532 --> 00:03:23.104
So it's one of those situations where you've just been doing a thing for a while and then all of a sudden, uh, the broader ecosystem comes up with the term for what it is that you've been doing for a while.
00:03:23.705 --> 00:03:28.375
Yeah, because I remember watching some of your episode on Bash.
00:03:28.375 --> 00:03:30.907
You know Bash as well as PowerShell.
00:03:30.907 --> 00:03:32.010
It's quite handy.
00:03:32.621 --> 00:03:34.067
Yeah, I got forced into that once.
00:03:34.067 --> 00:03:36.788
I Very interesting paradigm shift.
00:03:36.788 --> 00:03:45.131
So outside Microsoft was Windows System Administrator, powershell guru doing DSC like even wrote my own.
00:03:45.131 --> 00:03:49.128
Actually, dscv3 just came out and they have the ability to do JSON representation.
00:03:49.128 --> 00:03:56.165
Back in 2016, I did a talk where I wrote a module that abstracted DSC into JSON, so that's pretty cool to see that.
00:03:56.165 --> 00:04:00.001
I don't think they used any of my work or anything like that, but just that idea transferred.
00:04:00.001 --> 00:04:00.401
Yeah.
00:04:00.760 --> 00:04:09.927
Okay, so today's session is the world of container security, so let's talk about the main, like the introduction.
00:04:09.927 --> 00:04:12.729
So what's container security and why is it important?
00:04:12.889 --> 00:04:18.994
The area that I've been working in container security is pretty broad, so I'll narrow it down to the areas that I've been working in.
00:04:18.994 --> 00:04:35.041
I've been working with a couple open source projects Notation, which is a digital signer you can think of it that way yeah, and then they're ratify, which is an emission controller for Kubernetes, and then copacetic, so that would be for the patching, and all those are related to the software supply chain and containers.
00:04:35.041 --> 00:04:43.810
So container security most people would think immediately think chain guard that's really focused on the images and make sure that those images in the best possible way.
00:04:43.810 --> 00:04:45.052
That's a big part of it.
00:04:45.052 --> 00:04:48.009
A lot of the work that I've been doing is on somewhat on the other side.
00:04:48.290 --> 00:04:52.331
Right, okay, we have this good image that we trust or want to trust.
00:04:52.331 --> 00:04:53.625
How do we signal that trust?
00:04:53.625 --> 00:04:55.466
And that's where the signing comes into play.
00:04:55.466 --> 00:05:00.206
And then how do we know what's in here and that would be something like an S-bomb and all of these are.
00:05:00.206 --> 00:05:08.769
It's been an interesting thing to watch in the industry because it's more work and definitely not something that the average developer wants is more work.
00:05:08.769 --> 00:05:21.709
But there is a lot of pressure from especially like government agencies and working contracts with government agencies where they're now mandating these types of things to improve the supply chain, to avoid supply chain attacks from happening.
00:05:21.709 --> 00:05:23.526
It's an evolving ecosystem.
00:05:24.062 --> 00:05:32.545
One of the important points is keeping your images up to date and don't have updated images, because it would lead to having vulnerabilities in your code.
00:05:32.841 --> 00:05:35.565
So yeah, I never writing.
00:05:35.565 --> 00:05:49.072
So I wrote a series on Copasetic, which is a patching tool, and in writing that it just brought me back to my first kind of epiphany as I was learning Kubernetes and getting into containers, when Docker was really starting to come on the scene.
00:05:49.072 --> 00:05:58.899
And one of my first thoughts I remember because I had battled Windows updates like my entire career VM updates and I was like, oh, finally I'm not going to have to worry about this problem again.
00:05:58.899 --> 00:06:00.305
Here we are, 10 years later.
00:06:00.305 --> 00:06:07.706
I'm having to worry about, you know, vulnerabilities of my images that I have to patch or you can patch by updating the base image.
00:06:07.706 --> 00:06:13.526
We can get into more detail with that later, but at the end of the day there's still vulnerabilities in these container images and systems.
00:06:13.526 --> 00:06:17.182
It didn't solve the patching problems per se Exactly.
00:06:17.182 --> 00:06:22.622
You know like we still have to have systems around updating things to make them not vulnerable.
00:06:22.862 --> 00:06:29.322
Okay, so the free tool, Copasetica, and the other one take it there.
00:06:29.322 --> 00:06:31.730
Cncf approved yeah sandbox.
00:06:31.730 --> 00:06:36.865
Can you use it via Helm or is it like automated Ratify?
00:06:36.865 --> 00:06:37.406
You can.
00:06:39.360 --> 00:06:42.345
So that is the only of those three projects that would sit on your cluster.
00:06:42.345 --> 00:06:48.966
The other two, notation and Copasetic, are both CLI tools that have GitHub Actions, so we have to automate it.
00:06:50.321 --> 00:06:57.653
So what are some of the use cases of like impact, use cases that you think both of the tools do have?
00:06:57.653 --> 00:07:01.730
Do you just have to use all of them side by side?
00:07:02.322 --> 00:07:03.185
You can use all of them.
00:07:03.185 --> 00:07:04.021
So here let me.
00:07:04.021 --> 00:07:06.848
Actually let me just show you a repository.
00:07:06.848 --> 00:07:07.771
Would that work?
00:07:07.771 --> 00:07:13.009
Yeah, here let me, let me clean up my very messy desktop.
00:07:13.281 --> 00:07:19.031
Oh yeah, before I mention, I remember the tool that we use A net scaler, no, a new netter.
00:07:19.271 --> 00:07:20.093
Oh, I've not heard of that one.
00:07:20.399 --> 00:07:23.547
It's like a container scaler thing, it's like a container-skeletal thing.
00:07:23.567 --> 00:07:25.009
It's like an images thing.
00:07:25.009 --> 00:07:26.072
Does it have a scanner?
00:07:29.798 --> 00:07:31.088
Yeah, but you have to pay for a license.
00:07:31.088 --> 00:07:32.057
It's not open source, I think.
00:07:32.057 --> 00:07:35.107
Ah, okay, it may have a container-skeletal license.
00:07:35.127 --> 00:07:36.129
Share screen.
00:07:36.129 --> 00:07:36.471
Let's see.
00:07:36.471 --> 00:07:39.826
Oh no, Is it going to make me have to drop?
00:07:39.826 --> 00:07:41.903
I've used this before.
00:07:41.903 --> 00:07:42.903
Let's see here Window.
00:07:43.245 --> 00:07:44.125
I want this one.
00:07:44.125 --> 00:07:46.029
I think it's sent Yep.
00:07:46.430 --> 00:07:47.291
All right, fantastic.
00:07:47.291 --> 00:07:49.115
So this is a little outdated.
00:07:49.115 --> 00:07:51.682
I will get to.
00:07:51.682 --> 00:07:54.685
If my build talk gets submitted, I will definitely take another pass at this, but here this is kind of.
00:07:54.685 --> 00:07:55.964
This will give you the full picture here.
00:07:55.964 --> 00:07:58.846
So if we look at this, this little graph here will give you an idea.
00:07:58.846 --> 00:08:07.415
So there's another tool I haven't actively contributed to, but I use all the time, which is Trivy.
00:08:07.415 --> 00:08:08.723
So Trivy is a vulnerability scanner.
00:08:08.723 --> 00:08:11.788
It's probably you, might it might be using your tool, might be using it.
00:08:12.161 --> 00:08:20.442
I think you can scan like infrastructure tool, like Terraform code as well using Trivy oh yeah, yeah, yeah, misconfiguration and secrets and stuff.
00:08:20.562 --> 00:08:25.319
Sure, absolutely yeah.
00:08:25.319 --> 00:08:27.565
So Tribu is just a vulnerability scanner and misconfiguration scanner, for lack of a better word.
00:08:27.565 --> 00:08:29.672
I'm sure their bio will give a more accurate description.
00:08:29.672 --> 00:08:34.782
But essentially, what that allows you to do is to generate a report of, like what's vulnerable in this image.
00:08:34.782 --> 00:08:44.932
So what you can do is you can hook these all up with GitHub Actions, which I'll show you in a second, and then the next step would be wouldn't it be super cool if we could remediate those vulnerabilities based on the report?
00:08:44.951 --> 00:08:46.254
And that's where Copasetic comes in.
00:08:46.254 --> 00:08:54.197
So Copasetic will take care of OS level vulnerability patching for container images and it can ingest trivia results to do targeted patching.
00:08:54.197 --> 00:08:58.269
It also has the ability to just patch everything to the latest on your system If you want to.
00:08:58.269 --> 00:09:09.850
You're feeling risky and you want to, you want to go that route, you can, but the trivia one will just I found a Cve for libc, I'm going to update libc or whatever that might be, and the report takes care of that, uh, and then?
00:09:09.850 --> 00:09:11.826
So now we have this image that we feel pretty good about.
00:09:11.826 --> 00:09:13.714
How do we signal trust?
00:09:14.054 --> 00:09:18.796
And that's where the notary project or the notation clr comes in, and you can more or less stamp.
00:09:18.796 --> 00:09:30.785
You put a digital signature on that container image, a notation, so another another option would be Cosign from Sigstore, so you could use that as well and that creates a digital signature on the image and that becomes important later on.
00:09:30.785 --> 00:09:32.645
But there's a distinction between.
00:09:32.645 --> 00:09:33.830
Well, I think they can do both.
00:09:33.830 --> 00:09:47.688
If you use the beta features for Cosign, they both can use the referrers API or it'll create, like another artifact, a signature artifact on your registry, and I think I even might have some packages that we could see what that looks like on here.
00:09:47.688 --> 00:09:50.520
We'll see if I do no, but maybe yosh does so.
00:09:50.620 --> 00:09:54.754
Have you used this on production environments for, or is it just like poc?
00:09:55.176 --> 00:10:02.501
well, I know it's being used in production inside microsoft, but the nature of my role is I they don't let me in production very often, right?
00:10:02.501 --> 00:10:09.153
So they actually just, uh, you kick me out of, like production tenants and stuff, and that's just the nature of my role in the advocacy.
00:10:09.153 --> 00:10:12.528
You know I'm not responsible for any internal production systems.
00:10:12.528 --> 00:10:15.408
I don't want the Wasm cloud, it's all it's my cloud.
00:10:15.539 --> 00:10:20.687
Yeah, so to update the image, you can just have different base files for each of the image user.
00:10:27.019 --> 00:10:28.024
And then you can update it right via helm or cli.
00:10:28.024 --> 00:10:29.049
Yeah, uh, so there's different patching strategies.
00:10:29.049 --> 00:10:34.769
If you want to one second and we'll go in a little segue about like why would you even want to patch an image and not just update the base image?
00:10:34.769 --> 00:10:36.621
Because that's a valid, that's a valid question.
00:10:36.621 --> 00:10:39.809
I wanted to show signature real quick.
00:10:39.809 --> 00:10:41.033
Is this the right one?
00:10:41.033 --> 00:10:44.724
Now I want packages and not there, so so this is a signature here.
00:10:44.724 --> 00:10:46.806
This SHA-SIG, I guess.
00:10:46.806 --> 00:10:55.716
Yep, and so that's what Cosign does by default is it'll put the signature on the repository Notation by default, will make it through the refer, so it won't show up here.
00:10:55.716 --> 00:11:04.945
It'll actually be in, like the manifest metadata, if you were to like pull up latest and in here we would have some information to the digest for the signature and it can do that with.
00:11:04.945 --> 00:11:10.610
It works with, I think, most registries, any OCI compliant registry it works with.
00:11:10.789 --> 00:11:13.152
So you have to modify the image.
00:11:13.152 --> 00:11:15.575
If you want any custom image, make it own.
00:11:15.934 --> 00:11:16.635
It'll do it for you.
00:11:16.635 --> 00:11:21.662
Yeah, yep, so the copacetic.
00:11:21.662 --> 00:11:26.822
You could overwrite the tag if you wanted, or you could append, like attach, like a hyphen one to do a counter, but we can go into that in a second.
00:11:26.822 --> 00:11:33.489
So we'll pause right there, though, and just you know, like why would you even consider patching container images when, like, this is the build pipeline?
00:11:33.489 --> 00:11:35.961
I could just bump up, uh, the base image version.
00:11:35.961 --> 00:11:39.572
When you can do that, then you know, I would say patching, don't do patching.
00:11:39.611 --> 00:11:40.600
I think what was it?
00:11:40.600 --> 00:11:49.115
Chain guard just had like something on their x profile of a big billboard that they put up, that they had like patching something, and they crossed it out and they said secure future.
00:11:49.115 --> 00:11:50.582
I don't disagree.
00:11:50.582 --> 00:11:54.721
Like, if you have the ability to use a better base image that doesn't have CVEs, do that.
00:11:54.721 --> 00:12:06.259
The problem comes when those vulnerabilities are upstream and you have a dependency on some other image and their cycle is longer than yours and they can't resolve that cve.
00:12:06.259 --> 00:12:15.772
You're basically locked right, like I can't upgrade a version because it's going to break something or because there just is another latest version yeah, not a lot of the like couple seconds.
00:12:15.831 --> 00:12:23.447
Like the upstream repo probably won't tell you the vulnerability, like if you used to look at a changelog, it probably won't say just show the features.
00:12:23.447 --> 00:12:26.381
So you have to scan it to see if that is very good.
00:12:26.402 --> 00:12:35.673
Within the hash right image itself exactly, exactly, yeah, and so it's most useful when you've got an upstream dependency that you just can't update the base image for.
00:12:35.673 --> 00:12:39.711
How do you get rid of those vulnerabilities, and copasetic is a good way to automate that.
00:12:39.711 --> 00:12:45.072
Technically, it's super cool using build kit to to break apart your container and patch it and create a new page.
00:12:45.072 --> 00:12:45.633
It's pretty cool.
00:12:46.039 --> 00:12:49.965
Have you used this for Azure DevOps before, or is it just CLI?
00:12:50.220 --> 00:12:52.327
Oh yeah, it works in Azure DevOps, I believe.
00:12:52.327 --> 00:12:55.187
What's the GitHub action equivalent for Azure DevOps?
00:12:55.187 --> 00:12:56.291
Is it just a task?
00:12:57.322 --> 00:12:57.985
Yeah, it's a task.
00:12:58.782 --> 00:13:00.748
Yeah, so there's a copacetic task for it.
00:13:01.129 --> 00:13:07.240
Okay, and is it for the nodejs?
00:13:07.240 --> 00:13:07.902
Now, the other one is trivia.
00:13:07.902 --> 00:13:08.703
So I know that is a trivia and then.
00:13:08.703 --> 00:13:10.047
Okay, so a lot of them have to test.
00:13:10.047 --> 00:13:18.672
So is it possible, just like, like, if you create one, one base, one file, to update image instead of updating lots of image for each of the?
00:13:19.052 --> 00:13:20.664
the tool to have like a golden image.
00:13:20.664 --> 00:13:21.850
Is that what you're referring to?
00:13:21.850 --> 00:13:23.196
Yeah, uh, yeah.
00:13:23.196 --> 00:13:23.938
So let's see here.
00:13:23.938 --> 00:13:25.105
Let's go to copasetics documentation, one of like.
00:13:25.105 --> 00:13:25.667
You have a dot net image of.
00:13:25.726 --> 00:13:26.389
Is that what you're referring to?
00:13:26.389 --> 00:13:26.591
Yeah, yeah.
00:13:26.591 --> 00:13:26.952
So let's see here.
00:13:26.952 --> 00:13:39.575
Let's go to Copasetic's documentation, one of like you have a NET image of NET and you can update that, then it will pass the latest without and then you can pass it with whatever image you want, whether it's a patch image or an actual NET, that is image I think what you.
00:13:39.855 --> 00:13:49.150
So you could do this, uh, dynamic tagging where you basically say, like here's, here's the base tag that I'm patching, but then I'm always going to override this version tag.
00:13:49.150 --> 00:13:51.476
So the patching you would want to do per version.
00:13:51.476 --> 00:13:53.110
So there would be another version of this.
00:13:53.110 --> 00:13:57.831
Like there would be another tag for a point 1.25, you would have another one that is always patched.
00:13:57.831 --> 00:14:00.534
That's, that's one tagging strategy that you could you could do.
00:14:00.534 --> 00:14:01.736
That'd be the dynamic one.
00:14:01.736 --> 00:14:04.380
The other one is the, the incremental one.
00:14:04.380 --> 00:14:09.859
So you can just see there in the docs, you know, like the first time it goes through and patches it's going to be hyphen one.
00:14:09.859 --> 00:14:11.230
The second time will be hyphen two.
00:14:11.230 --> 00:14:17.238
The problem with that is you, you have to go up and up, you have to go back and update your manifests and I'll have this ready.
00:14:17.238 --> 00:14:18.299
This would be cool to add in.
00:14:18.299 --> 00:14:33.815
But I'm working on setting up a sample repo that uses Dependabot to be able to come back and say, hey, I detected a new patch version of your image, just like it would any other dependency update right, and do a PR into the repo to say, hey, let's update the image to a new patched tag.
00:14:34.134 --> 00:14:41.315
The issue with the other supply chain tools down the line is if you're not using the tag and you're using the digest.
00:14:41.315 --> 00:14:44.822
So, for context and people that don't know, each tag has its own digest.
00:14:44.822 --> 00:14:52.730
If you look back at this, the SHA right here is the digest, so you can actually reference container images in your manifest by digest.
00:14:52.730 --> 00:15:03.144
And that's the reason that's good is because it's not mutable, meaning if the image has changed in any way, it gets a new digest and it won't pick it up, which is inconvenient if you want that.
00:15:03.144 --> 00:15:10.399
But it also improves the security, because now you know for certain which exact tag and version of the image you are going to have deployed.
00:15:10.399 --> 00:15:16.340
But if copacetic patches changes the tag and then that digest is actually used by the signing tools.
00:15:16.340 --> 00:15:19.211
So the other tool that I didn't get Okay.
00:15:19.432 --> 00:15:27.777
so whenever you use the signing tool, do you just get a new digest you do yeah, okay, yeah, I'm curious to know.
00:15:27.777 --> 00:15:39.131
Sure, is there any like open source tool that's equivalent to this, that's like a web competitor, or is this, is the one that's well known in microsoft or in the cnc?
00:15:39.131 --> 00:15:39.533
Uh?
00:15:39.732 --> 00:15:40.193
which tool.
00:15:40.575 --> 00:15:41.255
Which tool?
00:15:41.255 --> 00:15:46.004
Whether it's going to be copacetic or notary.
00:15:46.403 --> 00:15:46.745
Notary.
00:15:46.745 --> 00:15:48.056
So the notation CLI.
00:15:48.056 --> 00:15:50.739
Its counterpart would be cosine.
00:15:50.739 --> 00:15:52.514
So the cosine CLI from Stixor.
00:15:52.514 --> 00:15:53.756
They just, they, just.
00:15:53.756 --> 00:15:55.562
They just create the signatures in a different way.
00:15:55.562 --> 00:15:56.172
The biggest.
00:15:56.352 --> 00:16:03.498
Here's what I would say If you have a mature paykey, public key infrastructure inside your company like you're a large enterprise you have your own keys.
00:16:03.498 --> 00:16:04.941
Notation is what you want to use.
00:16:04.941 --> 00:16:14.370
But if you don't and you just want to have a signature, cosign actually has something called keyless signing where it will basically get rid of the private key so it can't be used again, but then the public keys are made available.
00:16:14.370 --> 00:16:15.912
I believe it was a cold record.
00:16:15.912 --> 00:16:21.201
So that way you can verify the signature but you don't have to maintain your own PKI infrastructure.
00:16:21.201 --> 00:16:24.225
So that would be the big delineation, at least in my mind.
00:16:24.225 --> 00:16:30.615
If you've got a mature PKI system and you need to use your own certificates, that's where notation shines.
00:16:30.615 --> 00:16:45.645
And then cosign is on the other side, where maybe you're doing more local development or you just don't have the infrastructure, don't care to have the infrastructure and you want signing, and I'm sure they have offerings to kind of offload the PKI infrastructure and they have, I'm sure they have offerings to kind of offload the PayCat infrastructure.
00:16:45.686 --> 00:17:04.733
I'm just not super familiar with their product suite, so I can see this improving by having a little dashboard, seeing any outdated images, like when people like internal what developer or DevOps can monitor, and then we can just like get, we can just update the image accordingly using automation, whether it's GitHub or Azure DevOps Yep, there's something that can be built in like monitoring on it if we just update the image accordingly using automation.
00:17:04.733 --> 00:17:06.895
Whether it's GitHub or Azure DevOps Yep, there's something that can be built in like monitoring on it.
00:17:06.915 --> 00:17:08.237
Kubescape might be doing something like that.
00:17:08.237 --> 00:17:10.779
This is a tool that popped up on my radar recently.
00:17:10.779 --> 00:17:12.741
I haven't dove into it yet, but it does use.
00:17:12.741 --> 00:17:14.282
It integrates with Copasetic.
00:17:14.683 --> 00:17:14.903
Okay.
00:17:14.903 --> 00:17:22.383
So this might actually be kind of what you're envisioning you probably can interface into like a GUI, right?
00:17:22.383 --> 00:17:23.150
Yeah, when people like a front-end GUI.
00:17:23.150 --> 00:17:24.509
Okay that's quite nice.
00:17:24.509 --> 00:17:32.999
It's easier for like developers or anyone to have login and concede vulnerability and patch it themselves by running the automation tool.
00:17:33.369 --> 00:17:34.115
Yep, that would be.
00:17:34.115 --> 00:17:40.519
It's hard for me sometimes to put on my S3 platform engineering hat and want to build those types of things, but those are lots of time.
00:17:40.519 --> 00:17:43.698
This is more putting together open source tools, something.
00:17:43.698 --> 00:17:53.425
Yeah, a product could definitely sit on top and use all of these things together and make it a lot more seamless, and I think that's honestly what needs to happen to make it more palatable for companies and organizations.
00:17:53.425 --> 00:17:56.795
This is a lot of additional work, but this is what it would be doing under the hood.
00:17:56.795 --> 00:17:58.880
There are a number of features inside Azure.
00:17:58.880 --> 00:18:09.076
One would be the continuous patching for ACR, so it uses ACR tasks to do the copacetic stuff for you and then it takes it out of your workflow and makes it more of a platform checkbox type task.
00:18:09.549 --> 00:18:16.252
Yeah, because normally we're trying to make everything self-service so people can just run a pipeline and then it will update the image.
00:18:16.252 --> 00:18:18.855
So we're just putting the YAML and submit to PR.
00:18:19.550 --> 00:18:29.674
So whether it's looking at the dashboard to see vulnerability or not, yeah, this would all take care of it inside of a pipeline, but this would require all the dev teams to kind of adopt a similar workflow or a templated workflow.
00:18:29.674 --> 00:18:32.375
Right, yeah, so the other end of it.
00:18:32.395 --> 00:18:32.837
Oh, go ahead.
00:18:32.837 --> 00:18:34.949
What's the result you get at the end?
00:18:34.949 --> 00:18:47.108
So do you get, like a result of something that's outdated, images with SAP vulnerability and then someone can actually check whether it's an actual vulnerability or not?
00:18:47.430 --> 00:18:49.898
I don't know of anything that there's.
00:18:49.898 --> 00:18:57.938
I think there's some Azure features that would be able to do continuous scanning on your registry and then present you with diagnostic information there.
00:18:57.938 --> 00:18:59.174
I haven't looked into it too much.
00:18:59.174 --> 00:19:01.590
That's what I'd imagine it would be going Right now.
00:19:01.590 --> 00:19:03.736
It's point and click in the workflow.
00:19:03.736 --> 00:19:05.840
Trivia runs inside that report.
00:19:05.840 --> 00:19:07.031
You would see your vulnerability list.
00:19:07.031 --> 00:19:08.576
It's not displayed on any kind of dashboard.
00:19:08.576 --> 00:19:09.380
It's displayed here.
00:19:09.380 --> 00:19:18.076
There are flags with trivia that you can use to stop your workflow and fail it if a certain number of CVEs exist, even after you patch or during the scan.
00:19:18.217 --> 00:19:33.548
But yeah, there's no dashboard, at least in my current map markup there we go and you say you can, this can be integrated with github dependent bot or something that does the pr scanning for vulnerabilities and stuff, so you don't really have to scan it yourself.
00:19:33.548 --> 00:19:34.330
You can get pr.
00:19:34.330 --> 00:19:36.618
That's been raised and just so.
00:19:36.659 --> 00:19:47.881
It's just reviewing really yeah, that's the kind of the end to end for if you're just wanting to focus on GitHub workflows, and Dependabot obviously works for Azure DevOps too.
00:19:48.089 --> 00:19:51.661
Because I think I use something similar called Renovate.
00:19:51.661 --> 00:19:58.423
It does image for versions, but I think it probably can integrate with this.
00:19:58.423 --> 00:20:07.192
It's similar to Dependabot in a way Okay cool, great with this.
00:20:07.192 --> 00:20:08.678
It's similar to the pentabot in a way okay, cool.
00:20:08.678 --> 00:20:09.500
So have you got any?
00:20:09.500 --> 00:20:11.988
Do you know any examples where this is used in companies, or it's just microsoft at the moment?
00:20:11.988 --> 00:20:12.529
You know where?
00:20:12.971 --> 00:20:16.364
alibaba has been pretty good, uh, with adoption on ratify.
00:20:16.364 --> 00:20:17.268
They've got some solutions.
00:20:17.268 --> 00:20:19.212
So, yeah, let me go to the deploy phase.
00:20:19.212 --> 00:20:24.364
Um, a little bit and just about that, and then we can hang on to more customer adoption.
00:20:24.364 --> 00:20:31.269
So, if we look at the bottom part, so it assumed like we've got a signed image on our registry and now we want to deploy to our cluster.
00:20:31.269 --> 00:20:38.317
How do we put some teeth to all this work that we did during the build step right To produce this artifact that we trust?
00:20:38.549 --> 00:20:50.045
So ratifies and gatekeeper combined, so you deploy them together with a helm chart, and what that allows you to do is do admission control for digital signatures or other verification policies, so you can define some kind of rego policy.
00:20:50.045 --> 00:20:56.801
And in this example it's looking for does a digital signature exist for this public certificate, right?
00:20:56.801 --> 00:21:03.943
And so if I can't validate the signature with my public key, that signature either doesn't exist or is invalid, and I'm not going to let the pod run on this cluster.
00:21:03.943 --> 00:21:08.992
And so there's a lot of good use cases there, trying to think what are the other customers?
00:21:08.992 --> 00:21:13.940
By name or if I should say names, but yeah, there are definitely customers using it.
00:21:13.940 --> 00:21:20.555
A lot of these are actually being baked into features right Like features inside ACR features inside AKS or whatever it might be.
00:21:20.936 --> 00:21:30.140
For example, like with Ratify and Azure, you can define an Azure policy that converts to a Ratify configuration, so that way you don't have to apply another kubectl thing.
00:21:30.140 --> 00:21:36.958
It can just be any AKS cluster that gets deployed to this resource group has to have signed images, and then that translates down to Ratify.
00:21:37.390 --> 00:21:51.258
Yeah, I guess the good thing about it is that it's a security aspect as well, because it constantly gets updated as well with the images, and it's being maintained by the CNCF, so that's a good side about it.
00:21:52.070 --> 00:21:53.036
Yeah, all those projects are.
00:21:53.036 --> 00:21:57.381
I don't know about Trivi, but Copasetic, notation and Ratifier are all CNCF projects.
00:21:57.480 --> 00:21:58.103
Okay, brilliant.
00:21:58.103 --> 00:22:03.942
Do you have a picture of what it will look like as a result when you run it?
00:22:04.349 --> 00:22:07.934
Yeah, I think I have some successful runs, or maybe I don't.
00:22:07.934 --> 00:22:08.983
I deleted them, I don't know.
00:22:08.983 --> 00:22:09.710
We'll find out.
00:22:09.789 --> 00:22:12.239
No worries, so it'll just give you a trivia.
00:22:12.239 --> 00:22:22.659
I know trivia just gives you SVE, whether it's infrastructure code or any code in general of the list, but for the rest it just gives you the hash.
00:22:22.659 --> 00:22:26.119
So it just trivies that it's bring up right Most of the results.
00:22:26.390 --> 00:22:28.296
Well, here let's just look at the workflow, we'll see.
00:22:28.296 --> 00:22:29.881
So this is a little update.
00:22:29.881 --> 00:22:46.555
I'll show you a more recent one for people that are more on the platform engineering side, because a lot of this is being done on the developer side, which might be hard, and maybe you just care about patching the images in your registry and I'll talk about a continuous patching workflow here in a second for at least GHCR.
00:22:46.555 --> 00:22:56.433
So yeah, if we just let me bump this up, the general workflow is we're going to log into Azure, log into ACR, we're going to build our app, right, and then we scan it.
00:22:56.433 --> 00:23:01.830
So we scan with the trivia action that's going to output this patchjson and then we're going to patch it.
00:23:01.830 --> 00:23:03.310
So this can be replaced.
00:23:03.310 --> 00:23:05.171
There's a GitHub action for this.
00:23:05.171 --> 00:23:07.009
Now this can be replaced.
00:23:07.009 --> 00:23:08.348
I'll show you that in the next one.
00:23:08.549 --> 00:23:09.473
We're going to patch the image.
00:23:09.473 --> 00:23:13.557
So I'm just outputting what the digest is going to be and then push the patch image.
00:23:13.557 --> 00:23:17.320
So I'm going to push the patched image after Copasetic patches it.
00:23:17.320 --> 00:23:18.461
So Copasetic is going to patch it.
00:23:18.461 --> 00:23:30.296
It's going to create a new image locally with the taking prefix like hyphen one or whatever taking strategy you're going to use, and then I'm capturing that digest at the most optimal time, which is right after it's built.
00:23:30.296 --> 00:23:31.057
I'm going to capture it.
00:23:31.057 --> 00:23:41.134
So there's no chance that I'm I'm grabbing the wrong digest and someone put another layer on this image with something malicious in there and I didn't know.
00:23:41.064 --> 00:23:42.556
So I'm going to grab that digest right there and that's where notation comes in.
00:23:42.556 --> 00:23:50.977
Notation has a, an action as well, so I'm just setting it up and then I'm signing it and then that puts a signature on the refer API that can be validated with ratified later.
00:23:50.977 --> 00:23:53.917
And then so if you looked at, I think I have terraform code in here.
00:23:53.917 --> 00:24:00.300
This sets up the aks cluster and then I'm using it in the demo I actually set up ratify.
00:24:00.300 --> 00:24:06.027
So it's not, it's not part of my infrastructure as code, because it's part of the demo script, but all this you can walk through as a demo.
00:24:06.027 --> 00:24:12.489
This demo sh goes through every single step in the workshop, so that's from like a pipeline perspective.
00:24:12.489 --> 00:24:17.398
But let's say that you wanted to like what would it look like to continuously patch a registry?
00:24:17.398 --> 00:24:22.078
And so I started this repo and I walked through this in a blog post series for people that are interested.
00:24:22.279 --> 00:24:23.262
Okay, it looks like.
00:24:23.262 --> 00:24:23.744
Oh, go ahead.
00:24:23.744 --> 00:24:34.192
This would go through like ACR, like for your, and it's patched all the images to see if they're using the correct images or not, if they have any vulnerability.
00:24:34.625 --> 00:24:36.596
Yeah, to see if they're using the correct image or not, if they have any vulnerability, yeah.
00:24:36.596 --> 00:24:37.440
So there's a continuous patching.
00:24:37.440 --> 00:24:42.276
Private preview for the AZ-CLI that basically does just this.
00:24:42.276 --> 00:24:44.973
I just did it for GHCR and I wrote a little.
00:24:44.973 --> 00:24:47.710
It gave me an excuse to write some code which I can show you in a second.
00:24:48.105 --> 00:24:52.953
Does this look at images that's, for example, older than three years or three months, or something?
00:24:53.144 --> 00:24:56.335
It'll look at everything on your GHCR repo, everything.
00:24:56.335 --> 00:24:57.829
So essentially what it'll do.
00:24:57.829 --> 00:25:05.694
If we look at this little flow chart, it sets up, it gets your images from GHCR, it outputs a matrix and then it loops through the matrix and patches.
00:25:05.694 --> 00:25:07.432
So let's just look at the workflow real quick.
00:25:07.432 --> 00:25:09.570
So here's the setup right here.
00:25:09.570 --> 00:25:11.016
Make it a little bigger.
00:25:14.924 --> 00:25:15.385
I wrote a little go CLI.
00:25:15.385 --> 00:25:17.090
I turned it into a CLI instead of just a go file, which we can see in another branch.
00:25:17.090 --> 00:25:19.507
There's actually quite a bit of logic in there to detect the tags.
00:25:19.507 --> 00:25:25.227
So the advanced logic that I put in there would basically calculating the tagging strategy.
00:25:25.227 --> 00:25:27.090
So I'm only going to grab the latest.
00:25:27.090 --> 00:25:39.930
If, like, net monitor has a hyphen one, I'm only going to patch hyphen one and not zero, because zero came before hyphen one, and then if I patch both, they'd be at equal patch level.
00:25:39.930 --> 00:25:40.332
You should, you know.
00:25:40.332 --> 00:25:41.356
Then why not just flatten it down to one tag?
00:25:41.356 --> 00:25:42.281
So there's a little bit of strategy there.
00:25:42.281 --> 00:25:46.753
And then I'm calculating the incremental on that and that looks like let me open a new tab.
00:25:47.134 --> 00:25:50.007
The output for that CLI improved looks like this.
00:25:50.007 --> 00:25:51.108
I called it contiguous.
00:25:51.108 --> 00:25:52.089
I thought it was clever.
00:25:52.089 --> 00:25:52.810
Maybe it does.
00:25:52.810 --> 00:26:00.989
I don't have a screenshot maybe, but essentially you would do this list and then it would output a JSON of here's your current tag and here's your next tag.
00:26:00.989 --> 00:26:09.300
And then GitHub Actions is pretty cool because you can just use from JSON to generate your matrix, yeah, and then basically go through.
00:26:09.642 --> 00:26:15.017
So for each image in the matrix that I identify that needs a patch, I'm going to go through and generate a trivia report.
00:26:15.017 --> 00:26:16.631
I'm going to check the vulnerability count.
00:26:16.631 --> 00:26:22.637
So if the vulnerability counts zero, I'm not going to bother patching, because Copasetic is just going to fail and say I couldn't patch anything.
00:26:22.637 --> 00:26:25.532
This patch tag is taken care of and then the new version is CLI.
00:26:25.532 --> 00:26:27.165
But this is just some bash Again.
00:26:27.165 --> 00:26:28.949
This is why I've had to learn more.
00:26:28.949 --> 00:26:31.092
Bash is because of GitHub Actions, to be honest.
00:26:31.092 --> 00:26:38.628
This calculates the next patch tag based on a tag, then it goes through and runs Copasetic logs into GHCR and then pushes the patch tag.
00:26:38.628 --> 00:26:43.111
So the next part of this workflow would be like okay, I've got all these new patched images on my workflow.
00:26:43.111 --> 00:26:49.152
How do I automate that final step, which would be the PRs, into the code repos that are using those images?
00:26:54.865 --> 00:26:55.671
And that's where Dependable will come in.
00:26:55.671 --> 00:26:56.395
So that's next on my list to tackle.
00:26:56.395 --> 00:27:03.156
Okay, yeah, it's quite good that you can automate this and probably can make it a self-service for anyone or developers to run.
00:27:03.904 --> 00:27:18.527
Just add the images, yep, and so here's an example, here's the setup, and I only had one image, but two tags that needed to be patched, right, so let me shrink it a little bit so you can see, but then, yeah, it just goes through.
00:27:18.527 --> 00:27:20.794
This one didn't even need to be patched, so it skipped it.
00:27:23.397 --> 00:27:24.486
So this is probably good.
00:27:24.486 --> 00:27:27.009
You can probably build in this with some alerts as well.
00:27:27.009 --> 00:27:32.465
When someone's image has been outdated for like one month, you can alert you so someone can patch it.
00:27:32.645 --> 00:27:35.151
Yeah, you could this continuous patching thing.
00:27:35.151 --> 00:27:41.498
I have it on an on-demand trigger, but you could a cron job, like whatever you wanted to do, just kind of show you like what's possible with it.
00:27:41.498 --> 00:27:46.520
Like I said, cubescapes doing something with it, integrating with copacetic to make this more full-fledged.
00:27:46.520 --> 00:27:50.931
This was just to give an idea of, like what you could build with it and a lot of people are doing with that.
00:27:50.931 --> 00:27:57.008
I I think, yeah, in ACR they're working on the continuous patching thing, solving a lot of these problems as you dig into it.
00:27:57.028 --> 00:28:01.211
There's issues, like I mentioned, with the signing, like how do we make sure that the signatures are still valid?
00:28:01.211 --> 00:28:02.215
How do we update?
00:28:02.215 --> 00:28:08.751
It's funny just last year I contributed the logic for Ratify to support inversions of keys.
00:28:08.751 --> 00:28:20.467
So basically it'll look at your, your keyball instance and if you have more versions that you've created for a key, it'll update, auto update the cache, which solves the next problem that the continuous patching does like.
00:28:20.467 --> 00:28:22.291
Okay, what if I signed it with a new key?
00:28:22.291 --> 00:28:24.926
Do I have to go then update my verification engine?
00:28:24.926 --> 00:28:26.290
The answer is no, it'll.
00:28:26.290 --> 00:28:27.994
It'll dynamically pull um.
00:28:27.994 --> 00:28:35.958
So that was kind of cool to see that that work get used, okay so, yeah, it looks like it's quite a good tool to quite batches.
00:28:36.159 --> 00:28:36.865
Is there any?
00:28:36.865 --> 00:28:41.056
So, to get this set up and running, is there any challenges that you have to do?
00:28:41.056 --> 00:28:47.076
I take it one of the challenges you have to know more batches right or just use your task well.
00:28:47.478 --> 00:28:52.517
So the way that I've set it up, if you I'm tomorrow's on my to-do list, tomorrow is to take the.
00:28:52.517 --> 00:28:53.711
So if you want to use ghcr, that's what I scope.
00:28:53.711 --> 00:28:54.435
Mine to you tomorrow is to take the.
00:28:54.435 --> 00:28:59.055
So if you want to use GHCR that's what I scoped mine to you could actually just take my actions and plug them in and it would work.
00:28:59.055 --> 00:29:07.151
The only thing that you'd have to figure out if you're not using GHCR is figuring out what tags you want in your matrix, and you could manually define that.
00:29:07.151 --> 00:29:09.426
So if we go back and look at this one, I had an example.
00:29:10.107 --> 00:29:12.829
I had an example of using just a static matrix.
00:29:12.829 --> 00:29:18.255
So if you knew the tag names that you wanted, you could just define your matrix like this, right?
00:29:18.255 --> 00:29:25.622
But I wanted to be a little bit more pragmatic, and so then that required me to go in write my own logic against GHCR.
00:29:25.622 --> 00:29:28.130
So that would be.
00:29:28.130 --> 00:29:32.334
The biggest gap right now is you need some mechanism to feed into the copacetic workflow what images you care about.
00:29:32.334 --> 00:29:34.290
So if you want, you can use my Go code.
00:29:34.290 --> 00:29:34.752
That's fine.
00:29:35.484 --> 00:29:37.393
Yeah, it's open source anyway, so it's good.
00:29:37.393 --> 00:29:44.496
So do you reckon that the tool that you show us today is the future of container security, like containerized?
00:29:44.664 --> 00:29:46.653
I would imagine it's kind of the foundation.
00:29:46.653 --> 00:29:48.673
So I think they're like what you were talking about earlier.
00:29:48.673 --> 00:29:52.307
These will be wrapped into other tools that probably cost a little bit of money.
00:29:52.307 --> 00:29:54.073
That just make it easier, right.
00:29:54.073 --> 00:29:55.576
That's kind of the trade-off with open source.
00:29:55.576 --> 00:30:07.494
But if you want to build your own type of solution, all the tools are there to improve the supply chain and again in the CNCF, so like they're stable and you know that they've got good investment in them, they're being used internally for Microsoft.
00:30:07.494 --> 00:30:09.613
They're being used internally at other big competitors too.
00:30:09.613 --> 00:30:15.478
One interesting thing AWS and Microsoft are both working on Notation, so they're collaborating.
00:30:15.478 --> 00:30:17.070
We're collaborating on that one.
00:30:17.070 --> 00:30:22.153
But yeah, so the tools are there to put it in there if you want to go the open source route.
00:30:22.153 --> 00:30:28.617
But I'd imagine there's going to be some cool tools that kind of wrap all this and make cool button Right.
00:30:29.199 --> 00:30:29.942
Make everything a button.
00:30:29.942 --> 00:30:32.648
The only good side about it is that this is free.
00:30:32.648 --> 00:30:33.490
It's open source.
00:30:33.490 --> 00:30:39.218
Instead of other tools that you have to pay it, you know to use this stuff enterprise tool.
00:30:39.218 --> 00:30:49.685
That's more of a better suited.
00:30:49.685 --> 00:30:50.627
So that's about okay.
00:30:50.627 --> 00:30:52.934
So, as this episode is coming to end, we always like to get to know our guests.
00:30:52.934 --> 00:30:53.896
What do you normally do in?
00:30:53.896 --> 00:31:02.155
Your spare time, josh, like your job as an advocate on your job and then learning about security in containers.
00:31:02.395 --> 00:31:03.999
Yeah, well, the last few years.
00:31:03.999 --> 00:31:07.393
So I just my wife and I welcomed our third child a year ago.
00:31:07.393 --> 00:31:08.935
He just turned one last month.
00:31:08.935 --> 00:31:15.798
So the kids keep me pretty busy, so my spare time isn't as fruitful as it used to be, but I like to read, so I do a fair amount of reading.
00:31:15.798 --> 00:31:28.340
Lately it's been I've actually been revisiting some old books PowerShell books, no, actually more like productivity-type books Deep Work by Cal Newport, I think it was released in 2016, 2018.
00:31:30.085 --> 00:31:30.726
And then UltraLearn.
00:31:30.726 --> 00:31:31.929
I just got it right here on my desk I can show you.
00:31:31.929 --> 00:31:33.152
Okay, like you read 50 a book, but not kindle.
00:31:33.412 --> 00:31:38.590
I have a kindle um, I use the kindle to evaluate whether a book's useful or not.
00:31:38.590 --> 00:31:42.628
So like I'll download the sample, I'll burn through the sample on my kindle, uh.
00:31:42.628 --> 00:31:48.249
But I do prefer a physical book, um, even technical books, which are more annoying to do because you have to.
00:31:48.249 --> 00:31:51.715
But anyway, yeah, I like the physical book.
00:31:51.715 --> 00:31:54.249
So I I don't know, I'm, I'm all kind of over the place.
00:31:54.249 --> 00:31:57.992
I read a little bit of Rust, worked with a colleague on some Rust recently.
00:31:59.826 --> 00:32:01.512
That's not the language right Rust.
00:32:01.605 --> 00:32:03.351
No, yeah, the language Rust, yeah.
00:32:03.952 --> 00:32:06.511
Oh, okay, yeah, it's not very popular.
00:32:06.511 --> 00:32:12.628
So I don't think if you use Copilot to tell me about Rust, I don't think Copilot would know what's Rust it but Rust.
00:32:12.628 --> 00:32:15.054
I don't think a lot of people would know what's Rust.
00:32:15.054 --> 00:32:16.076
It's never helpful.
00:32:16.076 --> 00:32:17.638
Yeah, yeah, Rust is quite hard.
00:32:17.638 --> 00:32:18.299
It's like Ruby.
00:32:18.299 --> 00:32:19.883
It's quite an outdated language as well.
00:32:20.565 --> 00:32:21.550
So it's probably more people know Ruby than Rust.
00:32:21.550 --> 00:32:28.696
To be honest, I think I would have had an easier time picking up Rust, moving from PowerShell as my primary language than Go was.
00:32:28.696 --> 00:32:34.873
Just because of all, it is still static, but it has a lot of niceties to language Once you understand.
00:32:34.873 --> 00:32:35.555
They're very cryptic.
00:32:35.555 --> 00:32:41.732
At first they just look like hieroglyphics, but they're actually very useful shorthand macros that Go doesn't have.
00:32:41.732 --> 00:32:46.972
That makes it made the transition a little hard for me, but I do really enjoy Go still.
00:32:46.972 --> 00:32:49.586
It's made me learn things at a much deeper level, that's for sure.
00:32:49.747 --> 00:32:57.952
So I've remembered speaking to you before this goes live and you said you're going to the bill in May.
00:32:58.153 --> 00:32:58.712
Potentially.
00:32:58.712 --> 00:33:02.055
I have a CFP submitted, so I'll have word back.
00:33:02.055 --> 00:33:05.718
I don't know, maybe in the next month or so, so I might be there.
00:33:06.057 --> 00:33:08.779
Yeah, so it's going to be the same session as this talk.
00:33:08.960 --> 00:33:10.361
Yeah, yeah, it's this one right here, yep.
00:33:10.682 --> 00:33:19.074
Okay, nice, so yeah, so if anyone wants to interesting to meet josh, he might be in bill so you can catch him there.
00:33:19.074 --> 00:33:20.798
Absolutely okay, uh, thanks for joining josh.
00:33:20.798 --> 00:33:22.404
So yeah, as normal.
00:33:22.404 --> 00:33:27.714
Stay tuned and see which is going to be on youtube and spotify.
00:33:27.795 --> 00:33:29.759
Thank you, bye thanks for having me.