Managed Hosting and Ecommerce Security: Nexcess Digital Cloud

WEBVTT
00:00:05.040 --> 00:00:07.759
Hello everyone and welcome to the SourceForge podcast.
00:00:07.839 --> 00:00:09.439
I'm your host, Bo Hamilton.
00:00:09.599 --> 00:00:16.559
Now, earlier this month, a critical zero-day vulnerability known as Style Smuggler emerged in Magento and Adobe Commerce.
00:00:16.960 --> 00:00:25.679
The flaw could allow an unauthenticated attacker to remotely execute code on an affected store, and Adobe has confirmed that it was actually exploited in the wild.
00:00:26.160 --> 00:00:34.240
Now, before an official patch was even available, hosting providers had to make that real-time decision or decisions about how to protect merchants.
00:00:34.399 --> 00:00:42.399
And Nectus was among one of them by responding with a number of different solutions, but they responded by identifying suspicious activity.
00:00:42.640 --> 00:00:51.119
They introduced various protections, evaluated customer environments, and worked through remediation across different versions and configurations to help tackle the problem.
00:00:51.600 --> 00:01:00.880
But today in this episode, we're going behind the scenes of that response to better understand what managed hosting actually looks like when a serious security incident unfolds.
00:01:01.039 --> 00:01:06.000
So joining me today to unpack this is Miguel Balparta, senior product manager at Nexus.
00:01:06.239 --> 00:01:11.439
Miguel has a super long history with both Nexus and the Magento ecosystem.
00:01:11.680 --> 00:01:13.519
He was part of the original Nexus team.
00:01:13.680 --> 00:01:21.920
He contributed to its cloud infrastructure, represented the company at Magento events around the world, and actually returned recently to Nexus.
00:01:22.159 --> 00:01:27.519
He also spent several years contributing to the broader Magento community, including serving as a Magento community maintainer.
00:01:27.599 --> 00:01:29.439
So, Miguel, thank you for joining us today.
00:01:29.519 --> 00:01:30.560
I'm really excited to have you here.
00:01:30.879 --> 00:01:32.079
I'm really happy to be here today.
00:01:32.159 --> 00:01:39.840
And yeah, just like you said, I am one of the original Nexus guys, and I think I worked at this company probably seven or ten years ago.
00:01:40.000 --> 00:01:42.239
Back then the history was quite different.
00:01:42.319 --> 00:01:50.159
We were serving mostly, I'm gonna say small to medium stores, which were the ones who were using Magento back in the day.
00:01:50.400 --> 00:01:53.040
Today the idea is completely different.
00:01:53.280 --> 00:02:00.799
Magento went, you know, to the upper segments, and right now we are hosting probably larger stores than we used to do.
00:02:01.120 --> 00:02:04.319
But again, same companies, same problems.
00:02:04.480 --> 00:02:06.959
We still have a ton of incidents like this.
00:02:07.200 --> 00:02:14.080
And to be quite honest, I still enjoy trying to, you know, try to remediate this kind of stuff, try to understand what's going on.
00:02:14.240 --> 00:02:21.520
Um, for this particular incident, one of the one of the main things we saw was a particular uptick in GraphQL requests.
00:02:21.759 --> 00:02:25.599
They were trying to forge some of these requests to try to look legitimate.
00:02:25.840 --> 00:02:32.960
But with our malware rules and with our web application firewalls, we were able to quickly identify these requests were malicious.
00:02:33.199 --> 00:02:45.120
And they were trying to inject the web shell into a directory that was readable by everybody, which wasn't not particularly bad or not particularly uncommon, you know, mostly because this is how this application works.
00:02:45.280 --> 00:02:48.080
This is how Adobe Commerce slash Magento2 works.
00:02:48.319 --> 00:02:53.199
You can sometimes upload different, I don't know, files to web server.
00:02:53.280 --> 00:02:56.960
And when you try to execute them, some of the times they fail, some of the times they work.
00:02:57.439 --> 00:03:08.240
But most of the stuff we see around the style smuggler was to try to exploit the GraphQL endpoint and then try to host a file and then try to execute a web shell.
00:03:08.479 --> 00:03:10.479
So it was a pretty common attack.
00:03:10.800 --> 00:03:13.919
We still see a ton of these trying to get in.
00:03:14.080 --> 00:03:28.159
But one of the first things we were able to do was to block one of these endpoints to try to disallow these attackers to be able to inject this code or to run this code that was being able to write to the file system.
00:03:28.319 --> 00:03:33.039
And that was probably the first action we took against these particular attacks.
00:03:33.280 --> 00:03:43.360
One really interesting comment about these attacks is that they are becoming more and more common, mostly because the attackers are using AI to try to, you know, break into these systems.
00:03:43.520 --> 00:03:48.400
And it's not only one attack, but it's a chain of different types of attacks chained together.
00:03:48.560 --> 00:03:53.520
So first you get to write to the file system and then you get to exfiltrate some of the credit card data.
00:03:53.680 --> 00:03:56.960
So it's not like one-step attack where you are one and done.
00:03:57.120 --> 00:04:06.159
It's more like a chain of systematic injections and then a web channel or a schemer or something you want to inject into the checkout and try to scheme these cards, you know?
00:04:06.319 --> 00:04:14.639
Because at the end of the day, what these attackers are trying, it's mostly around getting credit card info or getting session data, but that's not that common.
00:04:14.719 --> 00:04:19.759
So it's mostly credit card attacks, track carding attacks, schemers, that kind of stuff.
00:04:20.240 --> 00:04:20.560
Okay.
00:04:20.720 --> 00:04:28.000
Yeah, I imagine AI has completely uh changed the game here with uh a lot of these incidents you're you're responding to and a lot of the exploits.
00:04:28.160 --> 00:04:35.279
Um before we unpack the incident that happened here, um, I I do want to ask you a little bit more about kind of your your background.
00:04:35.360 --> 00:04:43.120
And like you mentioned you you originally worked with Nexus and you went on to some other roles and then you came back to the company.
00:04:43.360 --> 00:04:52.560
I'm curious, like you mentioned you you were uh working with sort of smaller storefronts, and now you're working on a lot of like kind of the bigger, uh larger storefronts.
00:04:52.800 --> 00:04:58.800
Um what uh you know, what can you tell us a little bit about like uh what brought you back?
00:04:58.959 --> 00:05:05.839
Was there something about Nectus or where Magetto and e-commerce are today that sort of made you feel like now is the right time to come back?
00:05:06.319 --> 00:05:10.959
Well, I think being in the hosting space, hosting is not the right word we use.
00:05:11.120 --> 00:05:12.079
Now it's cloud.
00:05:12.240 --> 00:05:15.759
When I started 10 years ago, we used to say, yeah, we do hosting.
00:05:16.000 --> 00:05:20.639
Right now, nobody really understands what hosting is, and we all say, yeah, I do clouds.
00:05:20.800 --> 00:05:24.319
But then to be quite honest, I like the physical parts of the internet.
00:05:24.560 --> 00:05:27.199
I'm a computer nerd, I like hardware.
00:05:27.439 --> 00:05:40.160
And by coming back, I was able to fulfill one of my, I don't know, earliest wishes, which was to go back to the cloud world, to be closer to the actual, you know, underlying infrastructure everybody else is using.
00:05:40.319 --> 00:05:46.959
And yeah, just like you said, I worked here probably 10 years ago, and then I went back to work on different agencies.
00:05:47.120 --> 00:05:48.399
I did some time at FedEx.
00:05:48.560 --> 00:05:52.959
Now I'm doing a product for then I did a product for PayPal, sorry.
00:05:53.120 --> 00:06:06.399
And then I wanted to go back, you know, mostly because even when I like the development, the software development lifecycle, I kind of enjoy being closer to the hardware or like to the physical bits of the internet, and that's what brought me back.
00:06:06.560 --> 00:06:19.759
Now there are a ton of my friends who still work in here, and I really want to go back to work with them because we still talk, we still have conversations around like computers, hosting, AI, physical, like local inference, that kind of stuff.
00:06:19.839 --> 00:06:27.920
And I was like, I really want to go back to the hardware world, I want to go back to the bare metal servers because that's what we used to do.
00:06:28.160 --> 00:06:38.240
Nexus was an old school hosting company based on Detroit, and we used to do physical host, like physical hardware only, as in bare metal servers.
00:06:38.399 --> 00:06:48.720
Here we after, we did some virtualization after, but what caught my attention was that they had their own data center in the middle of nowhere in Southfield, Detroit.
00:06:48.879 --> 00:06:50.959
And I was like, this is absolutely incredible.
00:06:51.120 --> 00:06:56.720
I get to see the cables, I get to see the power sources, I get to see the fiber into the building.
00:06:56.879 --> 00:07:00.079
So it was really interesting for me to see as a kid.
00:07:00.319 --> 00:07:05.360
And now that I'm kind of grown up, I'm like, this is a really good space to be in.
00:07:05.519 --> 00:07:09.839
I really want to go back into the hardware world, and then my friends' world are working here again.
00:07:10.000 --> 00:07:12.160
So it was a no-brainer for me.
00:07:12.480 --> 00:07:13.040
I love that.
00:07:13.120 --> 00:07:13.600
Yeah, yeah.
00:07:13.680 --> 00:07:19.439
Listeners, we were talking a little bit before we hit record here about that sort of um the disconnect, right?
00:07:19.519 --> 00:07:27.360
With when you browse a website, you work with AI and insert various prompts and whatnot, and you don't realize sort of the infrastructure behind the scenes making everything possible.
00:07:27.519 --> 00:07:44.399
So I can just imagine that tangible uh, you know, connection and seeing all these like fiber optic cables and data, all this hardware, and just really kind of being fascinated by what's all required to make you know a website or uh an AI, you know, answer all possible.
00:07:44.720 --> 00:08:00.639
And I want to call out so you you're a certified internet nerd, as you uh call yourself, and uh you're also recognized as an Adobe champion, which is uh part of a group of product experts recognized by Adobe for sharing their expertise and and contributing to the broader Adobe community.
00:08:01.199 --> 00:08:03.120
Kudos to that, um, full stop.
00:08:03.519 --> 00:08:09.920
But you also helped create Nexus's safe harbor program when Magento 1 reached end of life back in 2020, I believe.
00:08:10.319 --> 00:08:20.560
Um, and the idea here was was to give merchants who weren't ready to move to Magento 2 a safer way to keep their Magento 1 stores running while they planned their migration.
00:08:20.720 --> 00:08:31.040
And the program included things like malware scanning, threat monitoring, and and backported security patches to help protect stores even after Adobe stopped providing official Magento 1 support.
00:08:31.360 --> 00:08:35.919
And then, of course, in the introduction, I also mentioned you worked as a community maintainer.
00:08:36.240 --> 00:08:49.519
Um so all that background info just really seems especially relevant when when obviously something goes wrong, inevitably when something, you know, like a major exploit happens, because your team isn't just looking at Magento as another application sitting on a server.
00:08:49.600 --> 00:08:52.720
You really sort of know the platform, you know the industry inside and out.
00:08:53.279 --> 00:09:02.639
My my next question for you is is so when we talk about quote unquote managed hosting, what should that actually mean when there's a serious security incident?
00:09:02.720 --> 00:09:06.799
Like what should a merchant expect their hosting provider to take care of?
00:09:07.360 --> 00:09:12.559
Well, this is actually one of the best questions we get when people is asking us, well, what's managed hosting?
00:09:12.720 --> 00:09:13.840
Where do you draw the line?
00:09:14.000 --> 00:09:22.320
As in what's application level support, what's not application level support, and one how far would you go as a hosting company?
00:09:22.559 --> 00:09:25.840
For me, I think the gray line is always security.
00:09:26.080 --> 00:09:33.039
If there's something urgent happening, if you are being actively exploited, I'm gonna go and I'm gonna act on behalf of these merchants.
00:09:33.279 --> 00:09:44.960
It could be blocking a request, it could be securing our environment, it could be a series of actions we do to try to bring the site into a safe space.
00:09:45.120 --> 00:09:51.759
But on the other hand, you gotta balance between being secure and not being able to access your own site.
00:09:51.919 --> 00:09:53.679
I'm gonna give you a really simple example.
00:09:53.919 --> 00:10:04.240
One of the first things we did for the style smuggler exploit was to try to block the slash GraphQL endpoint, and that actually left some of these sites unaccessible.
00:10:04.320 --> 00:10:05.919
And that's not an acceptable solution.
00:10:06.000 --> 00:10:10.559
You know, you don't get to make a merchant safer by making their site unavailable.
00:10:10.720 --> 00:10:18.799
So we were trying to better understand how far could we go into blocking this type of request, or if we wanted to block the entire endpoint.
00:10:18.960 --> 00:10:26.000
And that's when we realized that by blocking the request, we were blocking one of the ways the attackers were trying to get in.
00:10:26.080 --> 00:10:30.720
But at the same time, we were blocking some of legitimate, some of these requests were legitimate.
00:10:30.960 --> 00:10:47.360
So we had to be very specific about the stuff we wanted to block, or very specific about the actions we wanted to take in order to be able to one, secure the site, and two, be able to maintain the site active and merchants being able to process their own orders, you know.
00:10:47.519 --> 00:10:54.159
Because at the end of the day, if you are like, yeah, we secured your store, but you can no longer receive orders, that's not great.
00:10:54.240 --> 00:10:55.919
You know, that's not a great merchant experience.
00:10:56.000 --> 00:11:03.200
And even when, yes, we block these requests and yes, we make your site safer, it doesn't really work because we stop your business.
00:11:03.360 --> 00:11:07.600
So you gotta be you gotta be really careful where you draw the line.
00:11:07.919 --> 00:11:17.759
But I think at the end of the day, managed hosting usually means the hosting or the cloud provider actually knows the application you're installing and they are able to support that.
00:11:17.919 --> 00:11:25.840
Because it's not the same to go and look for a log file from a Laravel application or from a WordPress application or from a Magento application.
00:11:26.080 --> 00:11:40.000
It's very different, and that ability to be able to understand the underlying application and how it works under the hood is what makes a managed hosting interesting, or at least the one you want to use, you know, the one you want to hire.
00:11:40.159 --> 00:11:52.960
And at the same time, there's always a grain line where you are like, okay, I'm gonna go this far to try to support your application, but then at the end of the day, I am gonna be the one doing the development work.
00:11:53.120 --> 00:11:54.320
The answer is usually no.
00:11:54.480 --> 00:12:07.759
So new features, data are not included in the managed application stuff, but backfixing, it might be included or not, depending on the severity, critical issues like the security issue we are talking about, absolutely yes.
00:12:07.919 --> 00:12:20.960
And that's why we tried to one, scan the entire fleet first to try to understand where we were sitting, as in Magento versions, PHP versions, anything we wanted to know about these instances.
00:12:21.200 --> 00:12:25.200
And then we went ahead and we patched each and every of these sites.
00:12:25.279 --> 00:12:27.360
And that's a really big lift if you think about it.
00:12:27.440 --> 00:12:30.240
Because let's say I have 10,000 Magento sites.
00:12:30.320 --> 00:12:44.960
I have to one, scope the 10,000 Magento sites, two, try to understand their current architecture, their file system, how they are hosted, three, see the versions they are using, and four, apply the patch and be able to keep using that site.
00:12:45.120 --> 00:12:50.720
Because that's the other problem you face when you try to do N-Mass upgrades or N-Mass patching.
00:12:50.960 --> 00:12:54.720
Some of the times, these sites are going to be hugely customized.
00:12:54.799 --> 00:13:01.679
They will have tons of third-party modules that might or might not be compatible with the patch you are trying to apply.
00:13:01.759 --> 00:13:13.600
So the patching not only includes the code piece, but also trying to see how the site behaves before the patch, and then we apply the patch, and then we see how the site behaves after the patch has been applied.
00:13:13.759 --> 00:13:23.039
You know, some of the times it won't work, a lot of the times they will, but most of these merchants they don't want a broken site as a result of an unsolicited patch.
00:13:23.200 --> 00:13:33.919
Even if it's a security patch, we will want to make this site stable, even if it's not the latest solution or the patch we want to apply.
00:13:34.080 --> 00:13:41.679
But at the same time, we'll only release patches to up to a certain version, which are the ones who are being actively maintained.
00:13:42.080 --> 00:13:46.000
But that didn't cover all the versions we had in stock, you know.
00:13:46.159 --> 00:13:58.879
So maybe some merchants they are in the latest version or they are not even in maintained versions, and we had to patch them nonetheless because they were still in danger and they still were being actively exploited.
00:13:59.200 --> 00:14:06.480
So I had to go in there and backport these patches from the closest ones to the oldest ones.
00:14:06.720 --> 00:14:10.639
And that was great, mostly because I was able to use AI.
00:14:10.720 --> 00:14:19.919
And just like you said, yeah, well, AI is being used against all of these, but at the same time, we are also using AI to face or to battle the attackers, you know.
00:14:20.159 --> 00:14:25.840
And by giving a patch to one of these large models, I was able to backport them pretty quickly.
00:14:26.000 --> 00:14:29.279
And I was doing the same work manually 10 years ago.
00:14:29.440 --> 00:14:32.159
And that's how long we have come as an industry.
00:14:32.320 --> 00:14:40.320
Right now, we get to defend ourselves using the same tools the attackers are using, and we are faster and we are better because of that.
00:14:40.639 --> 00:14:42.000
Yeah, it's fascinating.
00:14:42.080 --> 00:14:55.840
There's so many, there's so many elements that that a managed hosting provider that you have to kind of be aware of and and consider when you're when you're managing all these different environments and individual, not just individual storage, you're managing all these different uh environments.
00:14:56.000 --> 00:15:11.279
I'm curious, like I actually I want to take a big step back really quick and ask maybe if you can give us a kind of a high-level overview of really what happened with the style smuggler exploit, starting with like when your team first realized when something might be wrong.
00:15:11.360 --> 00:15:13.200
And then maybe we can kind of go from there.
00:15:13.679 --> 00:15:22.639
Yeah, so I was browsing Twitter, well, now it's on LinkedIn on probably Saturday, a couple of weeks ago.
00:15:22.879 --> 00:15:32.960
And I kept seeing things about, yeah, there's an active exploit, there's an active attack from some of the security researchers I follow, for some of the ecosystem or like community people I follow.
00:15:33.120 --> 00:15:35.919
And I was like, this is something I have to bring to Nexus.
00:15:36.320 --> 00:15:41.679
And then the team was already aware of some of these requests being malicious.
00:15:42.000 --> 00:15:47.120
It kind of depends on the amount and the type of requests we see.
00:15:47.360 --> 00:16:01.600
But for example, we will see, I don't know, 100 requests that look the same, and we will flag them as suspicious, not because they might be problematic, but because they are being repeated on a ton of different stores setting different stuff.
00:16:01.840 --> 00:16:10.080
So that's the first type of alert we have similar requests being sent to different servers for different brandings for different verticals.
00:16:10.320 --> 00:16:18.639
That doesn't really happen that much, even if we are if even if they are all using the same application, because the request they might look the same.
00:16:18.799 --> 00:16:23.360
But we have our own web application fireworks to be able to detect some of these files.
00:16:23.440 --> 00:16:28.879
And then a second layer we have is the malware scans we do fleet-wide.
00:16:29.519 --> 00:16:34.240
And one of the things we do is to try to scan the fleet probably daily.
00:16:34.399 --> 00:16:43.120
And when we find similar files being hosted in different servers across the fleet, we will send an alert and be like, hey, this is something we have to manually look.
00:16:43.600 --> 00:16:53.679
And if you combine these two signals with a message from one of our security researchers or one of the guys I know from the community, it's probably a really clear picture.
00:16:53.840 --> 00:16:56.399
Something is happening, we gotta act fast.
00:16:56.639 --> 00:16:57.600
And that's what we did.
00:16:57.679 --> 00:17:20.240
We spun up an incident internally, we brought all the teams who were trying to understand what's going on, and then we did a new scan of the fleet to try to better understand how far these attackers were able to go, how complicated the attack was, and if the attack was being actively exploited, because just like I told you, these attacks they usually are divided into two pieces.
00:17:20.400 --> 00:17:34.880
One, they try to inject something into your site, usually a web shell, and then they call that web shell, which is running on the file system, to try to inject something into the browsing sessions or the checkout to try to steal your cards.
00:17:35.119 --> 00:17:39.359
Now, when this started, the pattern the pattern was clear.
00:17:39.519 --> 00:17:48.720
Something is being injected into the servers, nothing is being taken off of the server, so it wasn't being actively exploited, but actively, you know, injected.
00:17:48.960 --> 00:18:00.000
And that was one of the main differences we saw because a couple of days after that, we actually saw really a really large activity where these attackers were trying to hit the web shell they were trying to inject.
00:18:00.160 --> 00:18:05.039
And that was when we see it was a two-piece layer attack, you know.
00:18:05.599 --> 00:18:12.079
So yeah, you you you noticed that a really suspicious pattern, and then you realized you kind of figured out this wasn't just an isolated event.
00:18:12.160 --> 00:18:14.160
There was something much bigger going on here.
00:18:14.480 --> 00:18:23.200
How do you make that call like when protecting customers might also mean sort of temporarily disrupting some like legitimate functionality?
00:18:23.359 --> 00:18:23.680
You know?
00:18:24.000 --> 00:18:26.559
It's all about tackling the the you know the incident, right?
00:18:26.640 --> 00:18:30.720
So I mean it kind of you have to weigh those those pros and cons, the risk there.
00:18:30.799 --> 00:18:40.400
But how do you sort of make that call and and decide to to, you know, one, this is a credible threat, but also, you know, there needs to be uh a big sort of response here.
00:18:40.960 --> 00:18:51.519
Yeah, so this is a pretty large hosting company, and we get to see a ton of information or requests or flip-wide analysis even before we get to make a decision.
00:18:51.680 --> 00:18:54.240
So the way we decide is based on data.
00:18:54.480 --> 00:19:03.359
If this is a widespread attack who's been systematically targeting our infrastructure, it's gonna it's it's gonna need to be a fast actor.
00:19:03.440 --> 00:19:06.640
You know, we gotta act fast and be like, okay, what are we gonna do?
00:19:06.799 --> 00:19:12.079
First, we block the way they are trying to use to intrude these servers to we try to patch.
00:19:12.240 --> 00:19:15.920
This is not the first time, and it's not gonna be the last time something like this happens.
00:19:16.160 --> 00:19:20.799
So we already have a process and we have a way to determine yes, this is widespread.
00:19:21.039 --> 00:19:22.480
No, this is targeted.
00:19:22.640 --> 00:19:32.160
But at the same time, the attackers they already know us, they already know our AP range, they already know our fleet because we do this and we have been doing this for a really long time.
00:19:32.319 --> 00:19:35.759
So if you want to attack Magento, you gotta attack a company like this.
00:19:36.000 --> 00:19:41.599
Not because we are, I don't know, the only ones doing this, but we are one of the largest hosting companies.
00:19:41.680 --> 00:19:43.680
And that's how these attackers think.
00:19:43.839 --> 00:19:50.480
Where are they gonna target smaller merchants or smaller hosting companies when they can attack a company like this?
00:19:50.640 --> 00:19:52.720
A company like this in Europe, you know.
00:19:52.799 --> 00:20:00.160
So they know the AP ranges, they probably know most of the Magento domains, and they try to target this to the stores they can have.
00:20:00.319 --> 00:20:06.240
Because at the end of the day, these resources they are using this cost money, and it costs money to them to try to run these attacks.
00:20:06.400 --> 00:20:15.039
So they are getting each, like after each attack, we understand they are getting better and better to try to target us, to try to target our merchants.
00:20:15.200 --> 00:20:23.119
And that's why we have so many layers between the request that's being sent and the request your actual PHP application is getting.
00:20:23.279 --> 00:20:27.759
And that's why we analyze those requests with a why with a web application firewall.
00:20:27.920 --> 00:20:30.880
That's why we scan your file system every day.
00:20:31.119 --> 00:20:33.680
Because we know what these attackers are trying to do.
00:20:34.000 --> 00:20:50.640
But at the same time, and on the other hand, they are analyzing the application all day, every day to try to change this, to try to change these attacks and be able to, you know, sneak into the file system and then later on try to execute something they can use to steal cards.
00:20:50.880 --> 00:20:56.319
So it's more of like a process we already have and follow when you have this type of incidents.
00:20:56.559 --> 00:20:59.759
And sometimes it's gonna be like, okay, this is not widespread.
00:21:00.000 --> 00:21:08.319
This is something you can remediate with the malware scan on the, or this was targeted to one of our larger merchants because they think they have a better opportunity.
00:21:08.480 --> 00:21:13.599
But sometimes it's gonna be something like, hey, all of our servers are being hit with the same request.
00:21:13.839 --> 00:21:20.799
We found this file signature in 90% of the servers or 70% of the servers, this is a widespread attack.
00:21:20.880 --> 00:21:23.279
We have we have to spin up an incident now.
00:21:23.759 --> 00:21:30.559
So you you go from yeah, trying to contain it and then move on to the fixed portion once you you obviously contain the attack.
00:21:30.799 --> 00:21:39.119
And I know with with the containment phase, you uh you with with South Smuggler, you temporarily blocked GraphQL traffic before an official patch was available, right?
00:21:39.440 --> 00:21:51.920
And then that's sort of uh the product of that is that it reduces the risk, but it also sort of causes issues, potentially can cause issues with uh storefronts that utilize the progressive web app kind of dynamic there.
00:21:52.559 --> 00:21:59.759
So you kind of had to lay that sort of the the two sides of that decision when you're acting quickly of like, you know, do we you know contain this?
00:22:00.400 --> 00:22:05.519
This issue at the risk of rendering some of these store threats, you know, unusable.
00:22:05.680 --> 00:22:05.839
Right.
00:22:06.000 --> 00:22:10.480
So it's kind of like you're you're that's a tough decision to make, I imagine, especially in the heat of the moment.
00:22:11.039 --> 00:22:17.359
Yeah, it's it's pretty much always the same when it comes to security, not all, not only cloud security.
00:22:17.519 --> 00:22:23.680
When you're trying to secure something, you gotta understand what are the consequences of securing that.
00:22:23.920 --> 00:22:27.119
So consequences for this are pretty bad.
00:22:27.279 --> 00:22:34.000
As in some of these stores, they won't be able to process orders, which is probably the worst thing you can do for a merchant.
00:22:34.079 --> 00:22:38.319
You know, if they can't process orders, why are we even doing it here?
00:22:38.559 --> 00:22:48.640
So, first of all, we tried to act like that, but then we had to relax that GraphQL block, and after right after that, we got the patches, so it didn't last that long.
00:22:48.720 --> 00:22:58.559
But yeah, you gotta you gotta figure a way to not to disrupt the merchant operation and at the same time not putting the merchant customers at risk.
00:22:58.720 --> 00:23:03.440
Because at the end of the day, this is not a risk for Nexus as a company, this is a risk for the merchants.
00:23:03.599 --> 00:23:12.240
They are gonna be the ones being affected, they're gonna be the ones who are gonna have their customer race, I don't know, breach, their credit card stolen.
00:23:12.400 --> 00:23:19.119
It's not like for me as a hosting company, I can let these merchants run an infected store all day, every day.
00:23:19.279 --> 00:23:21.759
But at the end of the day, it doesn't look great.
00:23:21.839 --> 00:23:23.599
And if we can help them, we will.
00:23:23.759 --> 00:23:26.480
But just like I was telling you, this is a merchant issue.
00:23:26.559 --> 00:23:39.599
This is something they are gonna be facing, and this is something that it's gonna have consequences for them as a company, as a seller, as a merchant, and even for their customers, because who's gonna be paying for that?
00:23:39.759 --> 00:23:44.240
Who's gonna have their credit cash sold and it's gonna be the customer from these particular merchants?
00:23:44.400 --> 00:23:51.839
So you gotta find a way to one, let this merchant operate, and two, make the risks lower to sell in a way.
00:23:51.920 --> 00:23:54.960
But yeah, it's a pretty significant trade-off.
00:23:55.200 --> 00:24:08.799
And for attacks like this, it's gonna be it's gonna become a norm and it's gonna be harder and harder to decide what goes, what works as a containment and what doesn't, and what disrupts the merchant operation and what doesn't.
00:24:09.119 --> 00:24:09.839
Yeah, absolutely.
00:24:09.920 --> 00:24:16.480
And then it comes down to like a just a communication matter where you have to communicate, you know, why you you took the decision that you took.
00:24:16.640 --> 00:24:24.079
And um, you know, because a lot of times the merchant who's not very uh maybe technical oriented, they don't really know the lot of the back-end applications.
00:24:24.160 --> 00:24:28.559
They just want their store to work, they want people to be able to check out and paper things, et cetera.
00:24:28.720 --> 00:24:38.960
Um, you have to communicate that, you know, this is worth the sort of downtime if I try to contain this instant because it could result in a much bigger problem for the merchant.
00:24:39.359 --> 00:24:45.200
No, I absolutely love this approach uh because it gives us a glimpse of the reality we face.
00:24:45.359 --> 00:24:53.519
We have two very defined types of merchants, one highly technical, and the other one is not technical at all.
00:24:53.680 --> 00:24:58.319
So for the highly technical merchants, sure, we can communicate them fairly quickly.
00:24:58.480 --> 00:25:08.880
Hey, this happened, this is how we resolve, these are the steps you have to take in order to improve or I don't know, add these fixes to your own code base.
00:25:09.200 --> 00:25:15.519
But then if you tell the same to a non-technical merchant, they are gonna ask you, why don't you do it?
00:25:15.680 --> 00:25:17.519
or wonder why do I have to do it?
00:25:17.680 --> 00:25:22.640
You know, so we gotta balance the responses we get to these merchants.
00:25:22.720 --> 00:25:26.960
But just like you said, one of the key things in any incident is communication.
00:25:27.200 --> 00:25:32.400
And we were pretty fast in communicating, something is happening, we are already aware of this.
00:25:32.559 --> 00:25:37.359
We perform X, Y, and C to try to improve the situation.
00:25:37.599 --> 00:25:44.799
And then we send some more communications being like, hey, we are gonna be patching your site, these are the patches we are gonna be using.
00:25:44.960 --> 00:25:48.559
You need to include this in you into your own code base.
00:25:48.720 --> 00:25:52.240
Because most of these merchants are using Git for version control.
00:25:52.559 --> 00:26:00.559
And if I go and patch these sites directly in production, some of the patches they might be lost the next time they deploy this application.
00:26:00.720 --> 00:26:04.400
Because I applied these patches to their servers directly.
00:26:04.559 --> 00:26:08.720
I didn't go through their deployment process because we are not a part of it.
00:26:08.799 --> 00:26:14.160
You know, they won't be deploying through us, they usually deploy directly into the file system.
00:26:14.319 --> 00:26:20.559
So it's not like I can go and include the fix we did or the code we added directly into their code base.
00:26:20.799 --> 00:26:25.839
I was mostly patching their servers directly in production, which is a hot patch when you think about it.
00:26:26.000 --> 00:26:35.920
But then I need them to bring this patch to their code base and run it through their own QA, UAT, or development process, or deployment process or whatever.
00:26:36.079 --> 00:26:36.559
You know?
00:26:36.960 --> 00:26:46.240
Yeah, you mentioned the the Adobe patches, and specifically, I um I know that uh there's the term that backporting uh in regards to Adobe's patches.
00:26:46.319 --> 00:26:52.640
For someone who isn't deep in the Magento sort of world, um, what does backporting actually mean?
00:26:53.039 --> 00:26:54.319
It's a really simple idea.
00:26:54.480 --> 00:26:59.680
You take a patch for a supported version and you backport that to unsupported versions.
00:26:59.839 --> 00:27:01.359
I'm gonna give you a silly example.
00:27:01.519 --> 00:27:04.640
The latest Adobe Commerce version is 2.4.9.
00:27:04.880 --> 00:27:14.079
Adobe, I think, is currently supporting as an inactive support magento 2.4 point something and above, and everything below that is not supported.
00:27:14.319 --> 00:27:24.880
So if at any point you have to patch an unsupported site, you will need to take the patch Adobe created and then backport that to another version.
00:27:25.039 --> 00:27:30.000
And we did that to Adobe 2.2 and 2.3, if I recall correctly.
00:27:30.240 --> 00:27:35.359
And below that it wasn't needed because the feature we were patching wasn't present.
00:27:35.599 --> 00:27:41.440
But at the end of the day, Adobe will only patch that to a certain version, and below that, you're on your own.
00:27:41.599 --> 00:27:43.279
And that's what backporting means.
00:27:43.359 --> 00:27:49.039
You take one of these patches and you port that back to a lower version, which is not being actively supported.
00:27:49.200 --> 00:27:50.160
And that's what we did.
00:27:50.240 --> 00:28:03.359
You know, some of our merchants they are still on unsupported versions, and it's not like we can upgrade them to Magento 2.4.9 because of compatibility issues, third-party models, and a ton of other issues they might experience.
00:28:03.440 --> 00:28:06.079
So we wanted to keep them safe.
00:28:06.240 --> 00:28:12.480
And that's why we've imported some of these patches to be able to be applied to the other versions of Adobe Commerce.
00:28:12.799 --> 00:28:21.599
Yeah, that that is one of the challenges, right, with this issue is like and you have a patch, is uh customers are obviously running different, you know, Magento versions, different configurations.
00:28:21.759 --> 00:28:27.680
So it's you have to you have to kind of go back and roll out patches for different sort of versions.
00:28:27.920 --> 00:28:31.519
And you can't just push the exact same fix everywhere, I imagine.
00:28:32.000 --> 00:28:40.000
How do you uh like so once you have a patch that's been you know rolled out, you still have live e-commerce stores that need to keep running?
00:28:40.160 --> 00:28:47.359
Um, how do you make sure a patch fixes the security problem without creating you know a completely different problem for the merchant?
00:28:47.759 --> 00:28:48.880
That's a tough question.
00:28:48.960 --> 00:28:55.440
And it usually usually has that kind of assurance to say in a way.
00:28:56.000 --> 00:29:00.720
We know what the issue was, we knew how it was being exploited.
00:29:00.880 --> 00:29:04.400
We try to patch that, and I thought we tried to patch that.
00:29:04.720 --> 00:29:17.680
It's kind of hard to understand what a patch can do into your site, but I am gonna say that most of these patches are battle-tested, as in it's not like we backport the patch and we forget about it.
00:29:17.759 --> 00:29:19.839
We try to test this in a few scenarios.
00:29:20.079 --> 00:29:27.759
And not only we test other security issues, but we also test site stability once the patch is being applied.
00:29:27.920 --> 00:29:29.359
Because that's the other thing.
00:29:29.680 --> 00:29:34.880
You don't know how a patch is gonna work with the third-party modules you have in your store.
00:29:35.039 --> 00:29:51.200
Sure, Magento has a pretty comprehensive core, and you can get a ton of out-of-the-box features, but most of these stores they are heavily customized and they use a ton of third-party modules, extensions, and they don't always play well with these patches.
00:29:51.279 --> 00:30:05.359
So you gotta understand what you are touching and how these new patches are interacting with the current code and not only with the core code, which was tested against, but also against the third-party modules these merchants are using.
00:30:05.519 --> 00:30:07.039
And that combination is infinite.
00:30:07.119 --> 00:30:12.079
It's not like there's absolutely no way you can test all the permutations and all the combinations.
00:30:12.240 --> 00:30:20.240
So you try to test a few of them, the most well-known, and you have a bunch of canary sources you use to try to understand how these patches will behave.
00:30:20.480 --> 00:30:26.319
But at the end of the day, you are you have to tell the merchants, hey, we patched this, we did this, this, and that.
00:30:26.480 --> 00:30:38.480
Now you gotta go back onto your development team and try to pattern test this some more to better understand how your particular site, which is not like all the other sites, will behave once we apply these patches.
00:30:39.039 --> 00:30:42.559
How how much can that patching process actually be automated?
00:30:42.720 --> 00:30:47.680
Because like where do you still need that the sort of experienced people making the judgment calls during stuff like this?
00:30:47.759 --> 00:30:58.240
Because if you have all these patches that are like potentially, I don't know, also causing issues with all the different configurations and whatnot, I imagine it's a it's a hard problem to automate, right?
00:30:58.960 --> 00:31:00.559
It's a yeah, absolutely.
00:31:00.640 --> 00:31:14.079
It's a pretty hard problem to automate because you not all the sites look the same, meaning that some of these sites they would have different PHP versions, different Nginx versions, different Maria TV versions, even different Magento versions.
00:31:14.319 --> 00:31:26.319
And that means that you have to create or you have to maintain or have a patch for each of these versions, and that only includes the core code, meaning what comes with Magento out of the box.
00:31:26.640 --> 00:31:28.240
So that's great.
00:31:28.480 --> 00:31:36.319
We can probably automate patching an out-of-the-box store, but probably 1% of these stores are not really customized.
00:31:36.559 --> 00:31:44.720
So it's not like we can automate the entire patching and upgrade stores, mostly because of the third-party motives the merchants are using.
00:31:44.960 --> 00:31:48.720
Now, if you ask me, I will try to automate this as much as I can.
00:31:48.880 --> 00:31:51.920
And I'm gonna try to do it this year for Nexus.
00:31:52.240 --> 00:31:56.799
But at the end of the day, we will need to find a way to have better third-party extensions.
00:31:56.960 --> 00:32:05.039
We know work well, we know our well architecture, we know they are using the best practices to be able to automate this as much as we can.
00:32:05.279 --> 00:32:16.400
Because if we are all using the best practice and we have better extensions, we will be able to patch this pretty easily, even if they differ in versions, software versions, system requirement versions.
00:32:16.720 --> 00:32:23.759
But we gotta standardize the stores as much as we can to be able to test this in a standard way.
00:32:23.920 --> 00:32:40.960
Because if I go out there and patch a 2.2 store, 2.3 store, 2.4 store, and they all behave differently, it's gonna take me a really long time to be able to understand or to try to standardize the way these stores should look once the patch is applied.
00:32:41.119 --> 00:32:49.599
And it's the issue we have right now: a ton of different versions, a ton of different system requirements, and a ton of different third-party modules.
00:32:49.759 --> 00:33:12.000
So if we want to patch all these stores, or if we want to keep these stores updated, chances are we will break them, mostly because they are using third-party modules we do not control, and we don't have a really simple way to standardize the testing we need to try to understand that once we apply a patch, this site is still working normally.
00:33:12.640 --> 00:33:18.880
Yeah, that's a big problem to solve because you have a lot of these tacks that are being automated in the first place.
00:33:19.039 --> 00:33:29.039
And um, you know, it takes a lot of like kind of manpower and um and remediation time to roll out the patches and and the solutions.
00:33:29.279 --> 00:33:38.799
So yeah, you need to find a way to sort of speed up the the solution process as much as possible to prevent a lot of these incidents from unraveling into a bigger problem.
00:33:39.359 --> 00:33:43.599
Yeah, I have so here's the thing about these type of applications.
00:33:44.240 --> 00:33:50.880
This is a self-hosted type of application where you run your code like the code is yours.
00:33:51.119 --> 00:34:05.039
I'm gonna draw a parallelism between Magento and Shopify, for example, where if you are using Magento or Adobe Commerce, you will have to maintain the platform by yourself, which usually gives you way more flexibility and modularity.
00:34:05.200 --> 00:34:12.000
And in the case of Shopify, this is something they already solved because they don't let you touch the core code.
00:34:13.199 --> 00:34:18.559
It's not a self-hosted application and someone else is hosting and maintaining the application.
00:34:18.800 --> 00:34:21.760
So that's one of the options we are actually considering.
00:34:21.920 --> 00:34:34.800
We are trying to probably create some sort of package where merchants can run, I don't know, a hundred of better extensions, and we will try to keep these stores up to date.
00:34:35.039 --> 00:34:37.199
Well, not try, but actually do it.
00:34:37.360 --> 00:34:43.199
But it really depends a lot on the type of architecture, on the type of models and extensibility you want to have.
00:34:43.440 --> 00:34:48.159
And at the same time, we have merchants who really want to go into the opposite direction.
00:34:48.400 --> 00:34:57.920
They really want to own their own infrastructure, they really want to own and be able to audit all the code they are running, and that's why they choose platforms like this.
00:34:58.079 --> 00:35:06.559
And that's why we have merch, huge merchants who are selling a couple of millions per hour trying to still run Magento in a safe way.
00:35:06.800 --> 00:35:23.599
And it really depends on the type of merchant and the type of business, because at the same time, you would have merchants who are not allowed on platforms like Shopify or BCommerce, who are really trying to still sell online with different guidelines, with different rules or different laws.
00:35:23.760 --> 00:35:40.480
And that's why we still see a really large number of huge merchants running on Adobe Commerce, because it gives them the flexibility and the possibility to run their own stuff locally, or at least not being tied to a company telling them, yes, you can sell this, no, you can sell that.
00:35:40.960 --> 00:35:45.199
Yeah, because you're obviously concerned uh a lot with uh the hosting side.
00:35:45.280 --> 00:35:51.840
So much of what we've kind of talked about had happens on the hosting side and you know, the automation there, the patches and whatnot.
00:35:51.920 --> 00:36:02.880
And but like, how do you decide like what Nexus can safely do on behalf of a customer and what really needs to be handled by the merchant themselves or their development team, if that makes sense?
00:36:03.039 --> 00:36:08.480
Like, you know, there's that kind of the the the merchant needs to take some responsibility too, right?
00:36:09.039 --> 00:36:10.159
Absolutely, yeah.
00:36:10.480 --> 00:36:14.800
So theoretically, the entire responsibility lies on the merchant.
00:36:14.960 --> 00:36:25.519
100% of their operation is gonna be on the merchant, but when it comes to security, we always try to go above and beyond and try to, you know, help them as much as we can.
00:36:25.760 --> 00:36:28.000
Same goes with application level super.
00:36:28.639 --> 00:36:32.880
We not only have incidents, we also have a really large support operation.
00:36:33.039 --> 00:36:37.679
And these guys, they will try to go beyond and try to fix anything they found.
00:36:38.000 --> 00:36:48.480
But at the end of the day, if we can't fix something, we will have to push back and be like, hey, maybe you want to bring this into your development team because this is as far as we can go.
00:36:48.719 --> 00:36:53.599
Or at least we will be able to tell them, hey, the logs say X, Y, and C.
00:36:53.760 --> 00:36:56.320
This usually means this, this, and that.
00:36:56.559 --> 00:36:59.760
You can bring these discoveries to your development team.
00:36:59.920 --> 00:37:02.880
Some of these merchants, they would even have a development team.
00:37:03.039 --> 00:37:08.880
But most of them they will because that that's one of the requirements when you're trying to run an application like this.
00:37:09.119 --> 00:37:24.079
You don't get to rely entirely on the support, or you don't get to rely entirely on the cloud you are using, but you really need a development team, which usually looks like an agency, but sometimes looks like a freelancer working for you or your company.
00:37:24.239 --> 00:37:36.960
But yeah, when when we need to decide when to act, it's usually around security issues, or your site being down, or something that is actually something we can fix.
00:37:37.199 --> 00:37:45.679
But on the same, on the other hand, we need to be really careful when it comes to going into these merchant sites and changing something.
00:37:45.920 --> 00:37:56.719
Because again, this is not our code, this is not something we are supposed to be doing on a daily basis, and that's why we try to communicate clearly about the expectations.
00:37:56.880 --> 00:38:01.760
There's a security issue, this is what we are going to be doing, this is what you need to do once we are done.
00:38:01.920 --> 00:38:03.199
And it usually works.
00:38:03.360 --> 00:38:13.760
Some of these merchants they are pretty happy with the service, and most of these merchants they already had a plan, even before we had a plan because they had their own agencies or their own developers.
00:38:14.000 --> 00:38:20.880
So we are working with them or with their agencies with our findings and fleet-wide analysis.
00:38:21.119 --> 00:38:32.079
And they are on show also working with their agencies, with their developers to try to, you know, fix their issues, try to cover the security stuff and that kind of stuff.
00:38:32.400 --> 00:38:48.559
Yeah, it's so much of it comes down to the communication, like we kind of talked about a little bit earlier, just you know, establishing that good communication from the hosting provider, kind of outlaying the situation, um, you know, mentioning sort of like what's developing.
00:38:48.880 --> 00:38:59.920
You don't have all the answers yet, but you know, here's just the transparency of having uh of communicating the transparency to the merchants um who are just wondering, you know, is my store affected?
00:39:00.079 --> 00:39:01.119
Why isn't it working?
00:39:01.360 --> 00:39:02.480
You know, that sort of thing.
00:39:02.880 --> 00:39:07.760
That's that's the that's the huge part of this all is just the the communication I can imagine.
00:39:08.239 --> 00:39:15.920
I want to finish the the wrap the conversation up by making this really useful for somebody who might be choosing a hosting provider right now.
00:39:16.159 --> 00:39:31.360
Like if I'm a merchant, if I'm uh a technical leader at a company or an agency, maybe evaluating um let's say manage magento hosting, what should I really be asking a provider to figure out whether they're actually prepared for an instant like this?
00:39:31.920 --> 00:39:37.599
Well, I would start by asking them if they really know the underlying application they're hosting.
00:39:37.840 --> 00:39:41.760
Not if they know their own infrastructure as in, do you know your servers?
00:39:41.840 --> 00:39:43.440
The answer is going to be always yes.
00:39:43.679 --> 00:39:46.800
But do you really understand what's under the hood?
00:39:47.039 --> 00:39:49.599
Do you really understand how this application behaves?
00:39:49.840 --> 00:39:56.400
Do you have any experience in troubleshooting an issue that might be happening in your store?
00:39:56.639 --> 00:40:05.440
I think the most important thing to try to ask one of these cloud providers is how much they understand about the application you are trying to bring them.
00:40:05.679 --> 00:40:11.199
Now, most of these merchants they are running PHP applications, which is what we mostly support.
00:40:11.360 --> 00:40:24.079
But some of these merchants they will also have, for the larger ones, they will also have other requirements, as in host my ERP, connect to this system, try to add this web application firewall rule.
00:40:24.320 --> 00:40:32.400
So it really depends on how embedded these cloud providers are in the underlying applications they are hosting.
00:40:32.559 --> 00:40:37.679
For Nexus, we know this inside out because we have been doing this for probably 20 years.
00:40:37.840 --> 00:40:41.599
I'm one of the maintainers of the Magento 2 GitHub repo.
00:40:41.679 --> 00:40:45.920
I have a four-time Magento master and I am an Adobe Commerce champion.
00:40:46.079 --> 00:40:50.320
And that gives you some visibility into the inside of this ecosystem.
00:40:50.559 --> 00:40:55.199
And that's some of the resources we use to try to scope this type of issues.
00:40:55.360 --> 00:41:05.519
Because not only you need to understand what the underlying application is, but you also want to understand if the provider is involved into the ecosystem at a larger scale.
00:41:05.760 --> 00:41:20.320
So it there are a ton of things you can ask your provider, but for me, it usually depends on how much involvement they have into the ecosystem or if they collaborate at all, and if they really understand the underlying application and how it works internally.
00:41:21.599 --> 00:41:31.599
Those are good uh notes for people listening right now, if merchants listening, uh looking to talk to a potential uh managed hosting provider.
00:41:31.840 --> 00:41:45.599
Um is there anything in particular that you maybe you haven't mentioned yet or that you think merchants tend to maybe misunderstand about security or about like what their hosting providers should actually be doing for them?
00:41:46.079 --> 00:41:54.559
Yeah, what most of the merchants don't really understand, this is for smaller merchants maybe, is that the entire liability is on them.
00:41:54.800 --> 00:42:07.599
If they are attacked, if they have a card being stolen from their site, there's very little you can tell your cloud provider to do and the finds the PCI assessments or anything they need.
00:42:07.840 --> 00:42:09.199
It's gonna be on them.
00:42:09.519 --> 00:42:14.719
So the more upfront they are about security, the less it's gonna cost them in the long term.
00:42:15.119 --> 00:42:17.280
Mostly because sure we can help them.
00:42:17.440 --> 00:42:27.199
Yes, we will patch them, sure we can provide an attestation of compliance because we are PCI compliant, but at the end of the day, the liability is gonna be on the merchant.
00:42:27.360 --> 00:42:30.559
And this is something they want to prevent as much as they can.
00:42:30.960 --> 00:42:31.519
I love that.
00:42:31.840 --> 00:42:36.239
So the more upfront you are about security, the less it's gonna cost them in the long run.
00:42:36.400 --> 00:42:36.800
I love it.
00:42:36.960 --> 00:42:39.760
I think that's uh that's a great way, uh place to lead it, y'all.
00:42:39.840 --> 00:42:45.360
Um and I think uh uh one of the interesting things here is that you really see the value of just managed hosting when something gets wrong.
00:42:45.440 --> 00:42:58.400
It's not just about, you know, where your store is running, it's having people who understand the platform, can recognize what's happening, can respond quickly because the you know, time is of the essence in a situation like this, and then just ultimately help work through it with you.
00:42:58.559 --> 00:43:02.800
That's that's a that's a great value add for uh a managed hosting provider.
00:43:03.119 --> 00:43:19.760
But Miguel, thank you so much for just unpacking this incident and and helping kind of just understand the landscape of this crazy, this crazy world we're in with all these threats and um automations going on here, both on the threat actor standpoint and also the remediation standpoint here.
00:43:20.719 --> 00:43:22.239
I appreciate everything you shared with us.
00:43:22.880 --> 00:43:23.119
Right.
00:43:23.280 --> 00:43:27.199
Yep, it was great being here, and I sincerely appreciate all the time you took to prepare all these.
00:43:27.599 --> 00:43:28.400
So thank you again.
00:43:29.119 --> 00:43:29.679
Absolutely.
00:43:30.000 --> 00:43:32.320
Well, thank you all for listening to the SourceForge podcast.
00:43:32.400 --> 00:43:33.840
I am your host, Bill Hamilton.
00:43:33.920 --> 00:43:38.239
Make sure to subscribe to stay up to date with all of our upcoming B2B software related podcasts.
00:43:38.639 --> 00:43:40.639
I will talk to you in the next one.