TRANSKRİPT 🔗
WEBVTT
1
00:00:00.100 --> 00:00:07.660
Estimating is silly Mm-hmm It's a waste of time, it's a waste of money, and so on and so forth. Estimation, no. Fire the person who estimates.
2
00:00:07.680 --> 00:00:19.820
[laughs] I want to have as many dev environments as I have people in my company Mm-hmm I want to have thousands, but I want them to be active and to be running only when they're used.
3
00:00:20.350 --> 00:00:32.020
The easiest way to justify anything is to say, "If we invest this amount of money, then we will get this amount of money back." Right? Right. That's the easiest way to justify it. Yeah. But it's not really about FinOps.
4
00:00:32.759 --> 00:00:46.380
Y- you don't want to scale because it's cheaper. Cheaper, being cheaper is just a side effect. You want to scale because you will have no downtime releases, you will have better performance, and so on and so forth.
5
00:00:46.420 --> 00:00:59.080
You will get many other technical benefits, and also human benefits, many, many benefits, and all those benefits will at the same time result in, in lower cost. [laughs] Oh, we were very successful.
6
00:00:59.350 --> 00:01:07.860
We managed to survive Black Fri- Friday because we were scaling up, and then a week later kind of, "But did we scale down?" Thanks everyone for being here today.
7
00:01:08.100 --> 00:01:24.380
Um, today we have a very special guest into our FinOps podcast, uh, Viktor Farcic, which is someone that I've been following, I think, for two or three years, and then I happened to knew that, uh, that we have met up in several years before.
8
00:01:24.480 --> 00:01:47.220
Uh, so it was a, an honor for, for me to know that he was available for, um, for the podcast, and to talk him today into a topic that I think it's, um, really, really hot topic based on what we saw in, in FinOps, uh, X conference, that is all related to, um, DevOps costs, the Kubernetes, the resource optimizations.
9
00:01:47.760 --> 00:02:01.260
And so if you have had issues with Kubernetes costs, or you hate your DevOps because he's doing a lot of, or she is doing a lot of, uh, costs in your cloud bill, uh, you should stay and know what, what Viktor is, is going to bring us.
10
00:02:01.360 --> 00:02:14.480
So Viktor, thanks, thanks for being here today. Thank you for having me. So, and of course, uh, we always are honored to have our, uh, co-host, Damian. Damian Monafo, how are you doing, man? I'm doing great.
11
00:02:14.580 --> 00:02:30.240
First of all, it's very good to see Viktor here with us. I remember back in the days in the Sp- [laughs] Spain club, first Spain, Spain club summit, where Viktor joined us, um, to, um, uh, to that special event.
12
00:02:30.380 --> 00:02:41.240
I think it was great. We know, I know Viktor since then, and it's good to have you here with us and see you again. Thank you for joining. Yeah. It's amazing being back. Yeah.
13
00:02:42.760 --> 00:02:55.940
So, so Viktor, uh, even though you are a well-known, uh, person in the, in the community, especially in the DevOps, but since this podcast is also oriented to more FinOps that are more specialized, some people may not know you.
14
00:02:56.040 --> 00:03:09.700
Um, so yeah, go ahead and, and tell us who you are and, you know, why are you, uh, here today. [laughs] Oh, who I am? I'm Viktor. I just play with random things and- [laughs]...
15
00:03:09.880 --> 00:03:18.130
you know, I, I don't even know what I do anymore. [laughs] It changes from one week to another. It's, it's impossible- You do-... for me to describe what I do- Yeah... to be honest. Yeah. You do a lot of tools.
16
00:03:18.130 --> 00:03:28.210
And even if I would describe what you do, then what I do, then my boss might find out what I do and then start demanding things. [laughs] Yeah, yeah, yeah, yeah.
17
00:03:28.560 --> 00:03:40.760
It will be, it will prob- it's not probably beneficial, so we'll, we'll keep it secret then. [laughs] No, but it will be all right. But, uh, and why I'm here, I don't know. You called me. You tell me why I'm here.
18
00:03:41.180 --> 00:03:58.360
[laughs] No, you're, you're here because, because we have seen that there is a, a topic in, especially in the FinOps community, where two, uh, I don't know, roles or battles, uh, lands, which is like the DevOps against the, the costs, uh, in the cloud.
19
00:03:58.540 --> 00:04:10.920
[laughs] So there is, there is a, a, a hot topic there because I, for example, I am, I'm a cloud DevOps engineer, even though I do a lot of, uh, FinOps stuff, and that's why I, I, I don't know, I check your, your things.
20
00:04:11.720 --> 00:04:25.730
So we've seen that, uh, the DevOps sometimes are more focusing to the reliability and the scalability and the performance of the, of the systems that we are seeing, and now there is a, a consideration of the costs, uh- But-...
21
00:04:25.730 --> 00:04:35.660
that we are seeing in FinOps, so what are your thoughts on this? Scalability and cost are the same thing, aren't they? Why so? Because so let me backtrack a second, right?
22
00:04:35.900 --> 00:04:53.020
I believe that the, the, the m- big part of FinOps is- Uh-huh... really resource management. Okay. I think that it's all about how can we make sure that we, that our resources are not using more than they need,
23
00:04:53.960 --> 00:05:09.320
but also not less than what they need. That they're j- just using just the right amount of resources, right? Mm-hmm. Uh, so if our application uses more or less the exact amount of memory or CPU than it needs, then more-
24
00:05:10.260 --> 00:05:27.520
Mm-hmm... that's how we're reducing costs, right? If our clusters are scaling up and down, uh, to, to pr- to provide just enough, uh, compute to what we need and not more- Mm-hmm... that's how we reduce costs, right?
25
00:05:27.910 --> 00:05:45.420
Uh, so in my head, and I might be wrong because FinOps is not my main area of interest, it's all about, uh, the resource management, right?
Load full transcript
Mm-hmm. And, and autoscaling is a big part of that, right? Otherwise- Mm.
26
00:05:45.430 --> 00:05:59.640
You know, if I cannot scale... So let's say a simple scenario, right? Mm-hmm. If I cannot scale my application up and down, up and down to meet the, the changes of the traffic and the demand- Mm...
27
00:05:59.680 --> 00:06:16.019
what am I going to do if I'm not doing that? I'm going to probably set my applications and my servers to use the maximum capacityUh, the or, the capacity that is equivalent to the maximum traffic I will have. Mm-hmm.
28
00:06:16.070 --> 00:06:29.039
Yeah. So let's say that right now in the morning I have 100 users and afternoon I have 1,000 users. Mm-hmm. If I don't scale automatically, then my capacity will be hard-coded to 1,000 users. Yeah, for sure.
29
00:06:29.350 --> 00:06:33.240
And I will be losing money when- whenever I have less than that. Mm-hmm. Yeah.
30
00:06:33.760 --> 00:06:48.060
But do you think, uh, so coming from, from the DevOps front, the practices that I am seeing is that do you think that DevOps tend to probably overestimate, uh, the provisioning or, or not care that much about probably the base provisioning?
31
00:06:48.600 --> 00:06:59.070
And then even though they set up the auto-scaling, which I do, the base provisioning is normally way over-provisioning, especially in Kubernetes. What do you... What are your thoughts on that? So, um,
32
00:07:00.540 --> 00:07:16.240
base provisioning is- What I mean is like when you deploy the workload and it's more or less stable, uh, do you think that, uh, initial estimate could be overpro- tends to be overprovisioning or do you think it's, it tends to be like, well, uh, say- Nobody should ever estimate.
33
00:07:16.450 --> 00:07:27.660
Uh, estimating is silly. Mm-hmm. It's a waste of time, it's a waste of money and so on and so forth. Mm-hmm. You don't estimate. Yeah. At least mem- you maybe estimate first day and that's it. Yeah. Right?
34
00:07:28.120 --> 00:07:37.520
After that you look at data, you don't estimate, right? Yeah, for sure. That, that's just silly. That's how we were doing things 30 years ago, right? I mean, nobody knows.
35
00:07:37.560 --> 00:07:44.680
So if, if I now develop a new application- Mm-hmm... I honestly have no idea how much memory and CPU that application needs. Mm-hmm.
36
00:07:44.900 --> 00:07:55.660
I have no bloody idea and I will challenge anybody who tells me that they know in advance how much- [laughs]... how many resources they need. Whomever says that, that, that person is lying or is, or is delusional.
37
00:07:55.780 --> 00:08:05.120
Nobody knows and that's fine because we have now mechanisms to find out. Mm-hmm. Yeah, for sure. We can actually even go further. There are...
38
00:08:05.180 --> 00:08:17.539
Actually we, th- there are certain things that we can do that, uh, will actually help us never need to find out, right? Mm-hmm. We can... If you're using Kubernetes, right? You have...
39
00:08:17.810 --> 00:08:26.990
And I'm not going even to the fancy solutions and expensive ones. Uh, vertical pod autoscaler is part of every Kubernetes distribution. Mm-hmm.
40
00:08:26.990 --> 00:08:41.820
You can use it in a, in a observe only mode to find out how much memory and CPU your application is using, right? It's not the best solution, but it's a good start. So estimation, no. Fire the person who estimated.
41
00:08:42.620 --> 00:08:54.820
[laughs] Okay. That's, that's quite a statement, but do you think that from, from that point when, uh, the, the application is deployed and when you, you, you know, you do your...
42
00:08:54.860 --> 00:09:10.280
You deploy, you put your requests, I don't know, like by a random value, uh, do you think that there is the, the co- the concept or the habit to do the right sizing after that or do you think that that's a practice that is still to be implemented in the- I don't...
43
00:09:10.290 --> 00:09:18.660
... uh, life cycle? Most of us have bad habits. Most of us don't know- [laughs]... what we're doing, of course. There is a bad habit in everything. You, you, you don't need to talk about FinOps.
44
00:09:18.860 --> 00:09:33.320
Anything you can mention and if you, if you talk about and if you end that sentence with a question, is there a bad habit? Of course there is. Yeah. We should just- Yeah... uh, try to remove our bad habits, right? Yeah.
45
00:09:33.440 --> 00:09:46.060
Okay. That's, that's part of the, the role. I ha- I heard one, uh, uh, the other day I was talking to an architect and he say, "I start with the smallest, uh, you know, resource and then start scaling about."
46
00:09:46.360 --> 00:09:57.400
I'm not sure if that is the, the right approach, but I, I thought it was interesting, an interesting one. You know, start small and, and grow a, a bit by, a bit by bit. What do you think?
47
00:09:59.440 --> 00:10:10.170
Yeah, I mean, I, I think that that's how we were doing before when we had, when we... So you know there were three phases in software industry, right? Mm-hmm.
48
00:10:10.190 --> 00:10:17.920
One phase is that I don't see what's going on so I'm making pure random guesses and we'll see what happens, right? Yeah.
49
00:10:18.710 --> 00:10:31.300
And then we s- then we learned l- uh, still a long time ago, okay, actually I can observe what's going on, right? I can have something as simple... I'm going to stick with Kubernetes now, but that applies to everything.
50
00:10:31.720 --> 00:10:31.730
Yeah.
51
00:10:31.760 --> 00:10:49.580
I can have something as simple as Prometheus where I'm collecting memory and CPU and I can just go there once a week, once a day, once a month, a year, whatever the frequency is, go and see what, how were resources used over time, what so not and adjust.
52
00:10:50.460 --> 00:10:59.520
Mm-hmm. And then now we don't even need to do that. We can instruct literally machines and say, "You know what? Go there yourself.
53
00:11:00.380 --> 00:11:14.680
Find out what
are the averages, uh, what are the rates and so on and so forth over a period of time and scale up and scale down," right? So there is no need for that architect to exist even for that particular task.
54
00:11:15.280 --> 00:11:25.700
You just define your applications in a way, way that they themselves can ask query databases with data- Mm-hmm...
55
00:11:25.720 --> 00:11:39.260
and find out what, what is it that they were doing and how they were doing and so on and so forth and adjust automatically. Yeah. Interesting. But that, that kind of like, uh, let me...
56
00:11:39.660 --> 00:11:51.230
I understand the, the point that you are, okay, fine if we have in production, you know, the, the, you need to have flexibility and the, and, and in order it's not only to scale but also makes sense, right?
57
00:11:51.320 --> 00:12:04.760
To make sure that it both scale up and down and it's flexible. Uh, but do you really need- Hopefully left and right rather than up and down- Yeah... but yes. Right. You're right. Yes. Uh, yeah.
58
00:12:05.020 --> 00:12:15.244
That's a, a, a best practice. But, uh, [laughs] but tell me something, um-How, how, how will you separate the environment?
59
00:12:15.364 --> 00:12:25.084
Because now we are seeing, like, this approach, you have a multi-s- account strategy, and in which in dev you wouldn't need so much scaling, only for testing times, right?
60
00:12:25.154 --> 00:12:39.464
And then, and then I guess that the question that Victor asked before comes more into, into play. How, how do, how do you plan it or how, how do you do this kind of... Do, do you even work in a multi-account strategy?
61
00:12:40.384 --> 00:12:52.564
But, you know, in a dev the logic is still the same, right? Mm-hmm. So if I have a system that goes to some, to goes to some database, something that has metrics- Yeah...
62
00:12:52.664 --> 00:13:01.184
and asks, okay, so what, what was the usage, this or that, and adjust accordingly by scaling up and down or left and right or both, right?
63
00:13:01.524 --> 00:13:13.714
So then if you have in dev, uh, if you have that in dev, uh, your memory CPU usage will most likely be much smaller, right? Right. And it will find much smaller numbers. Mm-hmm.
64
00:13:13.714 --> 00:13:20.704
And it will scale down to smaller number of replicas, right? So is, it, it, the logic is still the same, right?
65
00:13:20.764 --> 00:13:32.224
The, the question is only whether you wanna bother or no, but if you already set it up for production, uh, to be like that, then it's basically no additional effort to have the same scaling capabilities and, uh,
66
00:13:33.324 --> 00:13:47.384
whatever else you have, uh, in other environments. Uh, however, in, in dev environments, the real cost is not really the, you know, how much memory CPU you're consuming and
67
00:13:48.384 --> 00:14:02.854
through that how big of your nodes are and so on and so forth. The, the, the biggest bottleneck in dev for many organizations is that they're n- they're using permanent environments. Hmm. Yeah.
68
00:14:02.864 --> 00:14:16.914
What that means is that you're spinning up some development environment for the whole team, for the company, for individuals, whatever it is, and that something is running all the time. Yep. Right?
69
00:14:17.444 --> 00:14:31.364
And that produces two very potentially big issues. One is that you're wasting money like crazy, right? Uh, that doesn't matter how, whether your memory consumption is correct or no, you're still running compute
70
00:14:32.304 --> 00:14:48.354
while you're sleeping. Yep. And unless you're very, very rich, that's just silly. And at the same time, because those things are expensive, you are not only wasting too much money but never having enough, right? Mm-hmm.
71
00:14:48.354 --> 00:14:57.284
That's why usually organizations have one or two dev environments, and I, and to me, that's unacceptable. I no-- I don't want to have two dev environments.
72
00:14:57.644 --> 00:15:12.144
I want to have as many dev environments as I have people in my company. Mm-hmm. I want to have thousands, but I want them to be active and to be running only when they're used. Mm-hmm.
73
00:15:12.204 --> 00:15:29.564
So at the same time I'm going to reduce cost but also enable people to be effective because big part, and, and I feel that this is probably the, the area that is often forgotten in dev, in FinOps, is that the biggest cost are people.
74
00:15:29.744 --> 00:15:39.144
Let's start with that. Yeah. That's what costs the most, right? And whenever somebody needs to wait for something, that's the cost. Mm-hmm. Right?
75
00:15:39.204 --> 00:15:51.224
Oh, I need to wait because Joe needs to finish something over there before I can start working on that something. That's cost. Mm-hmm. That is often overlooked because, hey, we actually measure the tangible things.
76
00:15:51.304 --> 00:15:58.744
We measure how much service costs and how much networking costs and what so not. We often forget that we are wasting people's time, right?
77
00:15:59.124 --> 00:16:09.844
So I want at the same time to make people more productive, but als- by, by giving them everything that they need, that there is no bottleneck in what we do,
78
00:16:10.784 --> 00:16:25.204
but at the same time use the absolute minimum amount of resources. And to accomplish that, the solution is simply to use ephemeral environments or ephemeral servers or ephemeral applications or whatever else.
79
00:16:25.304 --> 00:16:35.844
Everything ephemeral, right? Production is permanent. Mm-hmm. I have permanent clusters that run production, and the nodes in those clusters are going up and down, scaling down, scaling up. Mm-hmm.
80
00:16:36.064 --> 00:16:44.984
Everything else, "Hey, I'm going to start working on this." Excellent. I get the environment in a few seconds, maybe a minute. I do, do whatever I'm doing.
81
00:16:45.344 --> 00:16:57.824
The moment the, the, the system detects that I
haven't been working on it for whatever period of time, let's say five minutes, it goes down. Yep. I totally agree with that. Like, we are seeing...
82
00:16:58.324 --> 00:17:09.994
So we have, uh, I, I have seen two approaches. One is, like, the permanent dev and the permanent Q&A, like, um, shared environment when everyone gets there and all the stuff is deployed there.
83
00:17:10.554 --> 00:17:13.784
And then we are, uh, seeing another approach that is the ephemeral.
84
00:17:13.864 --> 00:17:25.784
Like, every developer has its own, uh, ephemeral, uh, cluster, or in this case, if it's a deployment or a workload, depends on if you do it on namespace basis, and then you deploy everything there.
85
00:17:26.184 --> 00:17:37.164
Uh, you test whatever you want, and then it will be deployed for, I don't know, whatever hours, uh, or whatever time. And then if there is no activity in there, it will be, um, shut down.
86
00:17:37.524 --> 00:17:51.404
And it's also, and I, I don't know if you, uh, test it out, but if you can connect it with the PRs, uh, stuff, so whenever, when everyone creates a pull request for testing a feature, you can just, uh, send it out to, to create the environment.
87
00:17:51.444 --> 00:18:01.184
And then whenever, when that merge, of course, is closed, but then beforehand it can be also closed for, for that. What do you think of that approach? Exactly. No, no, same, same thing with PRs.
88
00:18:01.464 --> 00:18:14.054
Uh, you know, as long as that is... And it doesn't really matter whether it's a PR or it's a dev environment or whatever it is. Mm-hmm. If it's not active, shut it down, right? Yeah.
89
00:18:14.054 --> 00:18:26.828
And we have solutions for that these days, right? Uh, you can, um-Let's say if we stick with Kubernetes, you can use Knative that will scale down your application to zero replicas. Mm-hmm.
90
00:18:27.688 --> 00:18:38.228
Uh, you can use vCluster to easily create cluster and destroy it. Uh- Mm-hmm... it's kind of a virtual cluster, uh, depending on some actions. There, there are plenty of solutions, right?
91
00:18:38.808 --> 00:18:55.028
And, uh, of course, the, it all requires work. Uh- Yeah... but it's not that much of a work either. The biggest bottleneck usually in getting there is that our applications need to change. And this is- Yeah...
92
00:18:55.038 --> 00:19:10.968
also the area where I feel that, uh, many FinOps, um, focused people are struggling, right? Mm-hmm. "Oh, I can do this, I can do that, but I cannot ask you to rewrite your application," for example, right?
93
00:19:11.088 --> 00:19:22.428
Oh, that's a no-go. Nobody allows, nobody allows that. Mm. And very often your application is a bottleneck, right? Maybe, for example, let's say that your application cannot scale to mul-multiple re-replicas.
94
00:19:22.488 --> 00:19:33.788
It is not fully stateless, uh- Yeah... whatnot, right? Uh, so if you cannot scale, we cannot optimize the usage of resources- Mm-hmm... because then we need to assume the worst-case scenario.
95
00:19:34.008 --> 00:19:46.028
The, the whatever is the maximum that we expect ever to, to happen, right? Mm-hmm. Yeah. That's... I, I, and, and I totally hear you on that point. Like, um, if you, if you are not able to...
96
00:19:46.068 --> 00:19:57.268
First, if you are not able to scale your application, you are, uh, bounded to the state, and if you are bounded to, uh, to the state, that's really, really, really problematic for, for the cost and especially for reliability as well.
97
00:19:57.328 --> 00:19:58.398
Like, the moment that the state- Exactly...
98
00:19:58.408 --> 00:20:12.688
has an issue, you are, you are- By the way, we are hearing a lot lately about, about what Viktor just mentioned, um, the, the thing that the application is being, uh, is not being looked at and th- that requires to do as well, where possible, right?
99
00:20:13.368 --> 00:20:24.918
Um- Mm-hmm... to start doing optimization at the, at the software level. And, you know, most of the time, let's say, uh, if I stick with the application example, right? Let's say that- Mm-hmm...
100
00:20:24.968 --> 00:20:39.648
we come to the conclusion we should change parts of the application, rewrite something- Mm... whatever. Yeah. Right? Uh, FinOps is not the reason to do that, right? FinOps is the best reason to justify it, right?
101
00:20:39.688 --> 00:20:50.358
Because the easiest way to justify anything is to say, "If we invest this amount of money, then we will get this amount of money back," right? Right. That's the easiest way to justify it. Yeah.
102
00:20:50.368 --> 00:20:58.868
But it's not really about FinOps. Uh, y- you don't want to scale because it's cheaper. Cheaper, being cheaper is just a side effect.
103
00:20:59.288 --> 00:21:10.738
You want to scale because you will have no downtime releases, you will have better performance and so on and so forth. You will get many other technical benefits, and also- Mm-hmm...
104
00:21:10.908 --> 00:21:20.208
flow benefits and human benefits, many, many benefits, and all those benefits will, at the same time, result in, in lower cost. Mm-hmm.
105
00:21:20.988 --> 00:21:31.408
And, and I want to, but I want to highlight something that you mentioned there, and maybe that's a clarification that it needs to be needed, is that the FinOps is not just to make the application cheaper, um, to reduce the cost.
106
00:21:31.838 --> 00:21:44.348
It's to... The whole idea is to maximize the value that you get from each of the dollars that you spend. So- Yes... that's why you can do, and you can do FinOp- you can justify it with FinOps. I mean, and you can...
107
00:21:44.408 --> 00:21:53.088
It's the best reason in terms of in this spe- specific term. Like, you can maximize the ROI that you get for your application, and that's the whole idea.
108
00:21:53.148 --> 00:22:12.828
Like, if you tell business, you tell your manager, the manager of your manager that this application will make, I
don't know, $50 more per user with the same amount of money, that's enough justification to do any change, uh, that you need if the cost that you said, the human cost and infrastructure cost, it will be worth.
109
00:22:13.008 --> 00:22:20.268
So if you are improving these, uh, KPIs and these, uh, ratios that we are seeing now, it's a very hot topic as well in FinOps.
110
00:22:21.108 --> 00:22:33.188
Once you are improving that metrics, uh, that are related to business, which is the important thing, uh, that's all the justification [laughs] you need for, for your manager, right? Oh, yeah. Exactly. That's great.
111
00:22:33.228 --> 00:22:33.478
That's great.
112
00:22:33.568 --> 00:22:43.928
No, no, it's, uh, because it's a very hot topic because sometimes, you know, DevOps and, and certain people will think like, "Yeah, it's only because to make it cheaper, but it will make it less re-reliable," and it's not to make it cheaper.
113
00:22:44.388 --> 00:22:58.688
It's to make it much more efficient in terms of business returns. Exactly. I mean, y- I think that we are, both of us are actually talk- uh, talking about the same thing- Mm-hmm... just not m-maybe using different words.
114
00:22:58.968 --> 00:23:10.708
It could be. [laughs] But yeah, it's, it's not about... I think that we both agree it's not about reducing cost, it's about being better, right? Like that. And at the same time, that reduces costs great, right?
115
00:23:10.728 --> 00:23:19.168
But yeah- Mm-hmm... it's about your users, uh, accomplishing the same res- outcome faster- Mm-hmm... that results in more income, right? Yep.
116
00:23:19.348 --> 00:23:32.188
Uh, hey, if Amazon can, can, can make me shop, uh, buy something in 15 seconds instead of five minutes, that's, that, that's a business value, right? Yeah. And for them to do that,
117
00:23:33.368 --> 00:23:41.148
their site needs to be blaz- lightning fast. It's kind of, it needs to happen like this. I'm not waiting for anything. I'm just shopping, shopping, shopping like crazy, right? Yeah.
118
00:23:41.158 --> 00:23:53.248
[laughs] Exact- And you accomplish that by your application simply being better in ma-many different ways. Uh, faster- Mm-hmm... whatever the, the, whatever the better means, right? Yeah.
119
00:23:53.308 --> 00:24:03.298
And regarding this topic, because I think it's very interesting to, to continue conversation, and now that we are recording this in a very, uh, you know, uh, shopping, uh- [laughs]...
120
00:24:03.428 --> 00:24:15.348
uh, amount of, uh, of the calendar, in a very da- uh, shopping date, um, in terms of especially load testing and scalability and, and costs, like how do... Because that's what you're saying, right?
121
00:24:15.668 --> 00:24:26.328
Uh, there is no good in Amazon that for reducing the cost, they will reduce the amount of orders that they will get on, uh- Yeah... Black Friday because they are selling like crazy, or any other, you know, uh, company.
122
00:24:26.408 --> 00:24:42.820
So how do you think this load testing and this curve of-Of course, end users should be considered on the, like, load testing, uh, dev ops side, uh, especially in the Q&A and testing when someone is testing, like, Black Friday workloads.
123
00:24:43.020 --> 00:24:57.240
Is- do you should consider, like, business metrics, KPI metrics also in this, in this Q&A, in this QA, uh, phase? Yes and no at the same time. Okay. I think that
124
00:24:58.400 --> 00:25:11.980
testing is important, of any kind of testing- Uh-huh... including load testing or performance testing. It's, it's definitely important, but the only testing that really, really, really matters is testing in production.
125
00:25:12.740 --> 00:25:21.300
Okay. That kind of... Until you reach production, you don't really know- Mm-hmm... because your tests are not... will never be able to simulate the real scale.
126
00:25:21.360 --> 00:25:28.760
Your tests will never be able to fully simulate the randomness of your users and so on and so forth, right?
127
00:25:29.260 --> 00:25:37.760
Um, I, I'm definitely in favor of test- I'm a testing freak, so I, I definitely think that everybody should be testing in any sort of way. Mm-hmm.
128
00:25:38.100 --> 00:25:53.380
But what is m- just as important or more important is testing in production. And when I say testing in production- Yeah... uh, I don't mean, "Hey, uh, let's run our unit tests in production or load test in production."
129
00:25:53.440 --> 00:26:08.760
What I really mean is that do you... Is your observability ready for that, right? Mm-hmm. And can you really see what's going on? Can you predict what will be happening based on production?
130
00:26:10.600 --> 00:26:25.560
And ultimately, can you automate the actions behind those predictions, right? Uh, what you really want to accomplish, of course, you should be testing what will happen d- during Black Friday before Black Friday, right?
131
00:26:25.720 --> 00:26:39.560
Yeah. But the ultimate goal is for your observability and your automation that uses those sa- that same observability to be able to react, right? Yeah. It's, it would be silly to say...
132
00:26:39.660 --> 00:26:53.730
Imagine if Amazon would say, "Hey, we predict that, uh, for Black Friday, we will need million servers," right? Mm-hmm. "Let's spin up million servers and have them running just in case." Yeah.
133
00:26:53.740 --> 00:26:55.929
That would make them bankrupt, right? Yeah.
134
00:26:55.940 --> 00:27:15.850
What they do have to do is that they tested, uh, tested things that are not necessarily related directly with Black Friday, kind of, okay, um, how many, what is the number, how many nodes can we bring up online if the traffic increases, and how much does it take to do that, right?
135
00:27:16.060 --> 00:27:30.780
Mm-hmm. Oh, we can, we can handle up to 100 new nodes with everything running, new replicas running in those nodes within, I dunno, five minutes' time, right? Yep. If that's what
you tested, you're ready, right?
136
00:27:30.900 --> 00:27:40.660
I mean- Yeah... th- that's not the only thing, just to be clear, but- No, no, no... kind of. Basically, you want to, you want to observe what's going on. Not you, sorry, not you.
137
00:27:41.040 --> 00:27:53.060
You want machines to observe what's going on and be able to act accordingly- Yeah... uh, on any situation that imaginable, right? I mean- Mm... it's never 100% like that. You can never- Mm...
138
00:27:53.110 --> 00:28:03.420
predict everything, but that's the goal, really. No, but that, that's a great point. You want to see that your environment can scale up and eventually, uh, uh, make it efficient
139
00:28:04.620 --> 00:28:12.780
and be, um, and make sure that the production is ready for at least as much as possible, uh, is ready for the, for the real thing, right?
140
00:28:13.000 --> 00:28:25.600
Because a- as you mentioned, uh, you cannot match it 100%, and even if you could, it would [laughs] it would make you bankrupt, even if you're Amazon. Uh- Mm... thi- this is interesting and, and interesting view.
141
00:28:27.660 --> 00:28:39.720
And it's, it's also a calculation, right? Uh, it's easy to make something that is the most performant, right? Yeah. And what I said before, let me spin up million nodes and stuff like that.
142
00:28:40.140 --> 00:28:53.700
It's also about the calculation. Okay, I want my users to have as good experience as possible, but not at any cost, right? Mm-hmm. So I w- I would make a calculation.
143
00:28:53.760 --> 00:29:08.480
Okay, so yeah, I'm going to start spinning up nodes, not in advance, but maybe after I see that the traffic increases, and that will maybe make a, some percentage of users have slightly worse experience.
144
00:29:08.500 --> 00:29:24.520
But that reduction of potential experience and the cost is maybe lower than, than, than the cost I would need to- Mm... pay for and so on and so forth, right? It's, the easy thing is to make it perfect.
145
00:29:24.600 --> 00:29:34.760
Not, not perfect, but kind of to make it, hey, everybody has great experience. Yeah. But I need to balance that cost versus return of investment, right? Yeah.
146
00:29:35.160 --> 00:29:42.400
You cannot either way to have 100% CPU consumption, but you cannot do it at 20%, of course. [laughs] Exactly. Exactly.
147
00:29:42.480 --> 00:29:54.340
So some, l- at least in Kubernetes, I feel that it of course depends on the size, but I would probably aim for maybe 80, 90% resource consumption, kind of- Yeah...
148
00:29:54.349 --> 00:30:05.520
that gives me a certain freedom without going crazy and saying, "Oh, 80% available for whatever happens." No, 10, 20% is the reasonable, uh, reasonable- Yeah...
149
00:30:05.600 --> 00:30:21.349
amount of available resources that you can say, okay, if things start going up, I have, I have just enough resources- Time... available to keep me going until new resources are brought, brought to the cluster, right?
150
00:30:21.540 --> 00:30:31.540
Yeah. And also very important that you should also consider the lower boundary for, uh, the down scale. [laughs] Oh, yeah. That, that, that's usually what people forget, kind of. [laughs] "Oh, we were very successful.
151
00:30:31.820 --> 00:30:41.236
We managed to survive Black Fri- Friday because we were scaling up." And then a week later kind of, "But did we scale down?" [laughs] That totally happens.
152
00:30:41.336 --> 00:30:49.756
Like, everyone, everyone uses the, the HPA to scale up, and nobody put the, the do- the cool down stuff. It's like y- there are two parts in scaling.
153
00:30:49.816 --> 00:31:04.895
One, it's, uh, scaling up, but then you also have to scale, uh, down your replicas because otherwise if you are spending the same ti- the same amount, uh, in Black Friday than, I don't know, a week after, your, your cost per user will be definitely a problem.
154
00:31:05.016 --> 00:31:14.526
Yeah. You know, Black Friday, s- Friday, big success. Survive. [laughs] Weeks after the Bl- Black Friday, we went bankrupt. [laughs] Yeah. That's a great thing.
155
00:31:14.636 --> 00:31:24.296
You know, we were talking with Guillermo the other day, and he woke up- Yeah... and we were talking about the same thing. Where is the limit or where is that line that you say, "Okay, this is enough." Okay?
156
00:31:24.336 --> 00:31:34.596
And then thi-this user experience is enough. We don't need to be perfect, as you said. We're just giving- Yeah... this enough for the user to, to continue to operate and the system to operate, yeah?
157
00:31:34.716 --> 00:31:47.616
And the user will be, be okay. Eventually, this is only at peak times. It's not all the time. Mm-hmm. Yeah. You know, that's the same thing, like, when we talk about availability in general. Mm-hmm.
158
00:31:48.256 --> 00:32:00.876
Most of the time we talk about how many nines you have, uh, 99 point- Yeah... and then how many nines. Nobody ever... No, no sane person or experienced person ever talks about 100. No. There is no
159
00:32:01.916 --> 00:32:16.296
advanced company that tries to make it 100, because simply there is a li- there, there is a point after which the cost is simply too high. Yeah. It's not worth it anymore. Yeah, yeah, for sure.
160
00:32:16.316 --> 00:32:23.696
And I mean, the, you only try to put another nine after the, the last nine, and that's it. [laughs] That's what you, what you try to do.
161
00:32:23.736 --> 00:32:48.576
So, and, and continuing with this, like, because we discussed a lot of topics and a lot of, uh, practices, for someone that is maybe, you know, having a, a cluster, you know, a, a standard cluster stuff, it's alwa- always running for Q&A, dev, and, um, production, uh, what do you think could be a practices to improve, uh, the resource of, uh, usage and the resource optimization in general?
162
00:32:48.645 --> 00:32:54.456
Something that you have seen in the industry or something that you would do for a, you
know, a standard use case.
163
00:32:54.716 --> 00:33:05.516
I'll, I'll give you the, the answer what I've seen and what I haven't seen, and I am surprised that I haven't. Um, what I've seen is, is more or less the topics we talked, right? Kind of- Mm-hmm...
164
00:33:05.796 --> 00:33:22.496
if mainly around ephemeral, uh, usage of- Okay... everything, and about right-sizing, uh, stuff, right? Mm-hmm. And scaling, of course, right? If you, if you combine those three things, uh, you are going to be good.
165
00:33:22.756 --> 00:33:36.016
Okay, right. Mm, yep. The, but the thing I'm not seeing, and I'm honestly surprised, uh, is usage of cloud providers that are not the big three, right? Okay.
166
00:33:36.166 --> 00:33:48.416
So we have AWS, Azure, Google Cloud, and I do think that for anybody but small companies, they are the right choices because there are many things that they have that other providers don't. Mm-hmm.
167
00:33:48.736 --> 00:33:58.536
Uh, like for example, almost no other provider has three availability zones in the same region, right? Yep. And without three availability zones, I cannot make it, um,
168
00:33:59.456 --> 00:34:12.866
I cannot make it highly available, and it's normal that they have it and smaller ones don't. But many of the reasons why we want to use AWS or Google Cloud or Azure are not the same, uh,
169
00:34:13.936 --> 00:34:31.405
for production as non-production, right? Um, people can, can really get amazing results in non-production by using, uh, Civo or DigitalOcean or Linode or Hetzner or any of those, right? Mm-hmm. Now,
170
00:34:32.316 --> 00:34:49.836
in the past, that, that was not a good option simply because the, the effort and the work that we need to put to make same thing running in- Yeah... very different places was too high. Because, hey, if I'm...
171
00:34:49.896 --> 00:35:00.636
Let's say that I'm running on nodes and what so not, running in AWS and in DigitalOcean is very, very different, right? The results are going to be different. The, the processes are going to be different.
172
00:35:00.676 --> 00:35:09.656
The automation is going to be different. Everything is different. But now we have a standard, and that standard is Kubernetes. Yep. Right? If I...
173
00:35:09.666 --> 00:35:19.176
I, now I know that Kubernetes in DigitalOcean and Kubernetes in AWS are not the same. I know that, but they're 99% the same, right? It's good enough,
174
00:35:21.236 --> 00:35:36.096
and I'm not sure why, at least I don't see combinations of those, kind of, hey, let's go, let's go over there for production, and let's go somewhere else, but it's much cheaper, and at the same time, also faster in many ways.
175
00:35:36.656 --> 00:35:47.426
Um- Mm-hmm... uh, if you spin up a node in Civo, you can spin up a cluster in literally seconds. Yeah. While in AWS, it, it can take, like, 20 minutes. Yes.
176
00:35:47.426 --> 00:35:56.756
And 20 minutes is not too much if I'm setting up production because I'm not destroying and creating new production every five minutes. I don't care, right, how much it takes. Right.
177
00:35:57.426 --> 00:36:03.276
But when I'm spinning up ephemeral environments, I, I do care how much it takes, 'cause I don't want people to wait.
178
00:36:04.516 --> 00:36:18.236
So they're cheaper and faster, not necessarily as reliable, but reliable enough for, you know, development or pull requests. Yep. I totally agree with the, with the ephemeral system. And do you think...
179
00:36:18.316 --> 00:36:29.116
So regarding that, do you think that the overhead, or when do you would start doing that because of the overhead that it would take for switching to a new Kubernetes provider?
180
00:36:29.176 --> 00:36:40.496
I, I know you said that it now is less, but there is still, um, a little bit of an overhead, especially with the connections. I would say that it could be more problematic to the new providers and stuff.
181
00:36:40.716 --> 00:36:42.056
Connections between applications or?
182
00:36:42.196 --> 00:36:59.498
On the, so the storing the connection credentials for the cluster, um, all the setup for, for the CI/CD maybe, um, that, that kind of switches that could change from one environment, let's say one provider to, to the otherI mean, there is always work involved, but- Mm-hmm...
183
00:36:59.528 --> 00:37:10.108
it's much, much smaller than before. Let's say- Yeah... I'm, I'll, I'll take one of the examples, right? You said, uh, keys to deploy stuff or something like that, right? Yeah. Yeah. We use Argo CD these days, right?
184
00:37:10.568 --> 00:37:17.368
Yeah. And I just need to have Argo CD pointing to the repo. That's it. Oh, makes sense. Yeah. Kind of like- Yeah. That's a good point... I have a cluster over there.
185
00:37:17.448 --> 00:37:28.558
I have Argo CD in that digital ocean, whatever it is, pointing to the repo. I just need to... I mean, yeah, it will take me time to set it up, but, uh- No, but, but you're right...
186
00:37:28.568 --> 00:37:33.088
an hour, kind of like not a big deal, right? No. No, it doesn't.
187
00:37:33.128 --> 00:37:44.648
Like, as you say, I didn't, I hadn't thought about the Argo CD for this, for this scenario, but I think it makes a lot of sense to switching for, for GitOps, let's say, um, to- 'Cause-... to do this stuff...
188
00:37:44.748 --> 00:38:00.928
you know, that's, uh, security is very often the, the thing that people overlook and probably the, one of the biggest reasons for me for adopting GitOps. Mm-hmm. Um, and is that it is a pull-based mechanism, right? Yeah.
189
00:38:00.938 --> 00:38:10.588
I, I don't want, I don't, I don't want my processes to, to go to a cluster and do something, because then I need to deal with keys that you mentioned before. Uh- Yeah...
190
00:38:10.608 --> 00:38:23.428
and you were probably mentioning keys from the perspective of, uh, the effort needed to, to do it, but there is also security part of the story. Pull-based mechanism simply
works so much better, right? Uh- Mm-hmm...
191
00:38:23.508 --> 00:38:34.028
I just need... I have my pipelines that are building images and pushing them to registry and doing whatever- Yeah... they're doing, and when the time comes, they push to Git, and that's it. Yep. Yeah, yeah, yeah.
192
00:38:34.068 --> 00:38:49.148
No, no, I totally agree that for this specific scenario, and now that people are doing more multi-cloud stuff and testing out new things, um, the GitOps approach can be so much efficient for, for, for these scenarios.
193
00:38:49.208 --> 00:38:56.748
Yeah, I didn't fall into that. I was more focusing like, because since we-- I've been doing like CI/CD all my life, I, I am now adopting the, the GitOps.
194
00:38:57.008 --> 00:39:05.848
Uh, recently I was always like, "Yeah, these secrets are driving me crazy." Where [laughs]... where I have to put it. I have to set it up to a new provider. The new connection is like, "Man, what's going on?"
195
00:39:06.428 --> 00:39:14.108
[laughs] Yeah, secrets are driving everybody crazy. [laughs] Yeah. You're not the one, the only one. Yeah. But even secrets, right? Uh- Mm-hmm... so, you know,
196
00:39:15.148 --> 00:39:28.468
there are so many tiny advancements that, you know, when you combine them, they make such a huge difference. Like, secrets was a pain before. Mm-hmm. But now we have, for example, external secrets operator. Yeah.
197
00:39:28.968 --> 00:39:40.818
That you just need to deploy it in a cluster, and it will be fetching secrets from wherever secrets are stored, and doesn't matter whether that, uh, ESO, uh, operator is, is in this cluster or that cluster.
198
00:39:41.728 --> 00:39:54.028
You just need to install it. Takes few minutes. Cool. You need to give him credentials to go to your secret store- Yeah... whatever you're using, and you're done, right? No, and- And secrets will automatically appear.
199
00:39:54.168 --> 00:39:59.168
You don't need to worry about them. They will just appear out of thin air. Yeah.
200
00:39:59.288 --> 00:40:11.628
And also with SOPS and KSOPS and all this stuff that you can also, like, uh, encrypt all the, the secrets within your code because it's already encrypted, and then you can decrypt it on the fly when you deploy.
201
00:40:11.828 --> 00:40:20.897
I think we, we have found that solution as well to be one of the most effective, and nobody is, like, fighting again, like, where this is stored, uh, where the, this addresses.
202
00:40:20.968 --> 00:40:33.428
You just, uh, encrypt it, leave it up there, and then the, the pipeline or Argo CD will do the, will do the work for you. [laughs] Exactly. Yeah. There, there is that on us. This- Mm-hmm? Just not to be misunderstood.
203
00:40:33.578 --> 00:40:37.818
Yeah. I might sound like I'm saying that it, it's no work involved. No, no, no. There is always work involved.
204
00:40:37.908 --> 00:40:49.498
I'm just saying that we got so many improvements, especially on standardization level, that, yeah, there is work, but it's infinitely smaller than 20 years ago. Like, I'm an old dude. I kind of like- [laughs]...
205
00:40:49.508 --> 00:40:55.558
you know, I, I lived in an era, I, I don't know whether maybe Damian lived in that era- [laughs]...
206
00:40:55.558 --> 00:41:10.548
when we would go over Friday night and, uh, sleep in the office for two days because it took whole weekend just to deploy a new release. Yeah. You know, like, uh, we're past that. [laughs] Yeah.
207
00:41:10.608 --> 00:41:22.738
Now the, the times to, to deploy are much, much less, and also the, you know, the, the watch times, like, the availability is way easier to, to achieve, definitely. Oh, yeah.
208
00:41:22.738 --> 00:41:30.648
One, one more thing that I wanted to ask you, Viktor, is, um, from, from what are you looking to every day? I mean, do you work with the FinOps?
209
00:41:30.788 --> 00:41:48.768
Do you work on dashboard, any visibility, any things that you're looking into? Or, or do you have your own f- your own way of looking into usability and cost? So, um, I'm not,
210
00:41:50.588 --> 00:42:06.908
I'm not much involved in, in whatever has the cost as a bird. Mm-hmm. That's not my focus. Mm-hmm. Uh, but I am, I'm very much involved in, in things that, uh, are about optimizations, right? Mm-hmm. Um, which I feel
211
00:42:08.528 --> 00:42:18.028
end up having a lot of overlap, right? Yeah. I am very much involved into scaling, uh, observability and so on and so forth, and
212
00:42:19.188 --> 00:42:27.568
I might be completely wrong because, as said, FinOps is not my area, but I feel that those are very, very overlapping. No, for sure.
213
00:42:27.608 --> 00:42:41.038
We, we can tell you that, that, that those are FinOps related, and that's hugely impacting the, the FinOps practices because, as I told you, Kubernetes is one of the, the hottest topics, and scaling down and scaling up, optimizing workloads- Right...
214
00:42:41.188 --> 00:42:54.328
um, it's, it's a core thing to, to- Oh, way-... this stuff... you know, whatever we were scaling until recently is nothing to the scaling that is coming now with people, companies training models. Now we are- Yeah...
215
00:42:54.368 --> 00:43:02.726
jumping from hundreds to thousands or tens of thousandsOf, of nodes and processes and what so not And EBOs.
216
00:43:02.726 --> 00:43:16.576
Now it's going to be go crazy Because now we are, we are going to be in a whole new metric, of course, which is the EBO consumption that often is forgotten, especially on Kubernetes, like you always focus on CPU memory stuff.
217
00:43:17.196 --> 00:43:35.656
But now it's like distributed training, um, to a huge scale and the amount of time and resources, because I've been, you know, I, I was working on AI on 2018, which nowhere close to what we are reaching right now, and there was still a lot of consumption.
218
00:43:36.076 --> 00:43:41.096
I cannot imagine the amount of money that could be invested now in...
219
00:43:41.356 --> 00:43:53.476
Well, I said millions in a train, like in a train round for a model is like millions of dollars, so the, what we are going to be now is going to be crazy, especially if Kubernetes starts with the GPO, which is already there.
220
00:43:53.576 --> 00:44:05.016
[laughs] Now, you know, all those people that were annoyed, that were not working in AI and were annoyed by you saying, "Hey, you should optimize this. You should do this," now they're fine.
221
00:44:05.076 --> 00:44:16.126
Now nobody's going to ask them anything because now their expenses are gonna be nothing compared to- [laughs]... to expenses of the data scientists. Now, uh, now you're fine. You don't have to do anything.
222
00:44:16.196 --> 00:44:29.236
Nobody cares about you anymore. Now we need to go after those guys. Yeah. And now, now we are seeing it at this, at Rendow as well, like there is a, a duality in FinOps now that is AI for FinOps or FinOps for AI.
223
00:44:29.596 --> 00:44:36.356
Like what do you do? Uh, do you start using AI to, to your FinOps tools or to your FinOps practices? Or
224
00:44:37.336 --> 00:44:56.476
you, you start doing your w- or the other part that is you do FinOps for your, your, your AI, you're trying to improve your prompts, you're trying to improve your training, you're trying to improve your RAG systems, how much, uh, each user costs in terms of the usage of the AI, and that's another whole level that is going to be there because AI is going to be
225
00:44:57.676 --> 00:45:09.476
a huge part of the IT budget in the next few years. Yes. Oh, yeah. Well, there's both of them, right? [laughs] Yeah, we will see both of them. FinOps for AI and AI for FinOps. For FinOps.
226
00:45:09.495 --> 00:45:21.156
And let's see, because now people are starting to do agents, which it seems to be working in the US and, you know, everything that works in the US that's going to be like all in our YouTube feed every now and then, so- [laughs]...
227
00:45:21.176 --> 00:45:22.196
it's going to happen.
228
00:45:23.076 --> 00:45:40.956
Uh, but yeah, um, that's going to be, that's going to be a whole new level to, to optimization and, you know, also reliability because, you know, to keep that training running, um, is not easy to have that amount of information running and available and training and not having any incident while training.
229
00:45:41.636 --> 00:45:53.736
Oh, yeah. Uh, it's going to be a nightmare. I mean, it already is. [laughs] Yeah, indeed, especially- Which is good. We all get to keep our jobs. Great. Yeah. [laughs] Um, that's great, man.
230
00:45:53.956 --> 00:46:02.976
No, I, I think it's a, it's a great point to, uh, to close the tab. So, um, thanks a lot for, for having today's call. It's been a pleasure.
231
00:46:03.356 --> 00:46:12.816
And for those that, uh, don't know you, where you can say where, where they can find you and, you know, what stuff do you do online because it's, it's someone to check on, uh, if you haven't. Definitely.
232
00:46:12.836 --> 00:46:19.336
If you're not following Victor- I do many things... start doing it now. [laughs] Yeah. I, I do many things. I have a YouTube channel.
233
00:46:20.416 --> 00:46:31.916
I, I, uh, uh, there is a podcast that a colleague, ex-colleague of mine is doing where I'm always, I speak in conference. And I... Just Google my name. It's- Yeah, definitely... Google my name. You'll find me, I promise.
234
00:46:31.926 --> 00:46:38.916
[laughs] Yeah. I, I can tell you you could find him. Um, so yeah, Victor, thanks, thanks a lot for, for being here today.
235
00:46:38.956 --> 00:46:49.876
It was a pleasure talking to you from, from the DevOps perspective, and also from, from FinOps to DevOps with Damian, and from DevOps to DevOps. I think it was an amazing talk, man. [laughs] So thanks for coming by.
236
00:46:50.216 --> 00:46:54.656
Thank you for having me. Thank you very much, Victor. Okay. And see you next time