WEBVTT
1
00:00:00.160 --> 00:00:03.780
Welcome, everyone, to another episode of [pop] the FinOps Weekly Podcast.
2
00:00:03.960 --> 00:00:22.259
And today, we are going to talk about AI and how it differs from FinOps', uh, usual work and how to control, uh, their cost, techniques that you can use, how to feed the models correctly to your workloads, and, you know, how to train your people here, uh, into AI cost management.
3
00:00:22.820 --> 00:00:30.300
And for helping me with that, we have Rafaela. Rafaela, how are you? I'm doing well. How are you, Victor? I'm doing great. I'm doing great. So we met in, in FinOps X.
4
00:00:30.320 --> 00:00:40.660
It was wonderful to finally meet you in, in person, and you know, definitely wanted to, to chat with you, uh, on FinOps here in the, in the podcast. So thanks for, for taking the time today.
5
00:00:40.960 --> 00:00:45.980
And, you know, wanted to, to talk about AI. So let's, let's go, uh, into it.
6
00:00:46.000 --> 00:00:57.940
So I've thinking and I've talked with a lot of people about the difference between the usual, uh, issue with, you know, cloud cost that FinOps, uh, try to help, and now all the issues that token...
7
00:00:58.080 --> 00:01:06.120
that, uh, you know, the tokenomics practice, the AI cost practice has. So can you tell us a bit more about the difference? Because AI is impacting now
8
00:01:07.040 --> 00:01:16.180
everyone in the company, and, you know, cloud is only used by, uh, the IT department. So can you tell us a bit more about the, those differences that you're finding? So traditionally, I...
9
00:01:16.420 --> 00:01:20.160
So my, in my current role, I am focusing on cloud cost only.
10
00:01:20.360 --> 00:01:32.460
So traditionally speaking, for FinOps practice within cloud, we only had to educate the engineers and the leadership on how to optimize costs and control costs, right? But nowadays in AI,
11
00:01:33.440 --> 00:01:44.280
everyone's using it in the company, everyone's encouraged to use it. So it w- it's not just technical people that we are talking about, but the HR, the marketing team, or admin people that everyone's using.
12
00:01:44.440 --> 00:01:56.720
So there's that gap of speaking their language, uh, talking to, to non-technical people as well as technical... usual technical users. Yeah. No, that's true.
13
00:01:56.780 --> 00:02:10.139
That's one of the things that we got from, uh, FinOps X that, you know, a lot of, like... I think the numbers were, like, 80 to 90% of the applications developed by AI are out of the IT, uh, uh, department.
14
00:02:10.199 --> 00:02:17.680
So that means, like, you know, uh, 9X, uh, the workload and, and definitely it's a problem that now is full company.
15
00:02:17.740 --> 00:02:32.040
So, you know, what, what are the challenges that you are finding and, and how you think, you know, the FinOps practice can help govern those, those costs now, now generated by, by AI? I think right now when...
16
00:02:32.150 --> 00:02:40.960
Let's go back to the term tokenomics. Hopefully, that term- Yeah... [laughs] remains relevant by the time this podcast comes out, 'cause who knows what term we're gonna be using next. But- Yeah...
17
00:02:41.040 --> 00:02:50.600
um, if I'm talking to somebody who's non-technical, even technical people nowadays, when I'm talking about, "Hey, you're using X number of tokens," that really doesn't mean anything, right?
18
00:02:50.700 --> 00:03:04.140
So we have to kind of transl- again, translate into language that people can understand in the context of their own work. So let's say you're cr- creating a customer support ticket from the s- support team.
19
00:03:04.700 --> 00:03:14.740
We have to say, "This... It, it costs X amount of money for X amount of tokens that you're using," instead of saying, "It's costing 5,000 tokens." What does that mean, right?
20
00:03:14.840 --> 00:03:22.120
So it's a- having the c- giving the right context to the metrics that we'll be using for AI.
21
00:03:22.900 --> 00:03:33.720
And the other issue is that there are all these related costs that, uh, to AI that, uh, that's not just strictly gen AI costs that you might be seeing as a line item in your billing.
22
00:03:34.580 --> 00:03:47.190
Um, that's hard to track because of tagging issues. But that's... I think tagging issues is something that we all have been struggling with since the beginning of FinOps', uh, practice- Yeah... has been, uh, flow, right?
23
00:03:47.280 --> 00:03:54.400
It's not something that's new. But there are many hidden cost items that people are not m-might, might not be familiar with right now.
24
00:03:54.450 --> 00:04:06.760
And I'm still, you know, figuring out myself within my company and, you know, on my own as well. So there are a lot of uncertainties that people face with, uh, that comes with AI cost. Yeah. No, that's, that's true.
25
00:04:06.820 --> 00:04:13.080
There is... There are also, I think, the, the undeterministic things, uh, related to AI.
26
00:04:13.160 --> 00:04:21.320
So, you know, you never know the output that it's going to be, how much tokens are going to be consumed in certain things unless you, you know, you constrain it a lot.
27
00:04:22.120 --> 00:04:33.220
Um, so there is a lot of variables that are, let's say, more out of our control. Rather than in cloud, you can be more predictable, let's say, unless you have spikes and everything. Um, but it's less predictable.
28
00:04:33.280 --> 00:04:40.780
And, you know, one, one of the major things, uh, that I'm seeing is the governance issue, um, because now everyone has access.
29
00:04:40.940 --> 00:04:51.840
So now, before in, in FinOps, let's say, so we had the issue in governance because only a certain set of people will use it, um, and we need to govern that, and that was a problem already.
30
00:04:51.860 --> 00:05:04.100
But if you have a full, uh, company-wide technology enablement like A- AI, now the cost governance is a whole company government, uh, like a whole com- company government issue.
31
00:05:04.160 --> 00:05:16.200
So, you know, why do you think this cannot stay only in FinOps, and how do you, can control all the cost governance across, across the company? I think it goes into...
32
00:05:16.640 --> 00:05:27.220
It goes back to the knowledge of technology that people might be having, right? Again, there is a different levels of technical knowledge that w- all the AI users will be having. And
33
00:05:28.200 --> 00:05:43.460
again, without guardrails, as you mentioned, it can go quite [laughs] devastating. It can have a quite devastating effect on the cost because people might be uploading, let's say, like, 10,000-page PDF, uh, to, uh,
34
00:05:44.360 --> 00:05:48.360
gen AI that they're not supposed to, 'cause that's going to drive a crazy amount of cost.
35
00:05:48.400 --> 00:05:57.460
So I think that having some kind of guardrails and training for, to people to know how to use AI correctly, and also from security perspective as well.
36
00:05:57.500 --> 00:06:07.644
It's not just a cost issue that, you know, people should be knowing what to put into AI, what not to put into AI, and what kind of-... questions to ask and how to ask those questions, that's very important.
37
00:06:08.444 --> 00:06:20.104
And right now, I think, uh, and I'm not saying this about in, in my particular company or, like, anyone in general, but I know that some people are, you know, prone to using shadow AI as well.
38
00:06:20.264 --> 00:06:31.704
That's not [laughs] approved, explicitly approved in their company, um, but they still upload it, uh, upload whatever information they want to into their own accounts in AI. Yeah.
39
00:06:31.744 --> 00:06:43.004
So I think that's the, um, one of the many it- risks that people are facing. Yeah. And, you know, but that's also, like, a security risk and, like, a lia- liability risk.
40
00:06:43.084 --> 00:06:54.644
We already have seen, uh, that, uh, popping up in several ways. We just have a, a recent, like, issue with, uh, OpenAI and Hugging Face weird thing- Yeah... that happened.
41
00:06:54.664 --> 00:07:07.234
[laughs] The AI, yeah, the AI acting upon itself, um, which is crazy. [laughs] Yeah. So that's also, like, another... I, I was discussing that before our call with another friend. It's like the,
42
00:07:08.444 --> 00:07:20.144
sometimes the value is not in the, in the value generation, but in the risk prevention. So, like, you, for example, let's say a fraud, uh, detection system.
43
00:07:21.004 --> 00:07:36.964
Fraud detection system is making you not to lose money, so it's making you, let's say, uh, you know, value, let's say value, value loss avoidance or however you wanna call it, because you are going to have, you may be a, you may have certain set of frauds or certain set of security attacks and everything.
44
00:07:37.304 --> 00:07:49.544
If you allow all of them, let's say you lose whatever, but if you are able to, to generate a system that prevents that, so you avoid having these security issues, you are adding value to your company, like in,
45
00:07:51.124 --> 00:08:04.044
you know, value, let's say, instead of, uh, value creation is not, is avoiding value loss, let's say. Uh, so that's also an interesting thing that we need to, uh, consider as well because we need to be...
46
00:08:04.224 --> 00:08:15.924
No, we are not always generating. We always, sometimes we need to, you know, uh, prevent losing value. [laughs] Yeah. I think it goes back to the concept of, like, ROI and unit metrics- Mm-hmm...
47
00:08:16.304 --> 00:08:26.804
um, unit costs, because you wanna look at how much AI tokens or whatever consumption, AI cost that you're consuming versus how much value you're creating out of it, right?
48
00:08:27.124 --> 00:08:28.824
The, it goes back to the whole concept of FinOps.
49
00:08:28.844 --> 00:08:39.364
It's not about saving the most amount of money, but how much value you're getting out of per dollar that you're spending, or per euro [laughs] per eu- for your case, or per token that we'll be using, right? Mm-hmm.
50
00:08:39.434 --> 00:08:41.264
So it's not, um...
51
00:08:41.404 --> 00:08:59.374
I remember, like, I was talking to a friend about their AI usage tracking within their company, and they have, like, a leaderboard, uh, showing which develop we're using, was using the most amount of tokens or u- using the AI the most, but they weren't really tracking what it was being used for- Yeah...
52
00:08:59.374 --> 00:09:03.024
which makes absolutely no sense. But right now, I think I've, I think a lot of companies are doing it.
53
00:09:03.044 --> 00:09:14.534
They wanna see how much AI usage is being done within the company, how many people are using it to do whatever they want, whatever tasks they wanna get completed. But we have to go back to value creation, right?
54
00:09:14.584 --> 00:09:23.714
Like, what kind of, like, h- what exactly are you doing with the AI that's saving us money, that's, uh, preventing certain things from happening, right? So that's the, um,
55
00:09:24.804 --> 00:09:37.284
metrics that we need to be looking at, rather than just looking at, uh, how much time we're cutting off, how much, uh, dollar we're saving from, by using AI instead of having another person do it. Yeah. No, that's true.
56
00:09:37.384 --> 00:09:51.284
I think that'll, that'll be solved once a significant bill or a bill shock, uh, happens that puts everyone in, in the right place. Yeah. Like, the bill doesn't lie. [laughs] So, uh, it's, it's going to be fun.
57
00:09:51.324 --> 00:10:01.024
And, and one of the things with the, with the bills and, you know, uh, that we suffer also in cloud was the, the volatility of prices, or like, let's say, the less predictability of the pricing.
58
00:10:01.064 --> 00:10:11.494
You know, we, coming from the, the data center or the on-prem perspective is, like, you buy this stack of or rack of servers that cost- Mm-hmm... that amount of money, and you amortize over time.
59
00:10:12.224 --> 00:10:22.443
Then cloud was more or less predictable because you are consuming and you are creating and, and reducing workload. But here is, like, everyone is spending, like, every time.
60
00:10:22.544 --> 00:10:38.624
Uh, like, uh, any command or anything, uh, introducing to cloud or whatever AI provider is a cost, uh, decision. So how are you seeing, like, the volatility of the, of the AI, you know, cost and, and ecosystem? Yeah.
61
00:10:38.904 --> 00:10:49.594
The g- I think it's more akin to serverless. Um, unless you have certain guardrails in place, serverless can drive up the cost quite significantly, right? Mm-hmm.
62
00:10:49.604 --> 00:11:00.824
And that's the, that's, you know, one of the things that people don't expect so much 'cause they think, "Oh, serverless option is going to be cheaper," which is not true unless you have, you know, everything configured correctly.
63
00:11:00.904 --> 00:11:05.904
So ins- uh, unlike, you know, as you mentioned, unless un- um, unlike traditional cloud
64
00:11:06.884 --> 00:11:18.404
infrastructure, where you can have more predictable usage in an hour, um, and AI is more driven by user interaction styles, the autonomous agent loops, you know?
65
00:11:18.444 --> 00:11:20.904
So that's the kind of guardrails that you can have in place.
66
00:11:21.204 --> 00:11:34.084
How many, uh, connect, API connections that can be made, for example, uh, how many loops you can try before it breaks, um, and also controlling the payload size. So, you know- Mm-hmm... controlling the context.
67
00:11:34.124 --> 00:11:47.924
And, um, having those guardrails in place and also training the users how to use it properly ahead of time would prevent that kind of volatility from, um, going out of hand. Yeah. No, that's true.
68
00:11:47.984 --> 00:11:54.144
Like, uh, everything reduces to, to guardrails and setting, you know, policies in place that, that makes everything controlled.
69
00:11:54.264 --> 00:12:02.484
And so you mentioned, like, uh, several topics that are related to AI, like techniques about, you know, the prompt, uh, the context, and the, the guardrails.
70
00:12:02.944 --> 00:12:13.224
On your side, like, what's your- I'm going to say your favorite to, to tweak, but also the ones that is providing you the most results. What are you, you know, looking, uh, for more?
71
00:12:13.284 --> 00:12:19.484
Like, it's the context or it's the prompts or it's creating guardrails. What's the one that you tend to work the most?
72
00:12:20.504 --> 00:12:30.784
So I used to laugh at people, [chuckles] like a few years ago when people used to use the term prompt engineering, because for me personally, I thought that was, uh, kind of like overkill.
73
00:12:30.844 --> 00:12:41.564
I thought people knew how to do it. But I've seen some prompts created by other people and realized that not everyone knows how to use it properly, right? So- Mm-hmm... again, the whole...
74
00:12:42.224 --> 00:12:52.224
It, it depends on the how the AI cost is triggered. Um, some companies, they have it triggered by license base, and for now it's kind of unlimited usage.
75
00:12:52.724 --> 00:13:08.064
But eventually what I think what all the AI companies are going to do is going to translate- uh, transfer it into, um, token usage or more limited usage based on, um, how many prompts or how many contexts or whatever they can have, right?
76
00:13:08.524 --> 00:13:16.324
So I think it needs to start from prompt engineering. You have to re- really know like how to ask the questions and in a more concise
77
00:13:17.304 --> 00:13:31.504
manner so that way y- you're giving it enough information for it to process, but also leaving it enough room to, for it to think on its own, rather than kind of feeding you the answer that you wanna hear from it.
78
00:13:32.204 --> 00:13:47.804
You have to be able to ask the right amount of... give it the right amount of context, right amount of words or prompt, and have the flexibility built into your, um, answers. Does that make sense? [chuckles] Yeah.
79
00:13:47.844 --> 00:13:59.264
No, no. I think- Yeah... it's finding that right balance between instructions and, and flexibility and being able to- Right... provide. Because if it's, you know, depends also on the answer that you want.
80
00:13:59.324 --> 00:14:06.224
If you want it very straightforward and, you know, same format repeatable, that's one thing and maybe AI is...
81
00:14:06.504 --> 00:14:15.574
I would say maybe AI, you need a lot of pre-processing before or post-processing after AI and to give it to the, like the right structure and everything. Right.
82
00:14:15.664 --> 00:14:22.714
But also if you want a more free, let's say documents, images, et cetera, you want a more creative thing, then that- Right... hmm,
83
00:14:23.664 --> 00:14:33.004
really, uh, needs a more flexible thing because otherwise it's going to be like the same answer. And you ask the another question, it's like the same answer. And you're like... For example, that happens a lot.
84
00:14:33.404 --> 00:14:44.064
I'm not going to, uh, cite, but if you, if you put like Gemini, I think it happened to me like a lot with Gemini Pro, uh, l- before one of the, one of the models just would like...
85
00:14:44.844 --> 00:14:50.564
It was so structured, like if you give it super structured answers, it will give you like the same answer over and over and over again.
86
00:14:50.604 --> 00:14:58.764
It's like at a given point that the context window, uh, I think it was like, you know, super compressed and that's also another point. [bell dings] Like the,
87
00:14:59.744 --> 00:15:07.724
the context and, and the, the window itself, like if you have long chats where, you know, uh, the model starts to compress the...
88
00:15:08.244 --> 00:15:15.964
now they compress the, the context, you end up with the latest ones, like the latest messages and everything, and that causes a, a case.
89
00:15:16.044 --> 00:15:23.624
I don't know if you've seen like these issues with the, with the context window and, you know, these long- Right... uh, situations. So yeah, it's...
90
00:15:24.024 --> 00:15:30.884
You don't want the conversations to go too long because then it'll be running everything each time you ask a new prompt, right?
91
00:15:30.944 --> 00:15:39.644
So you want a set of instructions which is, I guess, it, it can work as both as guardrail and also just kind of general guideline, right?
92
00:15:39.904 --> 00:15:52.324
Um, for example, like the instruction that I'll be using as a FinOps person, I would say like act as a cloud FinOps expert, and then that's how it would, um, construct its answers all the time based on my instructions.
93
00:15:52.724 --> 00:15:57.944
And always also you can put up like more security, um, related instructions based on that.
94
00:15:58.044 --> 00:16:12.504
Say, "Do not access this X, Y, and Z," and then AI would not be, um, running that prompt every time, but it would just know, um, consistently, so you don't have to y- type it each time that you ask a question.
95
00:16:13.284 --> 00:16:28.024
And by doing s- by setting instructions, because it's a repeated, uh, process that you h- always have to ask, you're saving the context, uh, um, you're, you're, you know, condensing the context window and, you know, saving tokens by having less words to repeat each time.
96
00:16:28.744 --> 00:16:38.224
And also like the other thing that I've been do- using, um, lately that I really enjoy is using skills. Uh, there are a lot of skills that people are, have been cr- creating. You know, you mentioned Gemini.
97
00:16:38.264 --> 00:16:47.584
The Gemini has their own, um, repository of skills. There are task specific playbooks that you can be using, um, but instead of have...
98
00:16:48.264 --> 00:17:01.584
But unlike instructions, you don't have to load it yourself, but you kind of trigger AI to pull from it, so that way it's not always sitting on your instructions, which can again blow it up and kind of limit your answer.
99
00:17:02.364 --> 00:17:15.624
Um, but it can always refer to it, um, acting as an- another like a expert source that you can pull from each time that you are asking the relevant questions to AI. Yeah.
100
00:17:15.674 --> 00:17:26.724
And that's also something that people need to be careful about is like depending on... I think it depends on the model and the system that you're using. Uh, the skills and the MCPs can generate a lot of
101
00:17:27.644 --> 00:17:40.044
context like or overhead to the model when, when it's trying to answer, and especially I think it's MCPs because there is a lot of, you know, words and things that could trigger and it could think like, "Yeah, I think that'll be..."
102
00:17:40.104 --> 00:17:51.744
And it can, uh, you know, uh, consume a lot of the context and the, the reasoning of the model. So you need to be careful about like, uh, that also relates to, you know, the context and the specialization.
103
00:17:51.764 --> 00:18:05.476
It's like you need to put it in constrained things and in guardrails that, okay, you have this set of skills, you do this, and you are specialized in that and you don't have like, I don't know, 100 skills enabled and your context is-...
104
00:18:05.496 --> 00:18:18.716
all skills and you cannot introduce anything because it's not going to happen. It's, you know, we try to make specialized people, uh, and in this case, specialized agents, uh, for, for making it, uh, work. And
105
00:18:19.896 --> 00:18:23.476
other things that, that we've seen in AI is the, the caching system.
106
00:18:23.576 --> 00:18:35.816
So caching is very, uh, interesting in, you know, in databases, like usual, uh, things, but in, in prompts it's kind of the same idea but different because it's less predictable.
107
00:18:36.306 --> 00:18:46.896
How are you seeing the, the prompt caching, uh, working for you, or how are you seeing, or how you are you taking advantage of, of the prompt caching? Yeah. That's, um...
108
00:18:48.516 --> 00:18:54.696
So I think right now that's something that we are working on as a company- Mm... to figure out for es- especially for each user.
109
00:18:54.836 --> 00:19:06.376
So, you know, as I mentioned, going back to the beginning of the recording that we were saying that it's now expanded to the entire company, so it's not just technical people that I'm speaking to. So it's kind of, I...
110
00:19:06.716 --> 00:19:28.976
right now, like in the process of gathering information on what's most, like, commonly asked information or what's something that it should be, um, maintaining, um, so we can cache the information rather than, um, uh, cache the, like, static repeated, uh, information from the prompt rather than having to send same information over and over again.
111
00:19:28.996 --> 00:19:41.236
And that's something, um, that, you know, each team or, like, you know, each user has to figure out, and we... Like, right now my process is to provide some kind of guardrails, 'cause a lot of people don't even...
112
00:19:41.276 --> 00:19:50.796
Like, they're not even aware of this whole idea of cach- caching and, you know, prompt caching or context caching. And there are different options to do it, um, but
113
00:19:51.756 --> 00:20:01.156
yeah, right now we're still [laughs] in the process of figuring it out. Yeah. It's not, it's not... I think it's not as easy as, uh, in the database world, let's say. Uh, even though it's...
114
00:20:01.276 --> 00:20:12.136
I always think of it like a double-edged sword, uh, because you're gonna have a lot of advantages if done right, but if done wrong, uh, caching can be a, a pain, uh, to, to solve.
115
00:20:12.736 --> 00:20:26.996
And, and other thing is y- that, that I've... Just in the, I think it was in the past podcast, was the selecting the right model, uh, like fitting the request to the right model. How are you seeing, like, people...
116
00:20:27.066 --> 00:20:39.816
Because I don't know, maybe people are using Opus 4.8 to request, uh, what's the hour or what time is it. Um, so how are you seeing that? What are your advice on, you know, on the right sizing of models?
117
00:20:39.956 --> 00:20:49.596
Let's call it that way. Yes. So yeah, right now the approach that I'm pushing for is a T-shirt size approach, um, that you've probably heard from, you know, FinOps practice.
118
00:20:49.676 --> 00:20:59.676
So what it is, is that we have, like, small, medium, large size, depending on the usage. So small is probably, like, you know, when you're thinking about lightweight usage, let's say composing an email,
119
00:21:00.576 --> 00:21:05.196
uh, or creating, um, meeting summary, et cetera.
120
00:21:05.316 --> 00:21:16.276
That should be using the lightest model possible because, again, like, different models have different token con- consumption, and you don't wanna overblow, like, Opus, for example, using, using it for your email consumption.
121
00:21:16.336 --> 00:21:27.136
I don't think it... Like, for email usage, I personally don't think you should be using agentic AI [laughs] to begin with. Uh, but some people do it. So that's the... where T-shirt size approach comes in.
122
00:21:27.256 --> 00:21:36.156
So you have small for lightweight usage and large or, like, medium or large for more heavyweight usage, meaning something more advanced, more technical.
123
00:21:36.316 --> 00:21:46.796
So let's say for developing a certain, like, non-proda- environment or coding something new that's more, a lot more complicated or, like, a lot more mathematical, et cetera.
124
00:21:46.916 --> 00:21:52.716
So, um, people would not have access, the same access to different types of models. It would have...
125
00:21:52.916 --> 00:22:01.196
It, it would depend on the department, def- de- depend on the apps they're working on, depending on the projects, et cetera. That's, that's true.
126
00:22:01.296 --> 00:22:13.876
We need to avoid, like, you know, setting default Opus or setting default Sonnet, and then every, uh, for everything we use Sonnet instead of, you know, right size and, and what's the task, and based on the task we can, we can set it right.
127
00:22:13.936 --> 00:22:24.395
So we need to be a little bit more conscious about it. Especially when it's your own money, you, you see it more and you are more careful about it, but when it's company money, it's, it's different. Um,
128
00:22:25.456 --> 00:22:29.856
and wanted to, to also dive into the training because, um,
129
00:22:30.796 --> 00:22:43.836
on FinOps there was a lot of effort into the, you know, educating the other departments, educating engineers, finance, and everyone involved to, uh, to be aware of costs, uh, in the, in the cloud side.
130
00:22:44.496 --> 00:23:03.636
So, and before, like, before in the, I think, 2025, mostly all the big companies were about, you know, AI, um, you know, um, scaling and AI training for the company, but less, like, a skill and less about AI consumption, uh, culture.
131
00:23:04.076 --> 00:23:15.396
So how are you seeing the differences and, you know, uh, now that it involves more non-technical crowd that in, in FinOps where it was targeting- Yeah... IT? Yeah. I think, like, you know,
132
00:23:16.496 --> 00:23:27.976
like, I think as late as, like, last year, it was a lot more free usage encouraged, uh, meaning- Yeah... we didn't have the concept of tokenomics. [laughs] So it was kind of run whatever you want in AI prompt.
133
00:23:28.016 --> 00:23:41.446
But right now it's, you know, a lot of companies, uh, AI providers, they're starting to curb the usage, um, by assigning token limits. Um, so again, it goes back to educating people from prompt engineering.
134
00:23:41.616 --> 00:23:47.176
So teaching people how to... like, what kind of questions to ask and how to ask them, I think that's important.
135
00:23:47.756 --> 00:23:59.336
Teaching them about the different concepts of prompts, context, instructions, skills, that's important as well. And also standardizing certain things for u- their usage. So when I tell people, "Okay, build a new...
136
00:23:59.486 --> 00:24:11.316
You know, build, um, AI skills or, as a template for your team or, you know, figure out context, um, or prompt caching on your own," they would not exactly know what to do.
137
00:24:11.396 --> 00:24:15.776
So that I think that's w- where the role of FinOps and AI and tokenomics comes in.
138
00:24:16.196 --> 00:24:25.596
We give them- We give, uh, non-technical teams, uh, prebuilt guidelines or even prebuilt templates for, like, AI skills that are approved.
139
00:24:25.656 --> 00:24:51.896
So rather than ha- having them figure out, you know, long prompts from, on their own from scratch, we can give them guardra- guidel- guidelines to work from so they don't have to start something that's really daunting, but they know exactly what we're asking for and, um, have them trained that way so they are optimizing their usage rather than us having to tell them each time, "Hey, this is how you should be writing this prompt."
140
00:24:53.076 --> 00:25:01.526
Yeah. I think it's, it goes all about, you know, uh, enablement and, you know, call it, like- Yeah... shift left or whatever you wanna call it.
141
00:25:02.036 --> 00:25:18.676
Uh, like, you know, putting the, the workload into a pre, you know, a proactive approach, or you try to people to use it efficiently from scratch and not going back after the, you know, the spend was made, like, "Hey, you need to do this, and you need to do, you need to do that."
142
00:25:19.485 --> 00:25:29.236
And on, on AI training, like, what are your tips or what do you think, let's say, what topics are the, let's say, the easiest to...
143
00:25:29.956 --> 00:25:36.276
that you see non-technical people, for example, learn that could help companies, you know, reduce their spend?
144
00:25:37.176 --> 00:25:48.716
I think right now I'm seeing a lot of, um, companies, a lot of AI leaders offering free, uh, courses on their website. I know Claude is doing it, OpenAI, Gemini.
145
00:25:48.876 --> 00:25:55.476
I think the basic foundational course, everyone should be taking it. If you wanna use AI, I think that should be for everyone.
146
00:25:55.876 --> 00:26:14.476
And from there, I think it would be really helpful if they have some kind of, like, AI champion, um, form in, within their own team, their co- um, their department, so that way that person is more, gets, maybe pursues, like, a further education on it.
147
00:26:14.496 --> 00:26:23.136
But again, it c- it can... You know, there is so much information everywhere. Like, you can watch it on YouTube, right [laughs] to learn- Yeah... more basic, uh, foundational AI knowledge.
148
00:26:23.736 --> 00:26:35.516
And it sh- again, it should go, I think it should really start from prompt engineering. I used to laugh at it, but a lot of people don't know how to do it, uh, don't know how to properly ask the right questions.
149
00:26:35.616 --> 00:26:44.836
I think it can start from prompt engineering, and then understand different terminology that we're using when it comes to AI costs. So AI, uh, not just AI cost, but, like, AI structure, right? Gen AI structure.
150
00:26:45.016 --> 00:26:59.996
So learning about prompt, caching, um, context, instructions, et cetera. So I think that's where they can start, and then once they go advanced, get advanced skills, and they can go anywhere [laughs] I guess right now.
151
00:27:00.556 --> 00:27:07.276
Yeah. It's... Who knows what other materials they will have, right? No, that's for sure, and we're seeing, like, marketing people doing PR request in GitHub.
152
00:27:07.316 --> 00:27:18.896
So, uh, there is a lot of, you know, a lot things going on and a lot of things changing, and there is a, I think there is a learning curve already for, especially for non-technical people to understand what's going on.
153
00:27:19.526 --> 00:27:29.676
Yes. Uh, but I think once we see, you know, the f- what we are already seeing, like, the first restriction on token usage, now everyone will try to be more efficient.
154
00:27:29.796 --> 00:27:40.396
I think probably, I estimate, like, 2027 is going to be more the year where everyone is going to be controlling AI costs and see what's going to happen.
155
00:27:40.536 --> 00:27:45.096
[laughs] Yeah, I think right now that's the direction that my role is taking.
156
00:27:45.276 --> 00:28:00.916
Uh, it's, you know, some of it is token based and some of it is license based, and right now I'm just trying to make sure there are all the guardrail, right, right, guardrails in place to prevent costs from going crazy once the token usage limitation kicks in, right?
157
00:28:01.016 --> 00:28:07.116
So, um, I think for technical people, that's the kind of, um...
158
00:28:07.156 --> 00:28:20.576
As FinOps people, actually, we can, what we can do to pr- prevent that is even now, like, you can set the monthly team usage for AI, and then eventually once it becomes token usage, you can translate into maybe showback, right?
159
00:28:20.676 --> 00:28:29.576
So show monthly team usage in simple dollar figures rather than just say, "Hey, this team used 10,000 tokens." Like, what does that mean? Nobody understands- Yeah... outside of FinOps team.
160
00:28:29.716 --> 00:28:40.296
So you wanna set it up as simple dollar value so you can see who's using how much, uh, in terms of dollar value in AI usage. Oh, that's, that's a pretty good advice.
161
00:28:40.576 --> 00:28:52.456
Um, so yeah, I think with that, we, we've already covered, like, a lot of value. So thanks, uh, Rafaela, for, for being here today, and I think you have provided a lot of wonderful insights to, to the people.
162
00:28:52.516 --> 00:29:02.156
So thanks for taking the time today. Thank you so much. I mean, w- hopefully this information will stay relevant [laughs] by the time the podcast gets released because who knows?
163
00:29:02.176 --> 00:29:07.976
The latest, uh, information from AI is, you know, is rapidly changing, so. [laughs] Yeah.
164
00:29:08.016 --> 00:29:17.746
Yeah, we hope so, and we hope that in the, in the time that we take to edit and publish the, you know, it's not a radical change in the, in the ecosystem. I hope summer- Yeah...
165
00:29:17.756 --> 00:29:25.156
you know, gives us a break and nobody does anything crazy even though the OpenAI thing was in July. So we never know. Right. Um. But, I mean, I don't know.
166
00:29:25.296 --> 00:29:34.716
Maybe another AI will hack into another one and change the whole environment, [laughs] so. Uh, let's hope we have a, a quiet summer, and then maybe September. Peaceful summer. [laughs] Yeah.
167
00:29:34.736 --> 00:29:44.396
And then in September we'll have all the, all the issues going on. Sure, the thriller. [laughs] Um, so yeah. Thanks. Thanks for taking the time, and see you in the next episode. Bye-bye. Thank you. Bye.
168
00:29:44.536 --> 00:29:55.776
[upbeat music]
1
00:00:00.160 --> 00:00:03.780
Welcome, everyone, to another episode of [pop] the FinOps Weekly Podcast.
2
00:00:03.960 --> 00:00:22.259
And today, we are going to talk about AI and how it differs from FinOps', uh, usual work and how to control, uh, their cost, techniques that you can use, how to feed the models correctly to your workloads, and, you know, how to train your people here, uh, into AI cost management.
3
00:00:22.820 --> 00:00:30.300
And for helping me with that, we have Rafaela. Rafaela, how are you? I'm doing well. How are you, Victor? I'm doing great. I'm doing great. So we met in, in FinOps X.
4
00:00:30.320 --> 00:00:40.660
It was wonderful to finally meet you in, in person, and you know, definitely wanted to, to chat with you, uh, on FinOps here in the, in the podcast. So thanks for, for taking the time today.
5
00:00:40.960 --> 00:00:45.980
And, you know, wanted to, to talk about AI. So let's, let's go, uh, into it.
6
00:00:46.000 --> 00:00:57.940
So I've thinking and I've talked with a lot of people about the difference between the usual, uh, issue with, you know, cloud cost that FinOps, uh, try to help, and now all the issues that token...
7
00:00:58.080 --> 00:01:06.120
that, uh, you know, the tokenomics practice, the AI cost practice has. So can you tell us a bit more about the difference? Because AI is impacting now
8
00:01:07.040 --> 00:01:16.180
everyone in the company, and, you know, cloud is only used by, uh, the IT department. So can you tell us a bit more about the, those differences that you're finding? So traditionally, I...
9
00:01:16.420 --> 00:01:20.160
So my, in my current role, I am focusing on cloud cost only.
10
00:01:20.360 --> 00:01:32.460
So traditionally speaking, for FinOps practice within cloud, we only had to educate the engineers and the leadership on how to optimize costs and control costs, right? But nowadays in AI,
11
00:01:33.440 --> 00:01:44.280
everyone's using it in the company, everyone's encouraged to use it. So it w- it's not just technical people that we are talking about, but the HR, the marketing team, or admin people that everyone's using.
12
00:01:44.440 --> 00:01:56.720
So there's that gap of speaking their language, uh, talking to, to non-technical people as well as technical... usual technical users. Yeah. No, that's true.
13
00:01:56.780 --> 00:02:10.139
That's one of the things that we got from, uh, FinOps X that, you know, a lot of, like... I think the numbers were, like, 80 to 90% of the applications developed by AI are out of the IT, uh, uh, department.
14
00:02:10.199 --> 00:02:17.680
So that means, like, you know, uh, 9X, uh, the workload and, and definitely it's a problem that now is full company.
15
00:02:17.740 --> 00:02:32.040
So, you know, what, what are the challenges that you are finding and, and how you think, you know, the FinOps practice can help govern those, those costs now, now generated by, by AI? I think right now when...
16
00:02:32.150 --> 00:02:40.960
Let's go back to the term tokenomics. Hopefully, that term- Yeah... [laughs] remains relevant by the time this podcast comes out, 'cause who knows what term we're gonna be using next. But- Yeah...
17
00:02:41.040 --> 00:02:50.600
um, if I'm talking to somebody who's non-technical, even technical people nowadays, when I'm talking about, "Hey, you're using X number of tokens," that really doesn't mean anything, right?
18
00:02:50.700 --> 00:03:04.140
So we have to kind of transl- again, translate into language that people can understand in the context of their own work. So let's say you're cr- creating a customer support ticket from the s- support team.
19
00:03:04.700 --> 00:03:14.740
We have to say, "This... It, it costs X amount of money for X amount of tokens that you're using," instead of saying, "It's costing 5,000 tokens." What does that mean, right?
20
00:03:14.840 --> 00:03:22.120
So it's a- having the c- giving the right context to the metrics that we'll be using for AI.
21
00:03:22.900 --> 00:03:33.720
And the other issue is that there are all these related costs that, uh, to AI that, uh, that's not just strictly gen AI costs that you might be seeing as a line item in your billing.
22
00:03:34.580 --> 00:03:47.190
Um, that's hard to track because of tagging issues. But that's... I think tagging issues is something that we all have been struggling with since the beginning of FinOps', uh, practice- Yeah... has been, uh, flow, right?
23
00:03:47.280 --> 00:03:54.400
It's not something that's new. But there are many hidden cost items that people are not m-might, might not be familiar with right now.
24
00:03:54.450 --> 00:04:06.760
And I'm still, you know, figuring out myself within my company and, you know, on my own as well. So there are a lot of uncertainties that people face with, uh, that comes with AI cost. Yeah. No, that's, that's true.
25
00:04:06.820 --> 00:04:13.080
There is... There are also, I think, the, the undeterministic things, uh, related to AI.
26
00:04:13.160 --> 00:04:21.320
So, you know, you never know the output that it's going to be, how much tokens are going to be consumed in certain things unless you, you know, you constrain it a lot.
27
00:04:22.120 --> 00:04:33.220
Um, so there is a lot of variables that are, let's say, more out of our control. Rather than in cloud, you can be more predictable, let's say, unless you have spikes and everything. Um, but it's less predictable.
28
00:04:33.280 --> 00:04:40.780
And, you know, one, one of the major things, uh, that I'm seeing is the governance issue, um, because now everyone has access.
29
00:04:40.940 --> 00:04:51.840
So now, before in, in FinOps, let's say, so we had the issue in governance because only a certain set of people will use it, um, and we need to govern that, and that was a problem already.
30
00:04:51.860 --> 00:05:04.100
But if you have a full, uh, company-wide technology enablement like A- AI, now the cost governance is a whole company government, uh, like a whole com- company government issue.
31
00:05:04.160 --> 00:05:16.200
So, you know, why do you think this cannot stay only in FinOps, and how do you, can control all the cost governance across, across the company? I think it goes into...
32
00:05:16.640 --> 00:05:27.220
It goes back to the knowledge of technology that people might be having, right? Again, there is a different levels of technical knowledge that w- all the AI users will be having. And
33
00:05:28.200 --> 00:05:43.460
again, without guardrails, as you mentioned, it can go quite [laughs] devastating. It can have a quite devastating effect on the cost because people might be uploading, let's say, like, 10,000-page PDF, uh, to, uh,
34
00:05:44.360 --> 00:05:48.360
gen AI that they're not supposed to, 'cause that's going to drive a crazy amount of cost.
35
00:05:48.400 --> 00:05:57.460
So I think that having some kind of guardrails and training for, to people to know how to use AI correctly, and also from security perspective as well.
36
00:05:57.500 --> 00:06:07.644
It's not just a cost issue that, you know, people should be knowing what to put into AI, what not to put into AI, and what kind of-... questions to ask and how to ask those questions, that's very important.
37
00:06:08.444 --> 00:06:20.104
And right now, I think, uh, and I'm not saying this about in, in my particular company or, like, anyone in general, but I know that some people are, you know, prone to using shadow AI as well.
38
00:06:20.264 --> 00:06:31.704
That's not [laughs] approved, explicitly approved in their company, um, but they still upload it, uh, upload whatever information they want to into their own accounts in AI. Yeah.
39
00:06:31.744 --> 00:06:43.004
So I think that's the, um, one of the many it- risks that people are facing. Yeah. And, you know, but that's also, like, a security risk and, like, a lia- liability risk.
40
00:06:43.084 --> 00:06:54.644
We already have seen, uh, that, uh, popping up in several ways. We just have a, a recent, like, issue with, uh, OpenAI and Hugging Face weird thing- Yeah... that happened.
41
00:06:54.664 --> 00:07:07.234
[laughs] The AI, yeah, the AI acting upon itself, um, which is crazy. [laughs] Yeah. So that's also, like, another... I, I was discussing that before our call with another friend. It's like the,
42
00:07:08.444 --> 00:07:20.144
sometimes the value is not in the, in the value generation, but in the risk prevention. So, like, you, for example, let's say a fraud, uh, detection system.
43
00:07:21.004 --> 00:07:36.964
Fraud detection system is making you not to lose money, so it's making you, let's say, uh, you know, value, let's say value, value loss avoidance or however you wanna call it, because you are going to have, you may be a, you may have certain set of frauds or certain set of security attacks and everything.
44
00:07:37.304 --> 00:07:49.544
If you allow all of them, let's say you lose whatever, but if you are able to, to generate a system that prevents that, so you avoid having these security issues, you are adding value to your company, like in,
45
00:07:51.124 --> 00:08:04.044
you know, value, let's say, instead of, uh, value creation is not, is avoiding value loss, let's say. Uh, so that's also an interesting thing that we need to, uh, consider as well because we need to be...
46
00:08:04.224 --> 00:08:15.924
No, we are not always generating. We always, sometimes we need to, you know, uh, prevent losing value. [laughs] Yeah. I think it goes back to the concept of, like, ROI and unit metrics- Mm-hmm...
47
00:08:16.304 --> 00:08:26.804
um, unit costs, because you wanna look at how much AI tokens or whatever consumption, AI cost that you're consuming versus how much value you're creating out of it, right?
48
00:08:27.124 --> 00:08:28.824
The, it goes back to the whole concept of FinOps.
49
00:08:28.844 --> 00:08:39.364
It's not about saving the most amount of money, but how much value you're getting out of per dollar that you're spending, or per euro [laughs] per eu- for your case, or per token that we'll be using, right? Mm-hmm.
50
00:08:39.434 --> 00:08:41.264
So it's not, um...
51
00:08:41.404 --> 00:08:59.374
I remember, like, I was talking to a friend about their AI usage tracking within their company, and they have, like, a leaderboard, uh, showing which develop we're using, was using the most amount of tokens or u- using the AI the most, but they weren't really tracking what it was being used for- Yeah...
52
00:08:59.374 --> 00:09:03.024
which makes absolutely no sense. But right now, I think I've, I think a lot of companies are doing it.
53
00:09:03.044 --> 00:09:14.534
They wanna see how much AI usage is being done within the company, how many people are using it to do whatever they want, whatever tasks they wanna get completed. But we have to go back to value creation, right?
54
00:09:14.584 --> 00:09:23.714
Like, what kind of, like, h- what exactly are you doing with the AI that's saving us money, that's, uh, preventing certain things from happening, right? So that's the, um,
55
00:09:24.804 --> 00:09:37.284
metrics that we need to be looking at, rather than just looking at, uh, how much time we're cutting off, how much, uh, dollar we're saving from, by using AI instead of having another person do it. Yeah. No, that's true.
56
00:09:37.384 --> 00:09:51.284
I think that'll, that'll be solved once a significant bill or a bill shock, uh, happens that puts everyone in, in the right place. Yeah. Like, the bill doesn't lie. [laughs] So, uh, it's, it's going to be fun.
57
00:09:51.324 --> 00:10:01.024
And, and one of the things with the, with the bills and, you know, uh, that we suffer also in cloud was the, the volatility of prices, or like, let's say, the less predictability of the pricing.
58
00:10:01.064 --> 00:10:11.494
You know, we, coming from the, the data center or the on-prem perspective is, like, you buy this stack of or rack of servers that cost- Mm-hmm... that amount of money, and you amortize over time.
59
00:10:12.224 --> 00:10:22.443
Then cloud was more or less predictable because you are consuming and you are creating and, and reducing workload. But here is, like, everyone is spending, like, every time.
60
00:10:22.544 --> 00:10:38.624
Uh, like, uh, any command or anything, uh, introducing to cloud or whatever AI provider is a cost, uh, decision. So how are you seeing, like, the volatility of the, of the AI, you know, cost and, and ecosystem? Yeah.
61
00:10:38.904 --> 00:10:49.594
The g- I think it's more akin to serverless. Um, unless you have certain guardrails in place, serverless can drive up the cost quite significantly, right? Mm-hmm.
62
00:10:49.604 --> 00:11:00.824
And that's the, that's, you know, one of the things that people don't expect so much 'cause they think, "Oh, serverless option is going to be cheaper," which is not true unless you have, you know, everything configured correctly.
63
00:11:00.904 --> 00:11:05.904
So ins- uh, unlike, you know, as you mentioned, unless un- um, unlike traditional cloud
64
00:11:06.884 --> 00:11:18.404
infrastructure, where you can have more predictable usage in an hour, um, and AI is more driven by user interaction styles, the autonomous agent loops, you know?
65
00:11:18.444 --> 00:11:20.904
So that's the kind of guardrails that you can have in place.
66
00:11:21.204 --> 00:11:34.084
How many, uh, connect, API connections that can be made, for example, uh, how many loops you can try before it breaks, um, and also controlling the payload size. So, you know- Mm-hmm... controlling the context.
67
00:11:34.124 --> 00:11:47.924
And, um, having those guardrails in place and also training the users how to use it properly ahead of time would prevent that kind of volatility from, um, going out of hand. Yeah. No, that's true.
68
00:11:47.984 --> 00:11:54.144
Like, uh, everything reduces to, to guardrails and setting, you know, policies in place that, that makes everything controlled.
69
00:11:54.264 --> 00:12:02.484
And so you mentioned, like, uh, several topics that are related to AI, like techniques about, you know, the prompt, uh, the context, and the, the guardrails.
70
00:12:02.944 --> 00:12:13.224
On your side, like, what's your- I'm going to say your favorite to, to tweak, but also the ones that is providing you the most results. What are you, you know, looking, uh, for more?
71
00:12:13.284 --> 00:12:19.484
Like, it's the context or it's the prompts or it's creating guardrails. What's the one that you tend to work the most?
72
00:12:20.504 --> 00:12:30.784
So I used to laugh at people, [chuckles] like a few years ago when people used to use the term prompt engineering, because for me personally, I thought that was, uh, kind of like overkill.
73
00:12:30.844 --> 00:12:41.564
I thought people knew how to do it. But I've seen some prompts created by other people and realized that not everyone knows how to use it properly, right? So- Mm-hmm... again, the whole...
74
00:12:42.224 --> 00:12:52.224
It, it depends on the how the AI cost is triggered. Um, some companies, they have it triggered by license base, and for now it's kind of unlimited usage.
75
00:12:52.724 --> 00:13:08.064
But eventually what I think what all the AI companies are going to do is going to translate- uh, transfer it into, um, token usage or more limited usage based on, um, how many prompts or how many contexts or whatever they can have, right?
76
00:13:08.524 --> 00:13:16.324
So I think it needs to start from prompt engineering. You have to re- really know like how to ask the questions and in a more concise
77
00:13:17.304 --> 00:13:31.504
manner so that way y- you're giving it enough information for it to process, but also leaving it enough room to, for it to think on its own, rather than kind of feeding you the answer that you wanna hear from it.
78
00:13:32.204 --> 00:13:47.804
You have to be able to ask the right amount of... give it the right amount of context, right amount of words or prompt, and have the flexibility built into your, um, answers. Does that make sense? [chuckles] Yeah.
79
00:13:47.844 --> 00:13:59.264
No, no. I think- Yeah... it's finding that right balance between instructions and, and flexibility and being able to- Right... provide. Because if it's, you know, depends also on the answer that you want.
80
00:13:59.324 --> 00:14:06.224
If you want it very straightforward and, you know, same format repeatable, that's one thing and maybe AI is...
81
00:14:06.504 --> 00:14:15.574
I would say maybe AI, you need a lot of pre-processing before or post-processing after AI and to give it to the, like the right structure and everything. Right.
82
00:14:15.664 --> 00:14:22.714
But also if you want a more free, let's say documents, images, et cetera, you want a more creative thing, then that- Right... hmm,
83
00:14:23.664 --> 00:14:33.004
really, uh, needs a more flexible thing because otherwise it's going to be like the same answer. And you ask the another question, it's like the same answer. And you're like... For example, that happens a lot.
84
00:14:33.404 --> 00:14:44.064
I'm not going to, uh, cite, but if you, if you put like Gemini, I think it happened to me like a lot with Gemini Pro, uh, l- before one of the, one of the models just would like...
85
00:14:44.844 --> 00:14:50.564
It was so structured, like if you give it super structured answers, it will give you like the same answer over and over and over again.
86
00:14:50.604 --> 00:14:58.764
It's like at a given point that the context window, uh, I think it was like, you know, super compressed and that's also another point. [bell dings] Like the,
87
00:14:59.744 --> 00:15:07.724
the context and, and the, the window itself, like if you have long chats where, you know, uh, the model starts to compress the...
88
00:15:08.244 --> 00:15:15.964
now they compress the, the context, you end up with the latest ones, like the latest messages and everything, and that causes a, a case.
89
00:15:16.044 --> 00:15:23.624
I don't know if you've seen like these issues with the, with the context window and, you know, these long- Right... uh, situations. So yeah, it's...
90
00:15:24.024 --> 00:15:30.884
You don't want the conversations to go too long because then it'll be running everything each time you ask a new prompt, right?
91
00:15:30.944 --> 00:15:39.644
So you want a set of instructions which is, I guess, it, it can work as both as guardrail and also just kind of general guideline, right?
92
00:15:39.904 --> 00:15:52.324
Um, for example, like the instruction that I'll be using as a FinOps person, I would say like act as a cloud FinOps expert, and then that's how it would, um, construct its answers all the time based on my instructions.
93
00:15:52.724 --> 00:15:57.944
And always also you can put up like more security, um, related instructions based on that.
94
00:15:58.044 --> 00:16:12.504
Say, "Do not access this X, Y, and Z," and then AI would not be, um, running that prompt every time, but it would just know, um, consistently, so you don't have to y- type it each time that you ask a question.
95
00:16:13.284 --> 00:16:28.024
And by doing s- by setting instructions, because it's a repeated, uh, process that you h- always have to ask, you're saving the context, uh, um, you're, you're, you know, condensing the context window and, you know, saving tokens by having less words to repeat each time.
96
00:16:28.744 --> 00:16:38.224
And also like the other thing that I've been do- using, um, lately that I really enjoy is using skills. Uh, there are a lot of skills that people are, have been cr- creating. You know, you mentioned Gemini.
97
00:16:38.264 --> 00:16:47.584
The Gemini has their own, um, repository of skills. There are task specific playbooks that you can be using, um, but instead of have...
98
00:16:48.264 --> 00:17:01.584
But unlike instructions, you don't have to load it yourself, but you kind of trigger AI to pull from it, so that way it's not always sitting on your instructions, which can again blow it up and kind of limit your answer.
99
00:17:02.364 --> 00:17:15.624
Um, but it can always refer to it, um, acting as an- another like a expert source that you can pull from each time that you are asking the relevant questions to AI. Yeah.
100
00:17:15.674 --> 00:17:26.724
And that's also something that people need to be careful about is like depending on... I think it depends on the model and the system that you're using. Uh, the skills and the MCPs can generate a lot of
101
00:17:27.644 --> 00:17:40.044
context like or overhead to the model when, when it's trying to answer, and especially I think it's MCPs because there is a lot of, you know, words and things that could trigger and it could think like, "Yeah, I think that'll be..."
102
00:17:40.104 --> 00:17:51.744
And it can, uh, you know, uh, consume a lot of the context and the, the reasoning of the model. So you need to be careful about like, uh, that also relates to, you know, the context and the specialization.
103
00:17:51.764 --> 00:18:05.476
It's like you need to put it in constrained things and in guardrails that, okay, you have this set of skills, you do this, and you are specialized in that and you don't have like, I don't know, 100 skills enabled and your context is-...
104
00:18:05.496 --> 00:18:18.716
all skills and you cannot introduce anything because it's not going to happen. It's, you know, we try to make specialized people, uh, and in this case, specialized agents, uh, for, for making it, uh, work. And
105
00:18:19.896 --> 00:18:23.476
other things that, that we've seen in AI is the, the caching system.
106
00:18:23.576 --> 00:18:35.816
So caching is very, uh, interesting in, you know, in databases, like usual, uh, things, but in, in prompts it's kind of the same idea but different because it's less predictable.
107
00:18:36.306 --> 00:18:46.896
How are you seeing the, the prompt caching, uh, working for you, or how are you seeing, or how you are you taking advantage of, of the prompt caching? Yeah. That's, um...
108
00:18:48.516 --> 00:18:54.696
So I think right now that's something that we are working on as a company- Mm... to figure out for es- especially for each user.
109
00:18:54.836 --> 00:19:06.376
So, you know, as I mentioned, going back to the beginning of the recording that we were saying that it's now expanded to the entire company, so it's not just technical people that I'm speaking to. So it's kind of, I...
110
00:19:06.716 --> 00:19:28.976
right now, like in the process of gathering information on what's most, like, commonly asked information or what's something that it should be, um, maintaining, um, so we can cache the information rather than, um, uh, cache the, like, static repeated, uh, information from the prompt rather than having to send same information over and over again.
111
00:19:28.996 --> 00:19:41.236
And that's something, um, that, you know, each team or, like, you know, each user has to figure out, and we... Like, right now my process is to provide some kind of guardrails, 'cause a lot of people don't even...
112
00:19:41.276 --> 00:19:50.796
Like, they're not even aware of this whole idea of cach- caching and, you know, prompt caching or context caching. And there are different options to do it, um, but
113
00:19:51.756 --> 00:20:01.156
yeah, right now we're still [laughs] in the process of figuring it out. Yeah. It's not, it's not... I think it's not as easy as, uh, in the database world, let's say. Uh, even though it's...
114
00:20:01.276 --> 00:20:12.136
I always think of it like a double-edged sword, uh, because you're gonna have a lot of advantages if done right, but if done wrong, uh, caching can be a, a pain, uh, to, to solve.
115
00:20:12.736 --> 00:20:26.996
And, and other thing is y- that, that I've... Just in the, I think it was in the past podcast, was the selecting the right model, uh, like fitting the request to the right model. How are you seeing, like, people...
116
00:20:27.066 --> 00:20:39.816
Because I don't know, maybe people are using Opus 4.8 to request, uh, what's the hour or what time is it. Um, so how are you seeing that? What are your advice on, you know, on the right sizing of models?
117
00:20:39.956 --> 00:20:49.596
Let's call it that way. Yes. So yeah, right now the approach that I'm pushing for is a T-shirt size approach, um, that you've probably heard from, you know, FinOps practice.
118
00:20:49.676 --> 00:20:59.676
So what it is, is that we have, like, small, medium, large size, depending on the usage. So small is probably, like, you know, when you're thinking about lightweight usage, let's say composing an email,
119
00:21:00.576 --> 00:21:05.196
uh, or creating, um, meeting summary, et cetera.
120
00:21:05.316 --> 00:21:16.276
That should be using the lightest model possible because, again, like, different models have different token con- consumption, and you don't wanna overblow, like, Opus, for example, using, using it for your email consumption.
121
00:21:16.336 --> 00:21:27.136
I don't think it... Like, for email usage, I personally don't think you should be using agentic AI [laughs] to begin with. Uh, but some people do it. So that's the... where T-shirt size approach comes in.
122
00:21:27.256 --> 00:21:36.156
So you have small for lightweight usage and large or, like, medium or large for more heavyweight usage, meaning something more advanced, more technical.
123
00:21:36.316 --> 00:21:46.796
So let's say for developing a certain, like, non-proda- environment or coding something new that's more, a lot more complicated or, like, a lot more mathematical, et cetera.
124
00:21:46.916 --> 00:21:52.716
So, um, people would not have access, the same access to different types of models. It would have...
125
00:21:52.916 --> 00:22:01.196
It, it would depend on the department, def- de- depend on the apps they're working on, depending on the projects, et cetera. That's, that's true.
126
00:22:01.296 --> 00:22:13.876
We need to avoid, like, you know, setting default Opus or setting default Sonnet, and then every, uh, for everything we use Sonnet instead of, you know, right size and, and what's the task, and based on the task we can, we can set it right.
127
00:22:13.936 --> 00:22:24.395
So we need to be a little bit more conscious about it. Especially when it's your own money, you, you see it more and you are more careful about it, but when it's company money, it's, it's different. Um,
128
00:22:25.456 --> 00:22:29.856
and wanted to, to also dive into the training because, um,
129
00:22:30.796 --> 00:22:43.836
on FinOps there was a lot of effort into the, you know, educating the other departments, educating engineers, finance, and everyone involved to, uh, to be aware of costs, uh, in the, in the cloud side.
130
00:22:44.496 --> 00:23:03.636
So, and before, like, before in the, I think, 2025, mostly all the big companies were about, you know, AI, um, you know, um, scaling and AI training for the company, but less, like, a skill and less about AI consumption, uh, culture.
131
00:23:04.076 --> 00:23:15.396
So how are you seeing the differences and, you know, uh, now that it involves more non-technical crowd that in, in FinOps where it was targeting- Yeah... IT? Yeah. I think, like, you know,
132
00:23:16.496 --> 00:23:27.976
like, I think as late as, like, last year, it was a lot more free usage encouraged, uh, meaning- Yeah... we didn't have the concept of tokenomics. [laughs] So it was kind of run whatever you want in AI prompt.
133
00:23:28.016 --> 00:23:41.446
But right now it's, you know, a lot of companies, uh, AI providers, they're starting to curb the usage, um, by assigning token limits. Um, so again, it goes back to educating people from prompt engineering.
134
00:23:41.616 --> 00:23:47.176
So teaching people how to... like, what kind of questions to ask and how to ask them, I think that's important.
135
00:23:47.756 --> 00:23:59.336
Teaching them about the different concepts of prompts, context, instructions, skills, that's important as well. And also standardizing certain things for u- their usage. So when I tell people, "Okay, build a new...
136
00:23:59.486 --> 00:24:11.316
You know, build, um, AI skills or, as a template for your team or, you know, figure out context, um, or prompt caching on your own," they would not exactly know what to do.
137
00:24:11.396 --> 00:24:15.776
So that I think that's w- where the role of FinOps and AI and tokenomics comes in.
138
00:24:16.196 --> 00:24:25.596
We give them- We give, uh, non-technical teams, uh, prebuilt guidelines or even prebuilt templates for, like, AI skills that are approved.
139
00:24:25.656 --> 00:24:51.896
So rather than ha- having them figure out, you know, long prompts from, on their own from scratch, we can give them guardra- guidel- guidelines to work from so they don't have to start something that's really daunting, but they know exactly what we're asking for and, um, have them trained that way so they are optimizing their usage rather than us having to tell them each time, "Hey, this is how you should be writing this prompt."
140
00:24:53.076 --> 00:25:01.526
Yeah. I think it's, it goes all about, you know, uh, enablement and, you know, call it, like- Yeah... shift left or whatever you wanna call it.
141
00:25:02.036 --> 00:25:18.676
Uh, like, you know, putting the, the workload into a pre, you know, a proactive approach, or you try to people to use it efficiently from scratch and not going back after the, you know, the spend was made, like, "Hey, you need to do this, and you need to do, you need to do that."
142
00:25:19.485 --> 00:25:29.236
And on, on AI training, like, what are your tips or what do you think, let's say, what topics are the, let's say, the easiest to...
143
00:25:29.956 --> 00:25:36.276
that you see non-technical people, for example, learn that could help companies, you know, reduce their spend?
144
00:25:37.176 --> 00:25:48.716
I think right now I'm seeing a lot of, um, companies, a lot of AI leaders offering free, uh, courses on their website. I know Claude is doing it, OpenAI, Gemini.
145
00:25:48.876 --> 00:25:55.476
I think the basic foundational course, everyone should be taking it. If you wanna use AI, I think that should be for everyone.
146
00:25:55.876 --> 00:26:14.476
And from there, I think it would be really helpful if they have some kind of, like, AI champion, um, form in, within their own team, their co- um, their department, so that way that person is more, gets, maybe pursues, like, a further education on it.
147
00:26:14.496 --> 00:26:23.136
But again, it c- it can... You know, there is so much information everywhere. Like, you can watch it on YouTube, right [laughs] to learn- Yeah... more basic, uh, foundational AI knowledge.
148
00:26:23.736 --> 00:26:35.516
And it sh- again, it should go, I think it should really start from prompt engineering. I used to laugh at it, but a lot of people don't know how to do it, uh, don't know how to properly ask the right questions.
149
00:26:35.616 --> 00:26:44.836
I think it can start from prompt engineering, and then understand different terminology that we're using when it comes to AI costs. So AI, uh, not just AI cost, but, like, AI structure, right? Gen AI structure.
150
00:26:45.016 --> 00:26:59.996
So learning about prompt, caching, um, context, instructions, et cetera. So I think that's where they can start, and then once they go advanced, get advanced skills, and they can go anywhere [laughs] I guess right now.
151
00:27:00.556 --> 00:27:07.276
Yeah. It's... Who knows what other materials they will have, right? No, that's for sure, and we're seeing, like, marketing people doing PR request in GitHub.
152
00:27:07.316 --> 00:27:18.896
So, uh, there is a lot of, you know, a lot things going on and a lot of things changing, and there is a, I think there is a learning curve already for, especially for non-technical people to understand what's going on.
153
00:27:19.526 --> 00:27:29.676
Yes. Uh, but I think once we see, you know, the f- what we are already seeing, like, the first restriction on token usage, now everyone will try to be more efficient.
154
00:27:29.796 --> 00:27:40.396
I think probably, I estimate, like, 2027 is going to be more the year where everyone is going to be controlling AI costs and see what's going to happen.
155
00:27:40.536 --> 00:27:45.096
[laughs] Yeah, I think right now that's the direction that my role is taking.
156
00:27:45.276 --> 00:28:00.916
Uh, it's, you know, some of it is token based and some of it is license based, and right now I'm just trying to make sure there are all the guardrail, right, right, guardrails in place to prevent costs from going crazy once the token usage limitation kicks in, right?
157
00:28:01.016 --> 00:28:07.116
So, um, I think for technical people, that's the kind of, um...
158
00:28:07.156 --> 00:28:20.576
As FinOps people, actually, we can, what we can do to pr- prevent that is even now, like, you can set the monthly team usage for AI, and then eventually once it becomes token usage, you can translate into maybe showback, right?
159
00:28:20.676 --> 00:28:29.576
So show monthly team usage in simple dollar figures rather than just say, "Hey, this team used 10,000 tokens." Like, what does that mean? Nobody understands- Yeah... outside of FinOps team.
160
00:28:29.716 --> 00:28:40.296
So you wanna set it up as simple dollar value so you can see who's using how much, uh, in terms of dollar value in AI usage. Oh, that's, that's a pretty good advice.
161
00:28:40.576 --> 00:28:52.456
Um, so yeah, I think with that, we, we've already covered, like, a lot of value. So thanks, uh, Rafaela, for, for being here today, and I think you have provided a lot of wonderful insights to, to the people.
162
00:28:52.516 --> 00:29:02.156
So thanks for taking the time today. Thank you so much. I mean, w- hopefully this information will stay relevant [laughs] by the time the podcast gets released because who knows?
163
00:29:02.176 --> 00:29:07.976
The latest, uh, information from AI is, you know, is rapidly changing, so. [laughs] Yeah.
164
00:29:08.016 --> 00:29:17.746
Yeah, we hope so, and we hope that in the, in the time that we take to edit and publish the, you know, it's not a radical change in the, in the ecosystem. I hope summer- Yeah...
165
00:29:17.756 --> 00:29:25.156
you know, gives us a break and nobody does anything crazy even though the OpenAI thing was in July. So we never know. Right. Um. But, I mean, I don't know.
166
00:29:25.296 --> 00:29:34.716
Maybe another AI will hack into another one and change the whole environment, [laughs] so. Uh, let's hope we have a, a quiet summer, and then maybe September. Peaceful summer. [laughs] Yeah.
167
00:29:34.736 --> 00:29:44.396
And then in September we'll have all the, all the issues going on. Sure, the thriller. [laughs] Um, so yeah. Thanks. Thanks for taking the time, and see you in the next episode. Bye-bye. Thank you. Bye.
168
00:29:44.536 --> 00:29:55.776
[upbeat music]
