UTSKRIFT 🔗
WEBVTT
1
00:00:00.100 --> 00:00:17.800
Welcome everyone to another episode of the Finops Weekly Podcast, and today we are here in our onsite secret location, um, talking about AI costs, and especially we're going to frame it a lot in GCP on how you can select the model, uh, as best as you can to, to your projects.
2
00:00:18.180 --> 00:00:27.919
And to help me with that, we have Vera Basman. Vera, how are you? Hi. Great. Thank you very much for hosting me. Yeah, it's a pleasure to, to have you, and you know, we've been, uh, knowing each other for a while. Yes.
3
00:00:28.020 --> 00:00:39.280
Uh, we met in, in Barcelona and of course in Madrid, uh, in the FinOps Six, and, and now here. And you know, it's great to have, uh, people here in, in Madrid, being able to, to talk to each other.
4
00:00:39.490 --> 00:00:53.020
And you know, today we are going to dive into, into GCP and AI. So for your experience in, in GCP and with AI, one of the factors is, uh, to select a, a good model is pricing. Okay? Yeah.
5
00:00:53.040 --> 00:01:01.920
So depending on the pricing, you select one model or the other based on your case study. You don't always need a more expensive, nor the cheapest one. So how do you think...
6
00:01:02.020 --> 00:01:15.960
how relevant it is, the pricing in the model selection? Oof. I think it is very relevant. Okay. [laughs] Because, well, I think that we are currently in the beginning of the AI revolution. Mm-hmm.
7
00:01:16.460 --> 00:01:28.080
And obviously many companies want to be as much as, you know, on top of things and implement AI, AI models, AI agents in their products or services.
8
00:01:28.820 --> 00:01:44.500
Um, so what we are seeing now a lot is just people going over everything, all the models or testing the, the, the priciest models for all use cases and stuff like that. Uh, and it is still very hard- Mm-hmm...
9
00:01:44.510 --> 00:01:55.620
to break down all those costs. We still see the AI spend like a black box that we really want to- Yeah... turn it into a white box and open it up- Yeah. [laughs]... [laughs] to analyze it.
10
00:01:55.640 --> 00:02:09.259
And you know, we all hear of those, uh, stories of like a developer just prompting, just playing with one AI model, and then in the end of the month you get like a bill of thousands of dollars.
11
00:02:09.979 --> 00:02:23.360
So this is where I think, uh, we came to a realization that, um, we have to implement FinOps in AI. Yeah. You know? Yeah. [laughs] Yeah, that's, that's true because we...
12
00:02:23.500 --> 00:02:32.910
I think we've, uh, overcome the phase of where everyone was experimenting and trying out new things, and you always wanted to try like the latest model- Exactly... and everything, but it was more...
13
00:02:33.520 --> 00:02:39.769
I think it changed when the AI availability was for everyone. So like before, and I wa- I...
14
00:02:39.800 --> 00:02:52.000
it's also the way that I have in FinOps is like the cloud, for example, just to, to make a similarity, is available for a certain side of the company, which is IT people and especially IT people focusing cloud. Yeah.
15
00:02:52.280 --> 00:02:59.160
But AI is available for everyone now, before- It's a very good point... and, and then you need to control it to everyone.
16
00:02:59.260 --> 00:03:09.650
So you need to apply FinOps as, well, as soon as you can because cloud has been there for a while, but AI has been a super short time, and then we are already, you know, handling token budgets- Yeah. [laughs]...
17
00:03:09.720 --> 00:03:19.560
uh, the token maxing thing and all the quotas thing. Yes. So it's, it's super relevant. And, you know, following up on this, we know that okay, in the model selection
18
00:03:20.760 --> 00:03:31.460
there are other factors that you s- that you have because you can not always select based on price. So what do you think are other factors that condition the, the selection of a, of a model? Yeah.
19
00:03:31.800 --> 00:03:45.020
A- actually, I think that what you just pointed out, the fact that unlike cloud costs- Mm-hmm... AI, everyone in the, in our c- in the company touches it. So this is one of the factors.
20
00:03:45.180 --> 00:03:55.480
So I can give you an example that in the recent months I've been working with a big, uh, company in the health sector. Mm-hmm. And we were implementing together Vertex AI. Okay. Okay?
21
00:03:55.540 --> 00:04:06.100
They work mainly with, uh, Google Cloud, and we were implementing Vertex AI. And obviously implementing AI tools, like you said, it touches all departments.
22
00:04:06.420 --> 00:04:19.479
If you want to be as up to date and as automated as possible and stuff like that, it touches all departments. So, uh, other factors beyond price, it will be, I would say two factors, two main factors.
23
00:04:19.620 --> 00:04:29.120
So the first thing will be who is my target audience- Mm-hmm... per model? So for example, I have internal team developers.
24
00:04:29.460 --> 00:04:42.500
They will need one, uh, one type of model, or let's say a R&D team would, would want a model that is more for reasoning. Okay? So, uh, the other factor will be a- Yeah... the use case.
25
00:04:42.550 --> 00:04:58.960
[laughs] So, uh, the, the R&D team would want a, a model that focuses more on reasoning and let's say more internal operational teams in our company, for example, um, the billing team- Mm-hmm...
26
00:04:58.969 --> 00:05:12.820
that needs to issue invoices. Perhaps for them what they need is accuracy, numbers or latency, so things won't be delayed and stuff like that. So, uh, these are two factors that we have to take into account.
27
00:05:13.540 --> 00:05:19.680
Who are we implementing this model or agent for? Mm-hmm. And what are we going to do with it?
28
00:05:20.520 --> 00:05:32.330
Yeah, I think that's a very, a very good set of factors because we're always thinking like, you know, some technology terms about the, you know, the token consumption,
Load full transcript
the efficiency of the model. Yeah.
29
00:05:32.360 --> 00:05:42.280
And you are phrasing it as of, uh, the needs of the people, you know, the, the business, the needs of- As simple as possible. Yeah. Let's not even talk about tokens, you know? Yeah. [laughs] Just what...
30
00:05:42.320 --> 00:05:52.310
tell me what do you need and who needs it, and this is how we'll solve the problem. Yeah. It's a very human [laughs] response to- Yes. [laughs] Exactly... to that. I don't think AI will, will definitely gen- Right.
31
00:05:52.310 --> 00:06:00.390
[laughs]... get that, that response. The most human thing in AI is this. [laughs] Is, is, is to think about, you know, who is using it and what they are using it for, right? Yeah.
32
00:06:00.400 --> 00:06:10.920
It's all about, you know, the value that you're providing to, to the people and, and of course the business reason- Yeah. Exactly... uh, behind. So, okay, now we have the... we have analyzed the pricing.
33
00:06:11.484 --> 00:06:16.384
And we've analyzed other reasons apart from pricing that we have related to, to business.
34
00:06:16.454 --> 00:06:26.004
So we know that, you know, we have analyzed, uh, how we consider pricing into the model selection, and then the other, uh, factors that influences. Yeah.
35
00:06:26.064 --> 00:06:36.194
And, you know, in GCP, there is the, the tool of Vertex, and, you know, we've been doing... Basically, we've been knowing it for a while on the ML and AI and all the AA parts- Yeah, that's true...
36
00:06:36.194 --> 00:06:46.624
like since 2018, since I was playing- So old. Yeah. Oh my God. [laughs] So, you know, can, can you explain to the audience now what's Vertex AI and- Yeah... and you know, all the, all the content that it has?
37
00:06:46.644 --> 00:06:58.314
Yes, definitely. So Vertex AI, uh, which by the way is now called Agent Platform. Mm. Okay. [laughs] They... Like we- You need to- You need to keep up. [laughs] Yeah, you need to give up the marketing side. Yeah. Yeah.
38
00:06:58.504 --> 00:07:07.883
Um, it is a Google Cloud's, uh, one-stop shop- Mm-hmm... for, uh, AI agents and models.
39
00:07:08.104 --> 00:07:19.334
So in Vertex AI, you can do the whole process, from choosing the model, configuring it, and deploying it to production. Mm-hmm. Basically starting to use it. Okay? Mm-hmm.
40
00:07:19.794 --> 00:07:28.524
So I think the good thing is the convenience obviously, and later you can, uh, combine it with, uh, Gemini Enterprise and stuff like that. Okay.
41
00:07:28.604 --> 00:07:35.444
So it's like, uh, the one place for people to handle all the, all the models and all the... Yes. Well, AM platform, it's a very well- Yeah... named.
42
00:07:35.524 --> 00:07:43.864
And, you know, can you devi- dive into more of the specifics about how, you know, you can play with Vertex AI, some of its components? Yeah. Definitely.
43
00:07:43.924 --> 00:07:56.604
And, um, well, we start the conversation saying that price is a big factor. Mm-hmm. Um, and
it is a big factor because people are still learning how to use it and how to use it right.
44
00:07:56.724 --> 00:08:07.584
So I do think that there is like a certain workflow that we can do, um, just to avoid errors, you know? Mm-hmm. Just to make it smoother.
45
00:08:08.464 --> 00:08:16.764
Um, and the good thing is that in Vertex AI, you have all those steps like lined down. Okay. Like in the same menu, you have all the options.
46
00:08:17.444 --> 00:08:29.544
So the first thing that I would do is going to Model Garden, and Model Garden is a catalog that just displays me all the models that Google Cloud offers.
47
00:08:30.304 --> 00:08:42.544
Um, it offers obviously, you know, the big names like Anthropic, OpenAI, obviously the, uh, Google in-house products of Gemini. Yeah. [laughs] Yeah. [laughs] Obviously.
48
00:08:43.264 --> 00:08:53.344
But also, uh, a cool thing is that, um, there are also more like task or industry related, uh, models. Mm-hmm.
49
00:08:53.444 --> 00:09:09.264
So for example, um, this health industry, health company that I've been working with, um, you could, in the Model Garden, uh, select, uh, health under industry. Okay. Like in the filters.
50
00:09:09.324 --> 00:09:18.704
You can go to filters, in the, the specific industry, industry which you're in, choose health, and like choose models that are way more specific for your use case.
51
00:09:19.344 --> 00:09:34.324
And something even cooler is that like you can even drill down even more per tasks. So for example, under health, you can even have like x-ray AI model. Yeah. Which is really cool, you know? It's like, it's very curious.
52
00:09:34.384 --> 00:09:45.244
Like you're like, "What is this?" [laughs] Yeah. It's very specific to the- Yes... to the point, right? Yeah. And it's amazing because AI obviously goes to a, to many sectors. Mm-hmm. Um, so it can be very useful.
53
00:09:46.124 --> 00:09:50.884
Um, yeah, so this is Model Garden. Okay. And this is the first step.
54
00:09:51.064 --> 00:10:04.824
So we are looking there for, you know, from the variety of the models that we have, and let's say we chose a number of models, and now it's time to, to evaluate it. And this is the name of the page. [laughs] Okay.
55
00:10:04.914 --> 00:10:18.384
[laughs] The name of the section in Vertex AI, it's called Evaluation. And there we are testing... Uh, this is the staging. Okay. Uh, uh, this is the second- The sta- the staging part of the- Yes... of the workload.
56
00:10:18.644 --> 00:10:30.484
Exactly. So we are testing the, the different models. Uh, we, uh, connect our databases that we work with, uh, different prompt engineering, stuff like that.
57
00:10:31.124 --> 00:10:41.214
Uh, and basically there we need to prepare it to, for production- Okay... later. So this is the main- So you are like validating that everything works, that the outputs are correct- Yes...
58
00:10:41.224 --> 00:10:48.454
that, uh- And, and it, that it answers our business needs- Okay...
uh, of what we want to do with these models, obviously. Yeah.
59
00:10:48.484 --> 00:11:04.844
So okay, now that we have the, the model, you know, evaluated, we have followed all the filters in, um, in Vertex AI to, you know, properly select, like what is the criterias that we need to apply in Vertex AI to actually select the model?
60
00:11:05.184 --> 00:11:16.404
So there are different, uh, FinOps best practices- Mm-hmm... that we can take into account when, uh, selecting and working with different models. Mm-hmm. Um, let's start with the more common ones.
61
00:11:16.524 --> 00:11:22.744
So obviously you have like different generations of models. Um, and, uh, the, the...
62
00:11:22.994 --> 00:11:34.964
There is a basic assumption of like, hey, if you t- if you're working with the newest model, from one hand it will give you better performance, but very, very high, uh, but on the other hand, it will give you also very high costs.
63
00:11:35.104 --> 00:11:45.064
Mm-hmm. Okay? And so the first assumption is saying, well, maybe you don't need to use, uh, uh, the, the most, the priceless model. Yeah. This is very obvious, right? Yeah, yeah, yeah.
64
00:11:45.584 --> 00:12:02.664
But, uh, a recent study was actually testing two different models- Mm-hmm... from the same, let's say same, uh, company, two different versions. Um, and what they wanted to see is, how fast does the model solve my query?
65
00:12:02.944 --> 00:12:14.894
Okay. Okay? Mm-hmm. Um, and obviously it, it is based on the complexity level of the query. On the query, of course. So let's say complexity, let's not take like the, the lowest or the highest.
66
00:12:14.924 --> 00:12:28.300
Let's take like seven out of 10. Okay. So a, a little bit above average. And, uh, what they found is that, uh, well, you can take the, the, the cheaper model- Mm-hmm...
67
00:12:28.320 --> 00:12:44.560
but a cheaper model will consume you more tokens to solve a pretty complex query, and it will consume more time. Okay. So there are cases that actually pricier models, they can cost you less.
68
00:12:44.640 --> 00:12:54.220
Are, are more efficient because the queries- Exactly... uh, solve in less time. In the overall. And I think this is what FinOps measures, because it's not only the, okay, how much does it cost?
69
00:12:54.320 --> 00:13:05.240
Oh, this is, uh, more expensive, I'm not going there. Yeah. Wait. Wait just a second. Uh, as we said, there are different use cases. Okay? So these are the things that you need to take to, into account.
70
00:13:05.420 --> 00:13:22.980
I would say measure the ROI of your query output and the tokens. Okay? And this is how to like, uh, just to take into, it into account when choosing a model. Um, a few more things that, uh- Yeah...
71
00:13:22.990 --> 00:13:35.340
that, um, you should consider when choosing a model is that first we all know what tokens are, but not all models are priced by tokens. Yeah.
72
00:13:35.380 --> 00:13:47.080
So if it is like a image
or video generator- Yeah, or audio generation, all these- Yeah... uh, that are different out- different outputs, that is not always language, right? Exactly. Exactly.
73
00:13:47.160 --> 00:13:58.560
So it's different, uh, it's different units. Yeah. It's measured by different units. So just take this into account. And another thing that I think is very painful, it is painful in terms of costs. Mm-hmm.
74
00:13:58.780 --> 00:14:11.560
It is that there is a difference between the price of the input token, going back to tokens, there is a difference between the, the price of the input token rate and the output token rate. Yeah, for sure.
75
00:14:11.870 --> 00:14:25.800
And usually it's like a ratio of like one to three or one to five- Mm-hmm... stuff like that. Mm-hmm. Okay? So this is also something just to take into account. Check the ratio, uh, before choosing the overall model.
76
00:14:26.120 --> 00:14:26.920
Yeah, definitely.
77
00:14:26.960 --> 00:14:40.620
And also like how your use case will be in terms of input and output, because you can have like huge inputs and then super small outputs, and you need to measure like the total, as you mentioned in the other case, like the total query cost.
78
00:14:40.640 --> 00:14:51.100
It's like how many queries you will send. Yeah. Uh, how many interactions you will have, how big are these interactions- Mm-hmm... how much context you need, all these factors. Yeah. That are super interesting.
79
00:14:51.140 --> 00:15:02.380
Yeah, I think there are like a lot of... And I think it depends really on the use cases. It will take us a long time to figure out like all the possible- Yeah... variables. Yeah. We are still like going step by step.
80
00:15:02.740 --> 00:15:06.200
Yeah, yeah, yeah. No, it's super early. [laughs] I think it's a super early practice. Yeah.
81
00:15:06.560 --> 00:15:17.080
And it's, it's still difficult because you have also the caching system that, you know, when, when we come from, from the developer side, like caching is super straightforward.
82
00:15:17.520 --> 00:15:27.980
Uh, you cache a query, and then if something happens or if the database is updated, then they cache a big test based on some policies. In LLMs, it's like caching is
83
00:15:28.900 --> 00:15:37.250
kind of sometimes magical that you, you know, the eviction process is not, uh, you know, you make an update in a database and the eviction happens. It's like- Mm-hmm. Yeah...
84
00:15:37.380 --> 00:15:50.140
and it's a factor that really also affects the pricing of the total. If you are able to cache well, uh, with the LLMs, you can reduce the pricing- Yes... significantly. Yeah, yeah.
85
00:15:50.280 --> 00:16:05.379
I think with caching, um, it is very useful, not for all use cases- Mm-hmm... like you, like you describe, but it can be useful. Um, well, you save more the longer the cache. Yeah, for sure.
86
00:16:05.660 --> 00:16:22.260
The longer the time to leave, so you save more. Uh, but I think like, um, in terms of where can it work, so maybe if there is like a, a repetitive query at a specific- Yeah... time or like, you know, internal use case.
87
00:16:22.620 --> 00:16:35.500
Mm-hmm. Like let's say we have like employees that work, talk to the NHR, uh, agent- Mm-hmm... and they want to ask always the same similar questions. Yeah. So this can be a good use case for caching.
88
00:16:36.180 --> 00:16:44.220
Um, it is definitely something that like should be taken into account. Yeah. Yeah. That's, that's true, and, and it's a good, it's a good one.
89
00:16:44.580 --> 00:16:54.160
Okay, so now that we have, you know, selected the model and we have the criterias that we know how to select it, it's like you'll probably play this model with your own data.
90
00:16:54.190 --> 00:17:06.360
As you mentioned, the healthcare, uh, situation, that's one of the most frequent that we've been seeing, is that data privacy is a problem, so, and data access is a problem. Yeah. And, you know, they,
91
00:17:07.440 --> 00:17:16.880
they need to be very accurate with the data, both in the security side and on the reply side, because you will be playing with lives of people. [laughs] Yeah. So you need to be accurate.
92
00:17:17.260 --> 00:17:28.220
[laughs] So what are the- Very important... you know, the things that you contemplate when you are evaluating the model for such cases, where data is super, needs to be super relevant and super accurate? Yeah.
93
00:17:28.280 --> 00:17:44.130
So when, when we are working, definitely when we are working with, with very sensitive data- Mm-hmm... okay, and we know that we need like to make our agent, um, a s- like a, a specialty- Yeah...
94
00:17:44.130 --> 00:17:56.280
an agent for our use case, a, a very common practice is to do a fine-tuning. Mm-hmm. So in fine-tuning, basically, uh, we train the data with our existing data. Okay?
95
00:17:56.880 --> 00:18:02.720
Uh, so that eventually the model or the agent will be experts. Mm-hmm.
96
00:18:03.280 --> 00:18:19.380
However, there are some risks that, uh, we should take into account, and it is, first, fine-tuning is a very, a high CPU intensive process, but sometimes we need it, but just take- Yeah. It'll-... take that into account.
97
00:18:20.060 --> 00:18:32.494
And second, when we are Creating a new specialist. Mm-hmm. Okay? AI agent, agent specialist. Uh, we might make him too specialist and, like, too narrow- Yeah...
98
00:18:32.544 --> 00:18:48.144
on very specific fields that he was been, he, he has been trained about, and perhaps our data is, won't be relevant and stuff like that. Or we'll be missing. Mm-hmm. Okay? So a great way to monitor it is by checkpoints.
99
00:18:48.464 --> 00:18:55.844
So the, the fine-tuning process is not like zero to one. Yeah. Okay? I see. You have a lot of, like, steps in between.
100
00:18:56.464 --> 00:19:08.424
So, uh, in the different checkpoints, you can decide whether, you know, how much data was, is good enough, how much data
is, I don't know, narrowed down- Yeah... too much. Too much. Yeah.
101
00:19:09.124 --> 00:19:22.264
And you can take each one of these versions, basically, of the checkpoints. Yeah. Okay? Um, another option, but maybe this is less for sensitive data, another option is RAG. Mm-hmm. Okay? Yeah.
102
00:19:22.524 --> 00:19:34.584
Retrieval, retrieval augmented generation. Augmented generation, yeah. [laughs] Yeah. [laughs] RAG. Easy as RAG. Easy as RAG. [laughs] Yeah. Um, so this is where we basically configure, uh...
103
00:19:34.754 --> 00:19:46.694
Like, RAG is, uh, configuring, um, our model to, uh, external resources, external databases that we already have and work with. Yeah.
104
00:19:47.324 --> 00:20:00.524
Um, so, um, it is just added on top of the model, um, and it doesn't narrow it down and stuff like that. Yeah. Yeah. No, I'm familiar with, with both processes, and I think...
105
00:20:00.564 --> 00:20:12.134
Like, I, I, I was in the AI field before it was popular, so 2018, I believe, on the image, uh, sector. Ah. And I recall the fine-tuning process was awful.
106
00:20:12.684 --> 00:20:23.244
And it's, you know, it's like when I, when, you know, you talk to people that are now, you know, in the LLM side and not on the... They are not in the AI field, let's say.
107
00:20:23.304 --> 00:20:32.354
They are not people that are starting in the AI field. And I recall, like, when every of the, all the bomb of the AI part was, like, booming. Mm-hmm, yeah.
108
00:20:32.384 --> 00:20:42.644
And, like, I recall from my time on AI that this was difficult as hell to train, uh, things, to fine-tune things. And RAG, I think it was, like, um,
109
00:20:43.684 --> 00:20:52.394
kind of a middle ground that was super helpful for people that are not, or for systems that are not as easy to fine-tune. Because fine-tune- Right...
110
00:20:52.404 --> 00:21:04.984
as you mentioned, is, like, basically it's retraining a bit, the model. I don't want to say rerunning, right? But it's like, uh- It isn't re-run the data. It doesn't, like, overrun. Sorry, but... It's kind of similar.
111
00:21:05.004 --> 00:21:17.604
It's like you are... It has, like, you have the base model. Yeah. And it's training the internet, and then you retrain it a bit so that it fine-tuned for your specific sector, so for your specific needs.
112
00:21:17.644 --> 00:21:31.824
So for example, what you mentioned about X-ray, it's like you will be trained on a lot of X-ray things, but you will be, before, the base model was trained on a lot of detection, uh, images, could, could be X-ray, uh, animals, et cetera.
113
00:21:32.864 --> 00:21:37.604
The problem of what you mentioned is, like, that requires technical knowledge. Like, to- Mm...
114
00:21:37.614 --> 00:21:46.524
fine-tune something, and what you mentioned with the checkpoints and everything, and I recall the optimization process of a fine-tune, that's not easy.
115
00:21:46.544 --> 00:21:56.244
That's not, uh, setting up a Pinecone,
uh, database, ingest everything- For sure... and vectorize everything almost automatically that we have now, we have the privilege now to have.
116
00:21:56.734 --> 00:22:03.904
So it sounds like this process doesn't only consume CPUs, but on, also a lot of time and human resource. A lot of time to prepare the data. Yeah.
117
00:22:04.224 --> 00:22:19.864
I recall, like, from my time, I recall we were, you know, you have your, let's say you have your Anthropic, your whatever model, and then you refine it in a lot of text or a lot of images that are, I don't know, YouTube thumbnails or, uh, X-rays.
118
00:22:20.104 --> 00:22:31.464
Mm-hmm. And you need to prepare those X-rays. You need to map it correctly. You need to, uh, you know, in-de- have the development. Now everything is AI development.
119
00:22:31.964 --> 00:22:43.424
Uh, you need to have the development pipeline so that you can fine-tune several times. As you mentioned, you have multiple versions. Yes. And you need to ingest that information to the model
120
00:22:44.524 --> 00:22:55.054
in a way that is easy, and that you can retrain it. I recall that from my time, the less time was retraining the model, and the more time was, like, manufacturing data properly,
121
00:22:56.024 --> 00:23:04.754
tag it properly, uh, with the metadata and everything. And it's about the training. The training was like, yeah, leave it alone, like, the eight hours of the night. And the next morning it was out. Yeah.
122
00:23:04.764 --> 00:23:13.384
Especially with the power that we have now. But the data preparation is like a project. No, no, no, it's- It's, it's like a few- It was the-... it sound like a few months project. Yeah, yeah, yeah. It's a months project.
123
00:23:13.424 --> 00:23:26.484
It's the most important thing, because if you- Of course, yeah... fine-tune it wrong- Yeah... it will either be too narrow, um, so it will be like, uh, you know, a coin, uh, drop, and then it's 50/50. Yeah. And it not...
124
00:23:26.544 --> 00:23:36.024
Because normally it's, like, binary and things like that. Like, you know, you have cancer or you don't have cancer. It's like there is no other option. Um, so it's like i- it needs to be very narrow.
125
00:23:36.684 --> 00:23:46.274
And RAG is, like, for most of the cases, I think it's one of the easiest solution and one of the evolutions about the AI, for, for sure. So, you know, it's, it's a different process.
126
00:23:46.384 --> 00:23:59.584
I definitely encourage people to dive into the fine-tuning process and how difficult it is, um, because it definitely gives you a perspective about what's behind- This is the foundation. Yeah.
127
00:23:59.724 --> 00:24:08.024
What's the foundation of- I agree... of AI. I agree. And it's, like, only the, you know, the, the tip of the iceberg. [laughs] Yeah. Because it's like...
128
00:24:08.664 --> 00:24:18.904
And then you start here, when you talk about checkpoints, I recall, you know, having TensorFlow checkpoints, like, seeing the graph with the checkpoints and everything, and that was like... [laughs] Oh, I'm sorry.
129
00:24:18.944 --> 00:24:29.164
I'm sorry for- No, no, no, you br-... for the bad memories. [laughs] Yeah, it's kind of the Vietnam, uh, the Vietnam memories I have until I... [laughs] But it's, it's fun. It's, it was a, a fun time.
130
00:24:29.204 --> 00:24:37.600
And now with that, it makes you appreciate, like-... all the work that people are doing on the AI and how it has reached all this, this step is super interesting.
131
00:24:37.700 --> 00:24:50.530
I wonder how you felt when RAG was, like, introduced to the world. And even for me, like, I, I haven't had a lot of experience, like, on the AI field, but my, let's say, my...
132
00:24:50.700 --> 00:25:05.560
I recall my bachelor's projects and my master project was A- on AI. And it was like that, see? Like, it's super easy now to do so. And it doesn't, uh, it, it was, like, super easy to do.
133
00:25:05.760 --> 00:25:14.730
It's like you ingest a bunch of images and you ingest a bunch of context, and it works. I was like... Well, yeah, definitely AI, like, changed- No, no, forever... everything.
134
00:25:14.820 --> 00:25:27.070
It's, in four years, and I'm not sure how the, the image, but I recall the, there was the, it wasn't LLMs, it was all this generic, this generation. It was more for image. Okay. Like, the- Image generation?...
135
00:25:27.070 --> 00:25:37.320
generate an image generation. It was awful. Awf- awful. [laughs] But now you see, like, the, this generation, it wasn't that good. Um, and now you see things that it's like...
136
00:25:37.340 --> 00:25:49.310
And the image generation field as well, like, the, the, the improvement that we've reached with, you know, people doing ads, uh, you know, Coca-Cola doing ads for- Or, or you, or you don't know what is real or not anymore, right?
137
00:25:49.360 --> 00:25:58.640
No, no, no, for sure. Like [laughs]... You see a lot of AI, you know. I, I recall the Spanish, uh, uh, one of the Spanish sponsors of the, of the World Cup. Ole.
138
00:25:58.720 --> 00:26:11.240
The a- the announcement is AI generated with the players in AI generated. Really? Yeah, yeah, yeah. It's like one guy is pulling, like, rows in the, next to the Big Ben. The other one is, like, cooking with, um,
139
00:26:12.340 --> 00:26:22.860
with a iron. Like, uh- Okay... like they are forging something. So it's like i- it's like a chef- Oh, yeah... forging things. Okay, okay, okay. So it's like AI generated for sure. For sure. For sure, yeah.
140
00:26:22.960 --> 00:26:30.920
[laughs] That is, like, how we reached that point, but, you know, it's, it's very- And, and this is only the beginning, I think. [laughs] No, no, no, for sure.
141
00:26:30.960 --> 00:26:42.430
We've been, like, I don't know, ChatGPT was 2022, if I'm not co- uh, if I'm not wrong, like, the original version, not the GPT-1. So we'll see. Uh, we'll see how these AI projects evolve, and-
[laughs]...
142
00:26:42.430 --> 00:26:53.430
we'll see how, how that goes. Um- Where it goes, yeah. Yeah, definitely. And, you know, moving on to, to the, to the, this whole, uh, let's continue with all the AI projects. So- Yeah...
143
00:26:54.260 --> 00:27:03.800
what are, uh, the, the different components or the different phases that you see you need to go through, uh, a- in an AI project, especially in GCP?
144
00:27:03.900 --> 00:27:14.160
Like, because we have the budgeting, we need to have all these, the, the other things like you mentioned, the data and everything. So what, what do you think are the steps that people need to go through this? Yeah.
145
00:27:14.440 --> 00:27:23.860
So you mentioned budget, which is- Yeah... I think very important, uh- Yeah, especially for us. [laughs] Yeah, especially, yeah, for, for FinOps and AI, very important.
146
00:27:24.520 --> 00:27:31.720
So a really cool feature that, uh, Google Cloud just recently launched in, uh, Google Cloud, uh, Next- Next... uh, event.
147
00:27:32.520 --> 00:27:48.060
Um, it's called, uh, Spend Caps, and it is, um, it is in the Google Cloud billing console under Budgets. Mm-hmm. Um, and basically you can, uh, define a monthly threshold- Mm-hmm...
148
00:27:48.200 --> 00:27:58.980
for your, uh, Vertex AI, uh, Google AI Gemini- Yeah... Gemini Enterprise, and Cloud Run. Uh, you can, you can define a monthly threshold for these services- Mm-hmm...
149
00:27:58.990 --> 00:28:13.350
uh, you know, just to avoid these huge spikes of costs- Yeah... if something goes out of control. Uh, and once the threshold is crossed, you're not only notified about it, it stops. It stops the model, okay?
150
00:28:13.380 --> 00:28:22.360
Like hard stop. Yes, hard stop. Uh, it doesn't, um, terminate what has been generated so far- Okay... so you don't lose any data.
151
00:28:23.020 --> 00:28:34.600
Uh, and if you want to continue generating, like for you it is totally fine that the threshold was crossed, you can disable the, this, uh, spend cap- Okay... and continue working. So you can continue working, okay.
152
00:28:34.660 --> 00:28:47.600
So I think this is a great, like, monitoring tool, um, you know, just to not get into those awkward situations of, like, crazy bills and you don't understand how it happened.
153
00:28:48.180 --> 00:29:03.610
Yeah, that's true, and but, uh, seeing, I was thinking when you were saying it is, like, do you think that will be a, let's say a, um, complex, or let's say a dangerous, uh, tool for production sometimes?
154
00:29:03.680 --> 00:29:12.820
Because some people could enable it, and then- Yes... you can leave a customer like, "I'm not replying." Even though I have to say that I was playing with a tool, and probably they are...
155
00:29:12.840 --> 00:29:16.820
I'm not sure if they are using that, that they could do, that you can, you know, limit the credits of someone,
156
00:29:17.720 --> 00:29:26.760
and you have, I don't know, 100 credits, and then you can put a spend cap on a specific account or on a specific model they have. Definitely. You can, you can define projects, for example. Mm-hmm. So...
157
00:29:26.960 --> 00:29:40.160
Ah, in the project level or even in the- Yes... like, uh, and you can put a, let's say, this user is Gold Tier, and Gold Tier has 1,000 credits, and 1,000 credits are whatever translated in terms of... Yeah.
158
00:29:40.440 --> 00:29:50.530
Well, spend caps for now it is per pro- You can, you can define either for all the billing account- Okay... or per project. Per project, okay. And, well, if you are working right,
159
00:29:51.520 --> 00:29:59.830
uh, each project will be a different environment. Well, it could be. No, but also it's like you can put a, especially in an scaling, like if you have it very well defined- Yeah...
160
00:29:59.860 --> 00:30:10.180
you may have like a project per user, like per account. Yes, definitely. It automatically generate a, a, a project under the same, uh, or- organization is in, is in GCP. Yeah.
161
00:30:10.330 --> 00:30:19.020
And then you can cap that project to like, hey, this is your spend, this is, you know, $1,000. Exactly. Exactly. And then you can measure it to the business value.
162
00:30:19.060 --> 00:30:27.200
Because if you cap it, it's like, hey, you reach to a limit, it will reset next month. Mm-hmm. And I've seen that already working.
163
00:30:28.020 --> 00:30:38.932
Uh, well, not in the, in the GCP, but this is a model that a lot of startups Are using to cap, like, the spend of, you know, token maximum. [laughs] Yeah. But by the way, I don't know if you saw this as well.
164
00:30:39.112 --> 00:30:44.782
In the thresholds, of course, it is not only one static threshold- Mm-hmm... so it's, like, by percentages. Mm-hmm.
165
00:30:44.782 --> 00:30:57.162
So I do set a monthly threshold, let's say $1,000, but notify me when it crosses 50%- Yeah, the budget alerts that-... 75. Mm-hmm. 150%, notify me. Yeah, yeah. Okay? So. No, that's cool. I, I wish my...
166
00:30:57.672 --> 00:31:05.562
You know, I always get the alert when I have, like, five credits left, and I was like- Exactly. [laughs] I don't have... Like, I don't know. If you had, if you had told me before. Yeah, yeah, yeah.
167
00:31:05.612 --> 00:31:12.932
I have to go to the billing process, and it's like, how much have we been spending? I was like, dude, I, I would like to have- Oh, yeah. Oh, my God... some alerts, like, at the half time.
168
00:31:12.972 --> 00:31:23.862
It's like, if I am spending half of my alerts, uh, half of my tokens at the, you know, at the very beginning, [laughs] like, I will stop, because if you tell me at the end, I won't stop. Yeah.
169
00:31:23.862 --> 00:31:34.932
I, as a user, it's like, "Can you advise me more?" Exactly. And disable alerts if I don't want to. No, no, but- And now you bring me to the second point that- Okay... I want to mention, context window. [laughs] Yeah.
170
00:31:34.952 --> 00:31:43.612
[laughs] You
think at the beginning- That's probably one of the- Yeah. [laughs]... of the main drivers of my expense is the, is I always try to keep, like, one long chat, even though it tends to lose it.
171
00:31:43.632 --> 00:31:55.071
But yeah, go ahead with the- So maybe I will ask you, y- do you write more? Where is, where is most of the mess, input or output? Uh, pff.
172
00:31:55.572 --> 00:32:07.612
[laughs] I think probably input is the problem that generates the output issue, and then I don't stop, and therefore- Yeah... I should start a new chat so everything is clean. Yeah.
173
00:32:07.872 --> 00:32:16.692
Uh, so I don't tend to clean in front of- Clean and deterministic, no? Yeah. [laughs] Well, even though- Oh, my God... with AI it's more difficult, but yeah, definitely with less. You know, when you start, you...
174
00:32:17.512 --> 00:32:26.502
Let's say when you are chatting too much, when you are talking too much, for example, it's like you tend to divert and say things that you shouldn't. It's a rabbit hole. Uh, it's kind of the same. It's a rabbit hole.
175
00:32:26.532 --> 00:32:35.182
Like, you need to reset the conversation- Yes... to, like, again. Yeah, okay, okay. I get what you're saying. Yeah. Yeah. So I think context window is definitely one of the- It is. It is.
176
00:32:35.292 --> 00:32:48.872
And, um, I think that one of the best practices, best, best practices which you can implement in context window is, uh, prompt engineering. Yeah. Okay? So as you are more clear
177
00:32:49.872 --> 00:32:58.552
as possible with less tokens, um, you have more tokens left for the output. Mm-hmm. Okay? So prompt engineering is very important.
178
00:32:58.572 --> 00:33:12.631
But just, just to explain, like, context window is the, all the amount of tokens- Mm-hmm... that, uh, we give per session. Mm-hmm. So for both out, input and output. And output, yeah. Okay? So we can control the input.
179
00:33:13.132 --> 00:33:23.092
We can't control, we can't control the output, so- Yeah. Well-... this is what we can, like- Even though with the... What you mentioned is a good point. With the prompt engineering, you can model a bit the output.
180
00:33:23.142 --> 00:33:31.452
Also, yeah. You can, you can play around with it, even though it's not... Like, it's not deterministic. It's not something like you would say this, and it will do this. Yeah. But you can
181
00:33:32.492 --> 00:33:44.732
play around how much, you know, rambling. I recall at, at the beginning, um, I think it was, uh, before all this thinking, uh, phases. You, you- Reasoning? Yeah, all this reasoning.
182
00:33:44.832 --> 00:33:56.692
Like, before it was like you need to make in the prompt engineering. Mm. You put a part that the, the out model output so that it thinked, let's call it that way, so it reason, like it put a lot of text.
183
00:33:57.112 --> 00:34:10.281
And then thanks to that thinking that was basically consuming your output, it will get a better answer at the end.
So you have, like, um, XML system, especially on Claude, that you have, like, thinking. Yeah.
184
00:34:10.292 --> 00:34:22.082
And it was, like, all the rambling thing, and then you have another XML section where it was, like, answer, and then you only had to... You need to do, like, XML trimming, so regex, to get the answer from that. Mm-hmm.
185
00:34:22.132 --> 00:34:32.172
And be- then they implemented thinking, which was like it didn't output that. It, it- It, it consumed, but it didn't output the, the thinking. Okay, yeah, which is, which is good. [laughs] Which is better.
186
00:34:32.872 --> 00:34:41.251
Because I recall for me doing a, a regex for XML trimming, which I don't want to do anymore. [laughs] Definitely.
187
00:34:41.292 --> 00:34:50.742
And other thing that I, I recall from, from this is, like, and something that I, I haven't experimented a lot, so maybe you can help us on that, is batching- Yeah... and asynchronous.
188
00:34:50.832 --> 00:34:59.752
So where do you see it is more helpful? What are the use case that you have seen? You know, what do you think about the catching and- Um-... sorry, the batching and, and asyncon? Yeah.
189
00:34:59.792 --> 00:35:10.981
So, uh, well, by default, we work in, uh, real time- Mm-hmm... with AI. We talk to him, and he gives... He supplies the answers real time, only real, real time. This is very simple. Yeah, yeah.
190
00:35:11.072 --> 00:35:21.092
Um, uh, but there are, uh, use cases where we can use batch, uh, processing or asyn- asynchronous, uh, processing- Mm-hmm...
191
00:35:21.192 --> 00:35:34.632
uh, which means instead of doing it right on the spot every time, okay, let's just compress everything to one, to one batch, and process it once in, let's say, once in a day, okay? Mm-hmm.
192
00:35:34.672 --> 00:35:47.872
So, uh, a, a very common, uh, use case is, um, a invoice, uh, invoice generation, okay? Mm-hmm. Or, like, in general, uh, data that is being collected that can wait.
193
00:35:48.312 --> 00:35:59.972
So we are talking here about lower environments, not productions. So data that is collected that can wait, uh, let's say invoice, invoice generation. Yeah. So I'm collecting, uh, I am,
194
00:36:00.972 --> 00:36:06.022
I am, uh, processing data of a number of customers. Yeah. Okay?
195
00:36:06.052 --> 00:36:18.232
And at night I'm sending all this data to my agent, and over the night it will be working on the- Yeah, it's a batch pro- it's like a daily process that- Exactly... runs every day, but you don't need real time access.
196
00:36:18.252 --> 00:36:28.152
Exactly, exactly. And what we save here is once we batch it, we, like, zip it. Mm-hmm. Okay? So the size is very- Is, is-... way smaller. Yes. Yeah.
197
00:36:28.192 --> 00:36:39.292
So you are already, um, compressing the, the input, and therefore it will reduce the, the cost of the- Exactly... of the overall. The total cost. The total cost. And also you don't depend
198
00:36:40.320 --> 00:36:52.210
Like on availability because we have seen that there are certain windows especially, you know, that there are sometimes outages. We always... Like, we come from the cloud side, so we are very used to outages. Mm-hmm.
199
00:36:52.210 --> 00:37:03.480
And whenever there is a disruption, it's like it could take two hours. I recall the AWS, the, the USD is one that was very, [laughs] uh, very- We're talking about GCP. It happened also in GCP.
200
00:37:03.540 --> 00:37:07.800
[laughs] Yeah, yeah, for sure. [laughs] But I, I have the, the record from this one. Yeah, it happens. Of course.
201
00:37:07.900 --> 00:37:15.319
Uh, it happens to, to all, and I recall that recently it was an Anthropic one, and then OpenAI has had themselves. That's true. So it's like you, you can...
202
00:37:15.360 --> 00:37:22.880
With batching especially, it won't affect your process unless it, you know, it will reach out to the end. It's like it's not immediate.
203
00:37:22.940 --> 00:37:31.670
You can, you know, save that data for later, and then you batch it on the nighttime, and then you'll be able to, to res- Good point. Yeah, yeah, yeah. Yeah. No, I think it's- Yeah...
204
00:37:31.700 --> 00:37:38.200
we need to always think about this, you know, disaster recovery that is very well known in- For sure... [laughs] in cloud because it's...
205
00:37:38.220 --> 00:37:45.490
Even though I don't think it's that problem, but you need to take into account that. Until it happens. Until it happens. [laughs] Until, you know, there are- [laughs]...
206
00:37:45.620 --> 00:37:56.680
there are certain situations where some DNS issue will happen, and then- Yes... what AI models won't work because of a DNS issue. Uh- Exactly, yeah... for sure. [laughs] Because- And for basically batch processing.
207
00:37:56.690 --> 00:38:03.680
[laughs] No, for sure. Like, it's like, you know, we are out of GPUs or whatever, and then someone is, like, change a DNS process.
208
00:38:03.720 --> 00:38:14.140
And if you recall the, the Facebook issue that they had to open the, the lab with the key, uh- Oh, yes... because they're, they need to fix it, like, in the router itself because it was, like, a DNS issue.
209
00:38:14.260 --> 00:38:25.120
And it was like all the outages are DNS issues, and probably AI ones will be a DNS issue. Love it. Um- Eventually you go to the metal, uh- The bare metal. [laughs] Yes, to the bare metal.
210
00:38:25.130 --> 00:38:33.140
[laughs] It's like DNS setting up, and name servers, and everything is about... You know, if it fails, if it's a huge fail, it's probably a DNS.
211
00:38:33.260 --> 00:38:44.060
And I recall, I think the, the DynamoDB issue that caused the AWS failure, I think it was a DNS issue as well. And like at the- That's true... very, very, very back end, it was, like, something related to DNS.
212
00:38:44.420 --> 00:38:54.040
And it's like, yeah, n- like servers, domain names, certificates. If anyone has configured a certificate, they know that they
will fail.
213
00:38:54.880 --> 00:39:04.540
Um, and certificate and all has related to, to DNS, which is like, you know, we are doing all these complex things, and then everything fails in, like, DNS, which we think is like the most simple thing.
214
00:39:04.590 --> 00:39:11.480
On the small things. It's like- Yes... it's not. It needs to work. And, you know, to, to summarize, because we've talked about Vertex, but I know
215
00:39:12.420 --> 00:39:22.920
specifically because if I've used it a bit, it's like there is the enterprise level of, you know, Gemini, of course, and one of the most well-known products of, of, uh, Google, and everything is called Gemini.
216
00:39:23.000 --> 00:39:31.440
So can you briefly talk about, you know, the Gemini enterprise side, how it integrates, how it differs from, from Vertex AI? You know, some tips on that. Yes, definitely.
217
00:39:31.560 --> 00:39:46.140
So, uh, Gemini Enterprise, um, is the platform of Google Cloud where it... Actually, it works with Vertex AI. Mm-hmm. Okay? It connects everything that we talked about, Model Garden, evaluation, everything.
218
00:39:46.780 --> 00:39:54.620
But on top of that, you also get to work with the Google Workspace products. Mm-hmm. So all the docs, presentations, all of this.
219
00:39:55.120 --> 00:40:09.340
So now they even expand even more, um, our, like, working ecosystem, and it can really help a lot of teams in terms of productivity and stuff like that because, um...
220
00:40:09.440 --> 00:40:24.180
Well, first you have, uh, Gemini implemented- Yeah, for sure... in all the workspace. In everything. [laughs] Everywhere. Um, and second, you can also later, uh, after certain reco- uh, configurations of RAG- Mm-hmm...
221
00:40:24.240 --> 00:40:38.100
you can later rely on this data when, uh, working in... with Google Docs, presentations, stuff like that. Yeah. So it is pretty, uh, very, uh, powerful, uh- True... offering. Mm-hmm.
222
00:40:38.360 --> 00:40:54.000
That, like, for a company just to work in one place, and all the steps are done there. Um, and yeah, and Gemini Enterprise is offered in two, two, uh, editions. Yeah. I think it's, like, standard and, uh, the...
223
00:40:54.280 --> 00:41:04.830
Business, uh? Plus. Plus. Yeah. Business and Business and Plus. Something like that. Yeah, yeah, yeah. Standard and Business and Plus. Yeah. Uh, and obviously it, it, uh, depends on the size of the company- Yeah...
224
00:41:04.840 --> 00:41:06.219
and the needs and stuff like that.
225
00:41:06.620 --> 00:41:22.340
No, but it's, it's a good point that you raise about that, that, you know, RAG, like, you know, you always think about, like, it's super complex thing with vectorized things, and then after all they simplify it so much that it's like, yeah, they are ingesting your Google Docs, and then you can talk to it with Gemini, and that's RAG.
226
00:41:22.480 --> 00:41:34.640
[laughs] And, and we are getting to the first point that we talked about. Yeah. Because basically, uh, it is n- AI is not only for developers. Yeah. It's for the HR team. It's for the operations team.
227
00:41:34.680 --> 00:41:45.480
It's for the marketing team, and they work with these docs, Excels, you know, Google Sheets, stuff like that. Everything. So... Like, all about, you know, RAG is that powerful because you can ingest- Yeah...
228
00:41:45.500 --> 00:41:48.440
data super easy, and you can then...
229
00:41:48.680 --> 00:42:01.640
And that's what has enabled the whole, the whole company, like all the company areas, HR, sales, uh, marketing, to be able to generate things and to create things that provide- Exactly... value to, to the company.
230
00:42:01.680 --> 00:42:10.580
It was a very interesting thing that I got from, from the talks in the FinOps, like, it's like 90% of the applications that are AI generated are not from the IT team.
231
00:42:10.960 --> 00:42:18.190
So that gives us, like, the picture of how important this is for, you know, for business and why- Exactly...
232
00:42:18.190 --> 00:42:24.520
is all this token economics and all this controlling the cost because it's like, hey, this is a lot of money- Yes. [laughs]... that we are playing.
233
00:42:24.600 --> 00:42:33.240
Like, uh, like cloud is, is a lot of money, but AI is a lot of money because it's all the company, so, like, a lot of it. Yeah. So I think...
234
00:42:33.280 --> 00:42:44.030
yeah, I think that's a great summary of what, you know, all the capabilities that we can do in GCP and how to actually apply some FinOps for AI there in, in GCP. So Vera, thanks a lot for- Thank you very much, Victor.
235
00:42:44.100 --> 00:42:50.460
Thanks, thanks for being here today. It's a pleasure to have you. Yeah. Uh, we are now going to have some lunch. [laughs] Yes, let's go have some lunch.
236
00:42:50.470 --> 00:42:56.720
[laughs] So thanks everyone for, for listening to us, and see you in the next episode. Bye bye. Thank you very much.
237
00:43:01.020 --> 00:43:14.020
[outro music]