Full Transcript for GCP AI Cost Optimization: Vertex AI Model Selection Guide

the efficiency of the model. Yeah. 29 00:05:32.360 --> 00:05:42.280 And you are phrasing it as of, uh, the needs of the people, you know, the, the business, the needs of- As simple as possible. Yeah. Let's not even talk about tokens, you know? Yeah. [laughs] Just what... 30 00:05:42.320 --> 00:05:52.310 tell me what do you need and who needs it, and this is how we'll solve the problem. Yeah. It's a very human [laughs] response to- Yes. [laughs] Exactly... to that. I don't think AI will, will definitely gen- Right. 31 00:05:52.310 --> 00:06:00.390 [laughs]... get that, that response. The most human thing in AI is this. [laughs] Is, is, is to think about, you know, who is using it and what they are using it for, right? Yeah. 32 00:06:00.400 --> 00:06:10.920 It's all about, you know, the value that you're providing to, to the people and, and of course the business reason- Yeah. Exactly... uh, behind. So, okay, now we have the... we have analyzed the pricing. 33 00:06:11.484 --> 00:06:16.384 And we've analyzed other reasons apart from pricing that we have related to, to business. 34 00:06:16.454 --> 00:06:26.004 So we know that, you know, we have analyzed, uh, how we consider pricing into the model selection, and then the other, uh, factors that influences. Yeah. 35 00:06:26.064 --> 00:06:36.194 And, you know, in GCP, there is the, the tool of Vertex, and, you know, we've been doing... Basically, we've been knowing it for a while on the ML and AI and all the AA parts- Yeah, that's true... 36 00:06:36.194 --> 00:06:46.624 like since 2018, since I was playing- So old. Yeah. Oh my God. [laughs] So, you know, can, can you explain to the audience now what's Vertex AI and- Yeah... and you know, all the, all the content that it has? 37 00:06:46.644 --> 00:06:58.314 Yes, definitely. So Vertex AI, uh, which by the way is now called Agent Platform. Mm. Okay. [laughs] They... Like we- You need to- You need to keep up. [laughs] Yeah, you need to give up the marketing side. Yeah. Yeah. 38 00:06:58.504 --> 00:07:07.883 Um, it is a Google Cloud's, uh, one-stop shop- Mm-hmm... for, uh, AI agents and models. 39 00:07:08.104 --> 00:07:19.334 So in Vertex AI, you can do the whole process, from choosing the model, configuring it, and deploying it to production. Mm-hmm. Basically starting to use it. Okay? Mm-hmm. 40 00:07:19.794 --> 00:07:28.524 So I think the good thing is the convenience obviously, and later you can, uh, combine it with, uh, Gemini Enterprise and stuff like that. Okay. 41 00:07:28.604 --> 00:07:35.444 So it's like, uh, the one place for people to handle all the, all the models and all the... Yes. Well, AM platform, it's a very well- Yeah... named. 42 00:07:35.524 --> 00:07:43.864 And, you know, can you devi- dive into more of the specifics about how, you know, you can play with Vertex AI, some of its components? Yeah. Definitely. 43 00:07:43.924 --> 00:07:56.604 And, um, well, we start the conversation saying that price is a big factor. Mm-hmm. Um, and

it is a big factor because people are still learning how to use it and how to use it right. 44 00:07:56.724 --> 00:08:07.584 So I do think that there is like a certain workflow that we can do, um, just to avoid errors, you know? Mm-hmm. Just to make it smoother. 45 00:08:08.464 --> 00:08:16.764 Um, and the good thing is that in Vertex AI, you have all those steps like lined down. Okay. Like in the same menu, you have all the options. 46 00:08:17.444 --> 00:08:29.544 So the first thing that I would do is going to Model Garden, and Model Garden is a catalog that just displays me all the models that Google Cloud offers. 47 00:08:30.304 --> 00:08:42.544 Um, it offers obviously, you know, the big names like Anthropic, OpenAI, obviously the, uh, Google in-house products of Gemini. Yeah. [laughs] Yeah. [laughs] Obviously. 48 00:08:43.264 --> 00:08:53.344 But also, uh, a cool thing is that, um, there are also more like task or industry related, uh, models. Mm-hmm. 49 00:08:53.444 --> 00:09:09.264 So for example, um, this health industry, health company that I've been working with, um, you could, in the Model Garden, uh, select, uh, health under industry. Okay. Like in the filters. 50 00:09:09.324 --> 00:09:18.704 You can go to filters, in the, the specific industry, industry which you're in, choose health, and like choose models that are way more specific for your use case. 51 00:09:19.344 --> 00:09:34.324 And something even cooler is that like you can even drill down even more per tasks. So for example, under health, you can even have like x-ray AI model. Yeah. Which is really cool, you know? It's like, it's very curious. 52 00:09:34.384 --> 00:09:45.244 Like you're like, "What is this?" [laughs] Yeah. It's very specific to the- Yes... to the point, right? Yeah. And it's amazing because AI obviously goes to a, to many sectors. Mm-hmm. Um, so it can be very useful. 53 00:09:46.124 --> 00:09:50.884 Um, yeah, so this is Model Garden. Okay. And this is the first step. 54 00:09:51.064 --> 00:10:04.824 So we are looking there for, you know, from the variety of the models that we have, and let's say we chose a number of models, and now it's time to, to evaluate it. And this is the name of the page. [laughs] Okay. 55 00:10:04.914 --> 00:10:18.384 [laughs] The name of the section in Vertex AI, it's called Evaluation. And there we are testing... Uh, this is the staging. Okay. Uh, uh, this is the second- The sta- the staging part of the- Yes... of the workload. 56 00:10:18.644 --> 00:10:30.484 Exactly. So we are testing the, the different models. Uh, we, uh, connect our databases that we work with, uh, different prompt engineering, stuff like that. 57 00:10:31.124 --> 00:10:41.214 Uh, and basically there we need to prepare it to, for production- Okay... later. So this is the main- So you are like validating that everything works, that the outputs are correct- Yes... 58 00:10:41.224 --> 00:10:48.454 that, uh- And, and it, that it answers our business needs- Okay...

uh, of what we want to do with these models, obviously. Yeah. 59 00:10:48.484 --> 00:11:04.844 So okay, now that we have the, the model, you know, evaluated, we have followed all the filters in, um, in Vertex AI to, you know, properly select, like what is the criterias that we need to apply in Vertex AI to actually select the model? 60 00:11:05.184 --> 00:11:16.404 So there are different, uh, FinOps best practices- Mm-hmm... that we can take into account when, uh, selecting and working with different models. Mm-hmm. Um, let's start with the more common ones. 61 00:11:16.524 --> 00:11:22.744 So obviously you have like different generations of models. Um, and, uh, the, the... 62 00:11:22.994 --> 00:11:34.964 There is a basic assumption of like, hey, if you t- if you're working with the newest model, from one hand it will give you better performance, but very, very high, uh, but on the other hand, it will give you also very high costs. 63 00:11:35.104 --> 00:11:45.064 Mm-hmm. Okay? And so the first assumption is saying, well, maybe you don't need to use, uh, uh, the, the most, the priceless model. Yeah. This is very obvious, right? Yeah, yeah, yeah. 64 00:11:45.584 --> 00:12:02.664 But, uh, a recent study was actually testing two different models- Mm-hmm... from the same, let's say same, uh, company, two different versions. Um, and what they wanted to see is, how fast does the model solve my query? 65 00:12:02.944 --> 00:12:14.894 Okay. Okay? Mm-hmm. Um, and obviously it, it is based on the complexity level of the query. On the query, of course. So let's say complexity, let's not take like the, the lowest or the highest. 66 00:12:14.924 --> 00:12:28.300 Let's take like seven out of 10. Okay. So a, a little bit above average. And, uh, what they found is that, uh, well, you can take the, the, the cheaper model- Mm-hmm... 67 00:12:28.320 --> 00:12:44.560 but a cheaper model will consume you more tokens to solve a pretty complex query, and it will consume more time. Okay. So there are cases that actually pricier models, they can cost you less. 68 00:12:44.640 --> 00:12:54.220 Are, are more efficient because the queries- Exactly... uh, solve in less time. In the overall. And I think this is what FinOps measures, because it's not only the, okay, how much does it cost? 69 00:12:54.320 --> 00:13:05.240 Oh, this is, uh, more expensive, I'm not going there. Yeah. Wait. Wait just a second. Uh, as we said, there are different use cases. Okay? So these are the things that you need to take to, into account. 70 00:13:05.420 --> 00:13:22.980 I would say measure the ROI of your query output and the tokens. Okay? And this is how to like, uh, just to take into, it into account when choosing a model. Um, a few more things that, uh- Yeah... 71 00:13:22.990 --> 00:13:35.340 that, um, you should consider when choosing a model is that first we all know what tokens are, but not all models are priced by tokens. Yeah. 72 00:13:35.380 --> 00:13:47.080 So if it is like a image

or video generator- Yeah, or audio generation, all these- Yeah... uh, that are different out- different outputs, that is not always language, right? Exactly. Exactly. 73 00:13:47.160 --> 00:13:58.560 So it's different, uh, it's different units. Yeah. It's measured by different units. So just take this into account. And another thing that I think is very painful, it is painful in terms of costs. Mm-hmm. 74 00:13:58.780 --> 00:14:11.560 It is that there is a difference between the price of the input token, going back to tokens, there is a difference between the, the price of the input token rate and the output token rate. Yeah, for sure. 75 00:14:11.870 --> 00:14:25.800 And usually it's like a ratio of like one to three or one to five- Mm-hmm... stuff like that. Mm-hmm. Okay? So this is also something just to take into account. Check the ratio, uh, before choosing the overall model. 76 00:14:26.120 --> 00:14:26.920 Yeah, definitely. 77 00:14:26.960 --> 00:14:40.620 And also like how your use case will be in terms of input and output, because you can have like huge inputs and then super small outputs, and you need to measure like the total, as you mentioned in the other case, like the total query cost. 78 00:14:40.640 --> 00:14:51.100 It's like how many queries you will send. Yeah. Uh, how many interactions you will have, how big are these interactions- Mm-hmm... how much context you need, all these factors. Yeah. That are super interesting. 79 00:14:51.140 --> 00:15:02.380 Yeah, I think there are like a lot of... And I think it depends really on the use cases. It will take us a long time to figure out like all the possible- Yeah... variables. Yeah. We are still like going step by step. 80 00:15:02.740 --> 00:15:06.200 Yeah, yeah, yeah. No, it's super early. [laughs] I think it's a super early practice. Yeah. 81 00:15:06.560 --> 00:15:17.080 And it's, it's still difficult because you have also the caching system that, you know, when, when we come from, from the developer side, like caching is super straightforward. 82 00:15:17.520 --> 00:15:27.980 Uh, you cache a query, and then if something happens or if the database is updated, then they cache a big test based on some policies. In LLMs, it's like caching is 83 00:15:28.900 --> 00:15:37.250 kind of sometimes magical that you, you know, the eviction process is not, uh, you know, you make an update in a database and the eviction happens. It's like- Mm-hmm. Yeah... 84 00:15:37.380 --> 00:15:50.140 and it's a factor that really also affects the pricing of the total. If you are able to cache well, uh, with the LLMs, you can reduce the pricing- Yes... significantly. Yeah, yeah. 85 00:15:50.280 --> 00:16:05.379 I think with caching, um, it is very useful, not for all use cases- Mm-hmm... like you, like you describe, but it can be useful. Um, well, you save more the longer the cache. Yeah, for sure. 86 00:16:05.660 --> 00:16:22.260 The longer the time to leave, so you save more. Uh, but I think like, um, in terms of where can it work, so maybe if there is like a, a repetitive query at a specific- Yeah... time or like, you know, internal use case. 87 00:16:22.620 --> 00:16:35.500 Mm-hmm. Like let's say we have like employees that work, talk to the NHR, uh, agent- Mm-hmm... and they want to ask always the same similar questions. Yeah. So this can be a good use case for caching. 88 00:16:36.180 --> 00:16:44.220 Um, it is definitely something that like should be taken into account. Yeah. Yeah. That's, that's true, and, and it's a good, it's a good one. 89 00:16:44.580 --> 00:16:54.160 Okay, so now that we have, you know, selected the model and we have the criterias that we know how to select it, it's like you'll probably play this model with your own data. 90 00:16:54.190 --> 00:17:06.360 As you mentioned, the healthcare, uh, situation, that's one of the most frequent that we've been seeing, is that data privacy is a problem, so, and data access is a problem. Yeah. And, you know, they, 91 00:17:07.440 --> 00:17:16.880 they need to be very accurate with the data, both in the security side and on the reply side, because you will be playing with lives of people. [laughs] Yeah. So you need to be accurate. 92 00:17:17.260 --> 00:17:28.220 [laughs] So what are the- Very important... you know, the things that you contemplate when you are evaluating the model for such cases, where data is super, needs to be super relevant and super accurate? Yeah. 93 00:17:28.280 --> 00:17:44.130 So when, when we are working, definitely when we are working with, with very sensitive data- Mm-hmm... okay, and we know that we need like to make our agent, um, a s- like a, a specialty- Yeah... 94 00:17:44.130 --> 00:17:56.280 an agent for our use case, a, a very common practice is to do a fine-tuning. Mm-hmm. So in fine-tuning, basically, uh, we train the data with our existing data. Okay? 95 00:17:56.880 --> 00:18:02.720 Uh, so that eventually the model or the agent will be experts. Mm-hmm. 96 00:18:03.280 --> 00:18:19.380 However, there are some risks that, uh, we should take into account, and it is, first, fine-tuning is a very, a high CPU intensive process, but sometimes we need it, but just take- Yeah. It'll-... take that into account. 97 00:18:20.060 --> 00:18:32.494 And second, when we are Creating a new specialist. Mm-hmm. Okay? AI agent, agent specialist. Uh, we might make him too specialist and, like, too narrow- Yeah... 98 00:18:32.544 --> 00:18:48.144 on very specific fields that he was been, he, he has been trained about, and perhaps our data is, won't be relevant and stuff like that. Or we'll be missing. Mm-hmm. Okay? So a great way to monitor it is by checkpoints. 99 00:18:48.464 --> 00:18:55.844 So the, the fine-tuning process is not like zero to one. Yeah. Okay? I see. You have a lot of, like, steps in between. 100 00:18:56.464 --> 00:19:08.424 So, uh, in the different checkpoints, you can decide whether, you know, how much data was, is good enough, how much data

is, I don't know, narrowed down- Yeah... too much. Too much. Yeah. 101 00:19:09.124 --> 00:19:22.264 And you can take each one of these versions, basically, of the checkpoints. Yeah. Okay? Um, another option, but maybe this is less for sensitive data, another option is RAG. Mm-hmm. Okay? Yeah. 102 00:19:22.524 --> 00:19:34.584 Retrieval, retrieval augmented generation. Augmented generation, yeah. [laughs] Yeah. [laughs] RAG. Easy as RAG. Easy as RAG. [laughs] Yeah. Um, so this is where we basically configure, uh... 103 00:19:34.754 --> 00:19:46.694 Like, RAG is, uh, configuring, um, our model to, uh, external resources, external databases that we already have and work with. Yeah. 104 00:19:47.324 --> 00:20:00.524 Um, so, um, it is just added on top of the model, um, and it doesn't narrow it down and stuff like that. Yeah. Yeah. No, I'm familiar with, with both processes, and I think... 105 00:20:00.564 --> 00:20:12.134 Like, I, I, I was in the AI field before it was popular, so 2018, I believe, on the image, uh, sector. Ah. And I recall the fine-tuning process was awful. 106 00:20:12.684 --> 00:20:23.244 And it's, you know, it's like when I, when, you know, you talk to people that are now, you know, in the LLM side and not on the... They are not in the AI field, let's say. 107 00:20:23.304 --> 00:20:32.354 They are not people that are starting in the AI field. And I recall, like, when every of the, all the bomb of the AI part was, like, booming. Mm-hmm, yeah. 108 00:20:32.384 --> 00:20:42.644 And, like, I recall from my time on AI that this was difficult as hell to train, uh, things, to fine-tune things. And RAG, I think it was, like, um, 109 00:20:43.684 --> 00:20:52.394 kind of a middle ground that was super helpful for people that are not, or for systems that are not as easy to fine-tune. Because fine-tune- Right... 110 00:20:52.404 --> 00:21:04.984 as you mentioned, is, like, basically it's retraining a bit, the model. I don't want to say rerunning, right? But it's like, uh- It isn't re-run the data. It doesn't, like, overrun. Sorry, but... It's kind of similar. 111 00:21:05.004 --> 00:21:17.604 It's like you are... It has, like, you have the base model. Yeah. And it's training the internet, and then you retrain it a bit so that it fine-tuned for your specific sector, so for your specific needs. 112 00:21:17.644 --> 00:21:31.824 So for example, what you mentioned about X-ray, it's like you will be trained on a lot of X-ray things, but you will be, before, the base model was trained on a lot of detection, uh, images, could, could be X-ray, uh, animals, et cetera. 113 00:21:32.864 --> 00:21:37.604 The problem of what you mentioned is, like, that requires technical knowledge. Like, to- Mm... 114 00:21:37.614 --> 00:21:46.524 fine-tune something, and what you mentioned with the checkpoints and everything, and I recall the optimization process of a fine-tune, that's not easy. 115 00:21:46.544 --> 00:21:56.244 That's not, uh, setting up a Pinecone,

uh, database, ingest everything- For sure... and vectorize everything almost automatically that we have now, we have the privilege now to have. 116 00:21:56.734 --> 00:22:03.904 So it sounds like this process doesn't only consume CPUs, but on, also a lot of time and human resource. A lot of time to prepare the data. Yeah. 117 00:22:04.224 --> 00:22:19.864 I recall, like, from my time, I recall we were, you know, you have your, let's say you have your Anthropic, your whatever model, and then you refine it in a lot of text or a lot of images that are, I don't know, YouTube thumbnails or, uh, X-rays. 118 00:22:20.104 --> 00:22:31.464 Mm-hmm. And you need to prepare those X-rays. You need to map it correctly. You need to, uh, you know, in-de- have the development. Now everything is AI development. 119 00:22:31.964 --> 00:22:43.424 Uh, you need to have the development pipeline so that you can fine-tune several times. As you mentioned, you have multiple versions. Yes. And you need to ingest that information to the model 120 00:22:44.524 --> 00:22:55.054 in a way that is easy, and that you can retrain it. I recall that from my time, the less time was retraining the model, and the more time was, like, manufacturing data properly, 121 00:22:56.024 --> 00:23:04.754 tag it properly, uh, with the metadata and everything. And it's about the training. The training was like, yeah, leave it alone, like, the eight hours of the night. And the next morning it was out. Yeah. 122 00:23:04.764 --> 00:23:13.384 Especially with the power that we have now. But the data preparation is like a project. No, no, no, it's- It's, it's like a few- It was the-... it sound like a few months project. Yeah, yeah, yeah. It's a months project. 123 00:23:13.424 --> 00:23:26.484 It's the most important thing, because if you- Of course, yeah... fine-tune it wrong- Yeah... it will either be too narrow, um, so it will be like, uh, you know, a coin, uh, drop, and then it's 50/50. Yeah. And it not... 124 00:23:26.544 --> 00:23:36.024 Because normally it's, like, binary and things like that. Like, you know, you have cancer or you don't have cancer. It's like there is no other option. Um, so it's like i- it needs to be very narrow. 125 00:23:36.684 --> 00:23:46.274 And RAG is, like, for most of the cases, I think it's one of the easiest solution and one of the evolutions about the AI, for, for sure. So, you know, it's, it's a different process. 126 00:23:46.384 --> 00:23:59.584 I definitely encourage people to dive into the fine-tuning process and how difficult it is, um, because it definitely gives you a perspective about what's behind- This is the foundation. Yeah. 127 00:23:59.724 --> 00:24:08.024 What's the foundation of- I agree... of AI. I agree. And it's, like, only the, you know, the, the tip of the iceberg. [laughs] Yeah. Because it's like... 128 00:24:08.664 --> 00:24:18.904 And then you start here, when you talk about checkpoints, I recall, you know, having TensorFlow checkpoints, like, seeing the graph with the checkpoints and everything, and that was like... [laughs] Oh, I'm sorry. 129 00:24:18.944 --> 00:24:29.164 I'm sorry for- No, no, no, you br-... for the bad memories. [laughs] Yeah, it's kind of the Vietnam, uh, the Vietnam memories I have until I... [laughs] But it's, it's fun. It's, it was a, a fun time. 130 00:24:29.204 --> 00:24:37.600 And now with that, it makes you appreciate, like-... all the work that people are doing on the AI and how it has reached all this, this step is super interesting. 131 00:24:37.700 --> 00:24:50.530 I wonder how you felt when RAG was, like, introduced to the world. And even for me, like, I, I haven't had a lot of experience, like, on the AI field, but my, let's say, my... 132 00:24:50.700 --> 00:25:05.560 I recall my bachelor's projects and my master project was A- on AI. And it was like that, see? Like, it's super easy now to do so. And it doesn't, uh, it, it was, like, super easy to do. 133 00:25:05.760 --> 00:25:14.730 It's like you ingest a bunch of images and you ingest a bunch of context, and it works. I was like... Well, yeah, definitely AI, like, changed- No, no, forever... everything. 134 00:25:14.820 --> 00:25:27.070 It's, in four years, and I'm not sure how the, the image, but I recall the, there was the, it wasn't LLMs, it was all this generic, this generation. It was more for image. Okay. Like, the- Image generation?... 135 00:25:27.070 --> 00:25:37.320 generate an image generation. It was awful. Awf- awful. [laughs] But now you see, like, the, this generation, it wasn't that good. Um, and now you see things that it's like... 136 00:25:37.340 --> 00:25:49.310 And the image generation field as well, like, the, the, the improvement that we've reached with, you know, people doing ads, uh, you know, Coca-Cola doing ads for- Or, or you, or you don't know what is real or not anymore, right? 137 00:25:49.360 --> 00:25:58.640 No, no, no, for sure. Like [laughs]... You see a lot of AI, you know. I, I recall the Spanish, uh, uh, one of the Spanish sponsors of the, of the World Cup. Ole. 138 00:25:58.720 --> 00:26:11.240 The a- the announcement is AI generated with the players in AI generated. Really? Yeah, yeah, yeah. It's like one guy is pulling, like, rows in the, next to the Big Ben. The other one is, like, cooking with, um, 139 00:26:12.340 --> 00:26:22.860 with a iron. Like, uh- Okay... like they are forging something. So it's like i- it's like a chef- Oh, yeah... forging things. Okay, okay, okay. So it's like AI generated for sure. For sure. For sure, yeah. 140 00:26:22.960 --> 00:26:30.920 [laughs] That is, like, how we reached that point, but, you know, it's, it's very- And, and this is only the beginning, I think. [laughs] No, no, no, for sure. 141 00:26:30.960 --> 00:26:42.430 We've been, like, I don't know, ChatGPT was 2022, if I'm not co- uh, if I'm not wrong, like, the original version, not the GPT-1. So we'll see. Uh, we'll see how these AI projects evolve, and-

[laughs]... 142 00:26:42.430 --> 00:26:53.430 we'll see how, how that goes. Um- Where it goes, yeah. Yeah, definitely. And, you know, moving on to, to the, to the, this whole, uh, let's continue with all the AI projects. So- Yeah... 143 00:26:54.260 --> 00:27:03.800 what are, uh, the, the different components or the different phases that you see you need to go through, uh, a- in an AI project, especially in GCP? 144 00:27:03.900 --> 00:27:14.160 Like, because we have the budgeting, we need to have all these, the, the other things like you mentioned, the data and everything. So what, what do you think are the steps that people need to go through this? Yeah. 145 00:27:14.440 --> 00:27:23.860 So you mentioned budget, which is- Yeah... I think very important, uh- Yeah, especially for us. [laughs] Yeah, especially, yeah, for, for FinOps and AI, very important. 146 00:27:24.520 --> 00:27:31.720 So a really cool feature that, uh, Google Cloud just recently launched in, uh, Google Cloud, uh, Next- Next... uh, event. 147 00:27:32.520 --> 00:27:48.060 Um, it's called, uh, Spend Caps, and it is, um, it is in the Google Cloud billing console under Budgets. Mm-hmm. Um, and basically you can, uh, define a monthly threshold- Mm-hmm... 148 00:27:48.200 --> 00:27:58.980 for your, uh, Vertex AI, uh, Google AI Gemini- Yeah... Gemini Enterprise, and Cloud Run. Uh, you can, you can define a monthly threshold for these services- Mm-hmm... 149 00:27:58.990 --> 00:28:13.350 uh, you know, just to avoid these huge spikes of costs- Yeah... if something goes out of control. Uh, and once the threshold is crossed, you're not only notified about it, it stops. It stops the model, okay? 150 00:28:13.380 --> 00:28:22.360 Like hard stop. Yes, hard stop. Uh, it doesn't, um, terminate what has been generated so far- Okay... so you don't lose any data. 151 00:28:23.020 --> 00:28:34.600 Uh, and if you want to continue generating, like for you it is totally fine that the threshold was crossed, you can disable the, this, uh, spend cap- Okay... and continue working. So you can continue working, okay. 152 00:28:34.660 --> 00:28:47.600 So I think this is a great, like, monitoring tool, um, you know, just to not get into those awkward situations of, like, crazy bills and you don't understand how it happened. 153 00:28:48.180 --> 00:29:03.610 Yeah, that's true, and but, uh, seeing, I was thinking when you were saying it is, like, do you think that will be a, let's say a, um, complex, or let's say a dangerous, uh, tool for production sometimes? 154 00:29:03.680 --> 00:29:12.820 Because some people could enable it, and then- Yes... you can leave a customer like, "I'm not replying." Even though I have to say that I was playing with a tool, and probably they are... 155 00:29:12.840 --> 00:29:16.820 I'm not sure if they are using that, that they could do, that you can, you know, limit the credits of someone, 156 00:29:17.720 --> 00:29:26.760 and you have, I don't know, 100 credits, and then you can put a spend cap on a specific account or on a specific model they have. Definitely. You can, you can define projects, for example. Mm-hmm. So... 157 00:29:26.960 --> 00:29:40.160 Ah, in the project level or even in the- Yes... like, uh, and you can put a, let's say, this user is Gold Tier, and Gold Tier has 1,000 credits, and 1,000 credits are whatever translated in terms of... Yeah. 158 00:29:40.440 --> 00:29:50.530 Well, spend caps for now it is per pro- You can, you can define either for all the billing account- Okay... or per project. Per project, okay. And, well, if you are working right, 159 00:29:51.520 --> 00:29:59.830 uh, each project will be a different environment. Well, it could be. No, but also it's like you can put a, especially in an scaling, like if you have it very well defined- Yeah... 160 00:29:59.860 --> 00:30:10.180 you may have like a project per user, like per account. Yes, definitely. It automatically generate a, a, a project under the same, uh, or- organization is in, is in GCP. Yeah. 161 00:30:10.330 --> 00:30:19.020 And then you can cap that project to like, hey, this is your spend, this is, you know, $1,000. Exactly. Exactly. And then you can measure it to the business value. 162 00:30:19.060 --> 00:30:27.200 Because if you cap it, it's like, hey, you reach to a limit, it will reset next month. Mm-hmm. And I've seen that already working. 163 00:30:28.020 --> 00:30:38.932 Uh, well, not in the, in the GCP, but this is a model that a lot of startups Are using to cap, like, the spend of, you know, token maximum. [laughs] Yeah. But by the way, I don't know if you saw this as well. 164 00:30:39.112 --> 00:30:44.782 In the thresholds, of course, it is not only one static threshold- Mm-hmm... so it's, like, by percentages. Mm-hmm. 165 00:30:44.782 --> 00:30:57.162 So I do set a monthly threshold, let's say $1,000, but notify me when it crosses 50%- Yeah, the budget alerts that-... 75. Mm-hmm. 150%, notify me. Yeah, yeah. Okay? So. No, that's cool. I, I wish my... 166 00:30:57.672 --> 00:31:05.562 You know, I always get the alert when I have, like, five credits left, and I was like- Exactly. [laughs] I don't have... Like, I don't know. If you had, if you had told me before. Yeah, yeah, yeah. 167 00:31:05.612 --> 00:31:12.932 I have to go to the billing process, and it's like, how much have we been spending? I was like, dude, I, I would like to have- Oh, yeah. Oh, my God... some alerts, like, at the half time. 168 00:31:12.972 --> 00:31:23.862 It's like, if I am spending half of my alerts, uh, half of my tokens at the, you know, at the very beginning, [laughs] like, I will stop, because if you tell me at the end, I won't stop. Yeah. 169 00:31:23.862 --> 00:31:34.932 I, as a user, it's like, "Can you advise me more?" Exactly. And disable alerts if I don't want to. No, no, but- And now you bring me to the second point that- Okay... I want to mention, context window. [laughs] Yeah. 170 00:31:34.952 --> 00:31:43.612 [laughs] You

think at the beginning- That's probably one of the- Yeah. [laughs]... of the main drivers of my expense is the, is I always try to keep, like, one long chat, even though it tends to lose it. 171 00:31:43.632 --> 00:31:55.071 But yeah, go ahead with the- So maybe I will ask you, y- do you write more? Where is, where is most of the mess, input or output? Uh, pff. 172 00:31:55.572 --> 00:32:07.612 [laughs] I think probably input is the problem that generates the output issue, and then I don't stop, and therefore- Yeah... I should start a new chat so everything is clean. Yeah. 173 00:32:07.872 --> 00:32:16.692 Uh, so I don't tend to clean in front of- Clean and deterministic, no? Yeah. [laughs] Well, even though- Oh, my God... with AI it's more difficult, but yeah, definitely with less. You know, when you start, you... 174 00:32:17.512 --> 00:32:26.502 Let's say when you are chatting too much, when you are talking too much, for example, it's like you tend to divert and say things that you shouldn't. It's a rabbit hole. Uh, it's kind of the same. It's a rabbit hole. 175 00:32:26.532 --> 00:32:35.182 Like, you need to reset the conversation- Yes... to, like, again. Yeah, okay, okay. I get what you're saying. Yeah. Yeah. So I think context window is definitely one of the- It is. It is. 176 00:32:35.292 --> 00:32:48.872 And, um, I think that one of the best practices, best, best practices which you can implement in context window is, uh, prompt engineering. Yeah. Okay? So as you are more clear 177 00:32:49.872 --> 00:32:58.552 as possible with less tokens, um, you have more tokens left for the output. Mm-hmm. Okay? So prompt engineering is very important. 178 00:32:58.572 --> 00:33:12.631 But just, just to explain, like, context window is the, all the amount of tokens- Mm-hmm... that, uh, we give per session. Mm-hmm. So for both out, input and output. And output, yeah. Okay? So we can control the input. 179 00:33:13.132 --> 00:33:23.092 We can't control, we can't control the output, so- Yeah. Well-... this is what we can, like- Even though with the... What you mentioned is a good point. With the prompt engineering, you can model a bit the output. 180 00:33:23.142 --> 00:33:31.452 Also, yeah. You can, you can play around with it, even though it's not... Like, it's not deterministic. It's not something like you would say this, and it will do this. Yeah. But you can 181 00:33:32.492 --> 00:33:44.732 play around how much, you know, rambling. I recall at, at the beginning, um, I think it was, uh, before all this thinking, uh, phases. You, you- Reasoning? Yeah, all this reasoning. 182 00:33:44.832 --> 00:33:56.692 Like, before it was like you need to make in the prompt engineering. Mm. You put a part that the, the out model output so that it thinked, let's call it that way, so it reason, like it put a lot of text. 183 00:33:57.112 --> 00:34:10.281 And then thanks to that thinking that was basically consuming your output, it will get a better answer at the end.

So you have, like, um, XML system, especially on Claude, that you have, like, thinking. Yeah. 184 00:34:10.292 --> 00:34:22.082 And it was, like, all the rambling thing, and then you have another XML section where it was, like, answer, and then you only had to... You need to do, like, XML trimming, so regex, to get the answer from that. Mm-hmm. 185 00:34:22.132 --> 00:34:32.172 And be- then they implemented thinking, which was like it didn't output that. It, it- It, it consumed, but it didn't output the, the thinking. Okay, yeah, which is, which is good. [laughs] Which is better. 186 00:34:32.872 --> 00:34:41.251 Because I recall for me doing a, a regex for XML trimming, which I don't want to do anymore. [laughs] Definitely. 187 00:34:41.292 --> 00:34:50.742 And other thing that I, I recall from, from this is, like, and something that I, I haven't experimented a lot, so maybe you can help us on that, is batching- Yeah... and asynchronous. 188 00:34:50.832 --> 00:34:59.752 So where do you see it is more helpful? What are the use case that you have seen? You know, what do you think about the catching and- Um-... sorry, the batching and, and asyncon? Yeah. 189 00:34:59.792 --> 00:35:10.981 So, uh, well, by default, we work in, uh, real time- Mm-hmm... with AI. We talk to him, and he gives... He supplies the answers real time, only real, real time. This is very simple. Yeah, yeah. 190 00:35:11.072 --> 00:35:21.092 Um, uh, but there are, uh, use cases where we can use batch, uh, processing or asyn- asynchronous, uh, processing- Mm-hmm... 191 00:35:21.192 --> 00:35:34.632 uh, which means instead of doing it right on the spot every time, okay, let's just compress everything to one, to one batch, and process it once in, let's say, once in a day, okay? Mm-hmm. 192 00:35:34.672 --> 00:35:47.872 So, uh, a, a very common, uh, use case is, um, a invoice, uh, invoice generation, okay? Mm-hmm. Or, like, in general, uh, data that is being collected that can wait. 193 00:35:48.312 --> 00:35:59.972 So we are talking here about lower environments, not productions. So data that is collected that can wait, uh, let's say invoice, invoice generation. Yeah. So I'm collecting, uh, I am, 194 00:36:00.972 --> 00:36:06.022 I am, uh, processing data of a number of customers. Yeah. Okay? 195 00:36:06.052 --> 00:36:18.232 And at night I'm sending all this data to my agent, and over the night it will be working on the- Yeah, it's a batch pro- it's like a daily process that- Exactly... runs every day, but you don't need real time access. 196 00:36:18.252 --> 00:36:28.152 Exactly, exactly. And what we save here is once we batch it, we, like, zip it. Mm-hmm. Okay? So the size is very- Is, is-... way smaller. Yes. Yeah. 197 00:36:28.192 --> 00:36:39.292 So you are already, um, compressing the, the input, and therefore it will reduce the, the cost of the- Exactly... of the overall. The total cost. The total cost. And also you don't depend 198 00:36:40.320 --> 00:36:52.210 Like on availability because we have seen that there are certain windows especially, you know, that there are sometimes outages. We always... Like, we come from the cloud side, so we are very used to outages. Mm-hmm. 199 00:36:52.210 --> 00:37:03.480 And whenever there is a disruption, it's like it could take two hours. I recall the AWS, the, the USD is one that was very, [laughs] uh, very- We're talking about GCP. It happened also in GCP. 200 00:37:03.540 --> 00:37:07.800 [laughs] Yeah, yeah, for sure. [laughs] But I, I have the, the record from this one. Yeah, it happens. Of course. 201 00:37:07.900 --> 00:37:15.319 Uh, it happens to, to all, and I recall that recently it was an Anthropic one, and then OpenAI has had themselves. That's true. So it's like you, you can... 202 00:37:15.360 --> 00:37:22.880 With batching especially, it won't affect your process unless it, you know, it will reach out to the end. It's like it's not immediate. 203 00:37:22.940 --> 00:37:31.670 You can, you know, save that data for later, and then you batch it on the nighttime, and then you'll be able to, to res- Good point. Yeah, yeah, yeah. Yeah. No, I think it's- Yeah... 204 00:37:31.700 --> 00:37:38.200 we need to always think about this, you know, disaster recovery that is very well known in- For sure... [laughs] in cloud because it's... 205 00:37:38.220 --> 00:37:45.490 Even though I don't think it's that problem, but you need to take into account that. Until it happens. Until it happens. [laughs] Until, you know, there are- [laughs]... 206 00:37:45.620 --> 00:37:56.680 there are certain situations where some DNS issue will happen, and then- Yes... what AI models won't work because of a DNS issue. Uh- Exactly, yeah... for sure. [laughs] Because- And for basically batch processing. 207 00:37:56.690 --> 00:38:03.680 [laughs] No, for sure. Like, it's like, you know, we are out of GPUs or whatever, and then someone is, like, change a DNS process. 208 00:38:03.720 --> 00:38:14.140 And if you recall the, the Facebook issue that they had to open the, the lab with the key, uh- Oh, yes... because they're, they need to fix it, like, in the router itself because it was, like, a DNS issue. 209 00:38:14.260 --> 00:38:25.120 And it was like all the outages are DNS issues, and probably AI ones will be a DNS issue. Love it. Um- Eventually you go to the metal, uh- The bare metal. [laughs] Yes, to the bare metal. 210 00:38:25.130 --> 00:38:33.140 [laughs] It's like DNS setting up, and name servers, and everything is about... You know, if it fails, if it's a huge fail, it's probably a DNS. 211 00:38:33.260 --> 00:38:44.060 And I recall, I think the, the DynamoDB issue that caused the AWS failure, I think it was a DNS issue as well. And like at the- That's true... very, very, very back end, it was, like, something related to DNS. 212 00:38:44.420 --> 00:38:54.040 And it's like, yeah, n- like servers, domain names, certificates. If anyone has configured a certificate, they know that they

will fail. 213 00:38:54.880 --> 00:39:04.540 Um, and certificate and all has related to, to DNS, which is like, you know, we are doing all these complex things, and then everything fails in, like, DNS, which we think is like the most simple thing. 214 00:39:04.590 --> 00:39:11.480 On the small things. It's like- Yes... it's not. It needs to work. And, you know, to, to summarize, because we've talked about Vertex, but I know 215 00:39:12.420 --> 00:39:22.920 specifically because if I've used it a bit, it's like there is the enterprise level of, you know, Gemini, of course, and one of the most well-known products of, of, uh, Google, and everything is called Gemini. 216 00:39:23.000 --> 00:39:31.440 So can you briefly talk about, you know, the Gemini enterprise side, how it integrates, how it differs from, from Vertex AI? You know, some tips on that. Yes, definitely. 217 00:39:31.560 --> 00:39:46.140 So, uh, Gemini Enterprise, um, is the platform of Google Cloud where it... Actually, it works with Vertex AI. Mm-hmm. Okay? It connects everything that we talked about, Model Garden, evaluation, everything. 218 00:39:46.780 --> 00:39:54.620 But on top of that, you also get to work with the Google Workspace products. Mm-hmm. So all the docs, presentations, all of this. 219 00:39:55.120 --> 00:40:09.340 So now they even expand even more, um, our, like, working ecosystem, and it can really help a lot of teams in terms of productivity and stuff like that because, um... 220 00:40:09.440 --> 00:40:24.180 Well, first you have, uh, Gemini implemented- Yeah, for sure... in all the workspace. In everything. [laughs] Everywhere. Um, and second, you can also later, uh, after certain reco- uh, configurations of RAG- Mm-hmm... 221 00:40:24.240 --> 00:40:38.100 you can later rely on this data when, uh, working in... with Google Docs, presentations, stuff like that. Yeah. So it is pretty, uh, very, uh, powerful, uh- True... offering. Mm-hmm. 222 00:40:38.360 --> 00:40:54.000 That, like, for a company just to work in one place, and all the steps are done there. Um, and yeah, and Gemini Enterprise is offered in two, two, uh, editions. Yeah. I think it's, like, standard and, uh, the... 223 00:40:54.280 --> 00:41:04.830 Business, uh? Plus. Plus. Yeah. Business and Business and Plus. Something like that. Yeah, yeah, yeah. Standard and Business and Plus. Yeah. Uh, and obviously it, it, uh, depends on the size of the company- Yeah... 224 00:41:04.840 --> 00:41:06.219 and the needs and stuff like that. 225 00:41:06.620 --> 00:41:22.340 No, but it's, it's a good point that you raise about that, that, you know, RAG, like, you know, you always think about, like, it's super complex thing with vectorized things, and then after all they simplify it so much that it's like, yeah, they are ingesting your Google Docs, and then you can talk to it with Gemini, and that's RAG. 226 00:41:22.480 --> 00:41:34.640 [laughs] And, and we are getting to the first point that we talked about. Yeah. Because basically, uh, it is n- AI is not only for developers. Yeah. It's for the HR team. It's for the operations team. 227 00:41:34.680 --> 00:41:45.480 It's for the marketing team, and they work with these docs, Excels, you know, Google Sheets, stuff like that. Everything. So... Like, all about, you know, RAG is that powerful because you can ingest- Yeah... 228 00:41:45.500 --> 00:41:48.440 data super easy, and you can then... 229 00:41:48.680 --> 00:42:01.640 And that's what has enabled the whole, the whole company, like all the company areas, HR, sales, uh, marketing, to be able to generate things and to create things that provide- Exactly... value to, to the company. 230 00:42:01.680 --> 00:42:10.580 It was a very interesting thing that I got from, from the talks in the FinOps, like, it's like 90% of the applications that are AI generated are not from the IT team. 231 00:42:10.960 --> 00:42:18.190 So that gives us, like, the picture of how important this is for, you know, for business and why- Exactly... 232 00:42:18.190 --> 00:42:24.520 is all this token economics and all this controlling the cost because it's like, hey, this is a lot of money- Yes. [laughs]... that we are playing. 233 00:42:24.600 --> 00:42:33.240 Like, uh, like cloud is, is a lot of money, but AI is a lot of money because it's all the company, so, like, a lot of it. Yeah. So I think... 234 00:42:33.280 --> 00:42:44.030 yeah, I think that's a great summary of what, you know, all the capabilities that we can do in GCP and how to actually apply some FinOps for AI there in, in GCP. So Vera, thanks a lot for- Thank you very much, Victor. 235 00:42:44.100 --> 00:42:50.460 Thanks, thanks for being here today. It's a pleasure to have you. Yeah. Uh, we are now going to have some lunch. [laughs] Yes, let's go have some lunch. 236 00:42:50.470 --> 00:42:56.720 [laughs] So thanks everyone for, for listening to us, and see you in the next episode. Bye bye. Thank you very much. 237 00:43:01.020 --> 00:43:14.020 [outro music]