WEBVTT
00:00:00.299 -->
00:00:09.455Small models are where a lot of the action is, because they allow you to do things in ways on device and at low cost, or, in some cases, effectively no cost.
00:00:09.455 -->
00:00:11.948That, to me, is exciting and can build a lot of applications.
00:00:11.948 -->
00:00:17.690Welcome to Sidecar Sync, your weekly dose of innovation.
00:00:17.690 -->
00:00:26.655If you're looking for the latest news, insights and developments in the association world, especially those driven by artificial intelligence, you're in the right place.
00:00:26.655 -->
00:00:34.713We cut through the noise to bring you the most relevant updates, with a keen focus on how AI and other emerging technologies are shaping the future.
00:00:34.713 -->
00:00:38.106No fluff, just facts and informed discussions.
00:00:38.106 -->
00:00:42.264I'm Amit Nagarajan, chairman of Blue Cypress, and I'm your host.
00:00:42.264 -->
00:00:49.868Welcome back to the Sidecar Sync, your resource for all things association and artificial intelligence.
00:00:49.868 -->
00:00:51.633My name is Amit Nagarajan.
00:00:52.259 -->
00:00:53.484And my name is Mallory Mejiaz.
00:00:54.247 -->
00:01:04.674And we are your hosts and we have another action-packed episode lined up for you today, with three interesting topics at that intersection of AI plus associations.
00:01:04.674 -->
00:01:08.724Before we dive in, let's take a quick moment to hear from our sponsor.
00:01:10.849 -->
00:01:13.313Today's sponsor is Sidecar's AI Learning Hub.
00:01:13.313 -->
00:01:19.545The Learning Hub is your go-to place to sharpen your AI skills, ensuring you're keeping up with the latest in the AI space.
00:01:19.545 -->
00:01:35.712With the AI Learning Hub, you'll get access to a library of lessons designed to the unique challenges and opportunities within associations, weekly live office hours with AI experts and a community of fellow AI enthusiasts who are just as excited about learning AI as you are.
00:01:35.712 -->
00:01:38.147Are you ready to future-proof your career?
00:01:38.147 -->
00:01:42.751You can purchase 12-month access to the AI Learning Hub for $399.
00:01:42.751 -->
00:01:46.990For more information, go to sidecarGlobalcom slash hub.
00:01:46.990 -->
00:01:58.549Hello listeners, this is a special hurricane edition of the Sidecar Sync podcast for Amit, who's in New Orleans, who is about to experience Hurricane Francine.
00:01:58.549 -->
00:01:59.272How's it going, amit?
00:02:01.180 -->
00:02:03.260Well, right now it's the calm before the storm.
00:02:03.260 -->
00:02:08.582The storm's supposed to come through here in a few hours, I think, maybe later this evening at this point.
00:02:08.582 -->
00:02:10.604At the moment we have power.
00:02:10.604 -->
00:02:13.745At the moment I've got an internet connection here.
00:02:13.745 -->
00:02:15.186We are recording the podcast.
00:02:15.186 -->
00:02:16.866I've got my kids at home.
00:02:16.866 -->
00:02:19.207They're at school's close today, probably tomorrow as well.
00:02:19.207 -->
00:02:22.709Hopefully the whole area will fare reasonably well.
00:02:22.709 -->
00:02:28.931I think it's going to be a category one when it hits the coastline and then quickly downgrade to a tropical storm.
00:02:28.931 -->
00:02:32.872So hopefully everyone in the area stays safe and there's a minimal amount of damage.
00:02:32.872 -->
00:02:36.115But these things usually do hit us roughly.
00:02:36.115 -->
00:02:48.569Once a year we get something, maybe not a direct hit, but something pretty close by, as you know from living here, and it's just kind of part of the flow of New Orleans life, and I'm sure there's a lot of people in the French Quarter just hanging out drinking hurricanes.
00:02:49.301 -->
00:03:01.343I do remember one year we had a set of mutual friends that had their wedding fall in the French Quarter over a hurricane and they still chose to have it and I didn't go, but apparently it was a great time because the storm was not that bad.
00:03:01.343 -->
00:03:06.670Of course, thankfully and you know, I'm sure you're right there will be people in the French Quarter having a grand old time.
00:03:07.491 -->
00:03:18.012Well, you know, if history is a guide for structural stability, the French Quarter can be, you know, a good place to think about, because it's a lot of those buildings are 250 plus years old and they're still there.
00:03:18.012 -->
00:03:22.406It is a little bit higher in the French Quarter than it is in other parts of the town.
00:03:22.406 -->
00:03:26.393Up here I'm in Uptown, close to the universities, and our ground is just a touch higher.
00:03:26.393 -->
00:03:26.873It's still.
00:03:26.873 -->
00:03:30.948I think maybe we're like a couple of feet above sea level instead of five feet below sea level.
00:03:30.948 -->
00:03:35.185But yeah, it's dicey when you're in New Orleans and there's a big storm coming.
00:03:35.185 -->
00:03:38.760So hopefully AI will help us figure out how to fix that over time.
00:03:39.442 -->
00:03:40.402And it really should.
00:03:40.402 -->
00:03:43.487I will say this has made me happy that I'm in Atlanta.
00:03:43.487 -->
00:03:52.407This is the first time, having basically being from Louisiana, having grown up in Louisiana for most of my life, that I am not impacted by this hurricane, so that feels good.
00:03:52.407 -->
00:03:56.907But we're absolutely thinking of everybody in the storm's path and it got me thinking of me.
00:03:56.907 -->
00:04:10.811So this is episode 47 of the Sidecar Sink podcast, and we have never missed a week of posting, come internet problems, hurricanes, other bad weather, travel, and so I was just thinking I'm pretty proud of us.
00:04:11.419 -->
00:04:12.362Yeah, that's pretty awesome.
00:04:12.362 -->
00:04:21.680And also you say 47, I'm thinking that's pretty amazing because I don't keep track of the episode numbers that closely in my head, but I do have a general idea of where we're at.
00:04:21.680 -->
00:04:26.944I knew we're approaching a year, so we'll have to think of something fun to do for the 52nd episode.
00:04:27.526 -->
00:04:28.249And the 50th.
00:04:28.249 -->
00:04:29.771I feel like the 50th is going to be a big one too.
00:04:29.790 -->
00:04:30.61750th is good too, yeah.
00:04:30.817 -->
00:04:33.026Yeah, so maybe we'll do two back-to-back promos.
00:04:33.026 -->
00:04:39.685Y'all stay tuned, all right.
00:04:39.685 -->
00:04:48.096Today we will be talking about Hugging Face's new speech-to-speech model, and then, finally, we'll wrap up with a talk around AI and grid infrastructure tech.
00:04:48.096 -->
00:04:57.475Open AI's Project Strawberry is an upcoming AI model that's generating a lot of interest in the tech world due to its reported advanced capabilities.
00:04:57.475 -->
00:05:09.168As a reminder, open AI is the company behind ChatGPT, and you may recall, as a listener of the Sidecar Sync podcast, that we covered Strawberry in an earlier episode, when the name of the model was Q-Star.
00:05:09.168 -->
00:05:11.322Now I want to give a little disclaimer.
00:05:11.322 -->
00:05:19.315While Project Strawberry seems like it will be quite impressive, it's important to note that a lot of the information we have so far is based on reports and leaks.
00:05:19.315 -->
00:05:25.086The full extent of its capabilities will become clearer upon its official release and public testing.
00:05:25.086 -->
00:05:26.468So what is it?
00:05:27.209 -->
00:05:33.750Project Strawberry is said to possess significantly improved reasoning and problem-solving abilities compared to current AI models.
00:05:33.750 -->
00:05:42.353Some of its recorded capabilities include advanced mathematical skills, able to solve unseen math problems that current chatbots struggle with.
00:05:42.353 -->
00:05:48.992Improved programming abilities, capability to solve complex word puzzles, like the New York Times Connections.
00:05:48.992 -->
00:05:53.408The ability to perform deep research autonomously on the web.
00:05:53.408 -->
00:05:59.930And enhanced reasoning for answering subjective questions like those related to product marketing strategies, for example.
00:05:59.930 -->
00:06:08.726In terms of development and release, it may be integrated into chat gpt 5, potentially releasing as early as fall of this year.
00:06:08.726 -->
00:06:23.471Open ai is aiming to launch strawberry as a part of a chat bot, potentially within chat gpt, and the project is said to be in the final stages, with a possible release within the next two weeks, so we might have a very interesting episode coming up soon.
00:06:24.684 -->
00:06:37.423Strawberry is described as a reasoning model representing the second of five stages of AI innovation defined by OpenAI, and it appears to have the ability to trigger self-talk reasoning steps multiple times throughout a response.
00:06:38.586 -->
00:06:45.144Now OpenAI is reportedly considering high-priced subscriptions for access to its next-generation AI models, including Strawberry.
00:06:45.144 -->
00:06:55.011Executives are weighing charging users potentially as much as $2,000 over an undetermined period of time for access to their most advanced models.
00:06:55.011 -->
00:07:04.369So, amit, when you sent this to me, we had a quick discussion around the idea of system one and system two thinking, so I want to define that really quickly for our listeners.
00:07:04.369 -->
00:07:09.029You can think of system one thinking as fast, intuitive and automatic.
00:07:09.029 -->
00:07:14.591It's your brain's quick response system requiring little conscious effort, so you can think fight or flight.
00:07:14.591 -->
00:07:16.564And then, with system two thinking.
00:07:16.564 -->
00:07:33.290It's slower, more deliberate and analytical and involves conscious reasoning and problem solving, and you mentioned to me, amit, that the current models that we see right now use system one thinking, or that's the way that you can think about it, whereas something like strawberry falls into the system two category.
00:07:33.290 -->
00:07:34.862Can you talk a little bit about this?
00:07:35.625 -->
00:07:53.545Sure, and for those that aren't particularly familiar with this classification of system one, system two this is drawing directly on the work of Daniel Kahneman, who is a famous, or was a famous psychologist who recently passed, who wrote a book called Thinking Fast and Slow, and we'll include a link to that in the show notes.
00:07:53.545 -->
00:08:03.105I would highly recommend that book to anyone who's just interested in digging deeper and understanding how this works in the biological brain and the similarity with AI.
00:08:03.105 -->
00:08:11.255I think is worth noting in some ways that we're simply borrowing those terms to help us understand the way AI works currently.
00:08:11.255 -->
00:08:19.507So if you think of system one as the thinking fast side, where it's intuitive, you're not actually thinking about thinking, you're just doing stuff right, you're thinking.
00:08:19.507 -->
00:08:21.062It's kind of like a reaction to.
00:08:21.062 -->
00:08:33.850I think I mentioned to you in the text thread when we were talking about this if you see an alligator on the street of New Orleans, which doesn't happen too often, but if you did see one here, you could and you definitely see them in the ponds.
00:08:33.850 -->
00:08:40.880If you saw one and if you were very close to it, you might react by running away or you might react by jumping back.
00:08:40.880 -->
00:08:46.245You did not stop and say, oh no, there's an alligator, it might eat me.
00:08:46.245 -->
00:08:47.466I better leave.
00:08:47.466 -->
00:08:51.431You didn't think through that, you just jumped away, right.
00:08:51.431 -->
00:08:53.913So there's basically a rapid response.
00:08:53.913 -->
00:09:08.672And then system two might be oh, we need to figure out how to plan the Digital Now conference and we need to figure out how to market it well so that we get hundreds of people there and that it's an awesome event, complex, and we say, well, how do you break that down?
00:09:08.672 -->
00:09:10.485How do you break that into smaller pieces?
00:09:10.485 -->
00:09:23.364What is the series of steps and thoughts, or a problem that has no known solution, where you say, oh well, we want to create a room temperature superconductor or something like that, where it's like you have a very different approach to that, like how you break down a problem like that.
00:09:23.443 -->
00:09:46.157So system two thinking more broadly, I would say is something that requires metacognition, or thinking about thinking where you have to basically come up with a plan and current models, in spite of their incredible capability, do not plan, they do not reason, they simply predict the next token, as we've talked about.
00:09:46.157 -->
00:09:54.062Essentially, think about the current generation of AI that we have available to us right now as very powerful statistical models.
00:09:54.062 -->
00:09:58.351These are doing probabilistic, next token or next word prediction.
00:09:58.351 -->
00:10:15.471Now what's happened is these models have scaled so much and they're so powerful that they actually have these amazing capabilities to write prose and poetry, to write code, to interact with us, to be brainstorming partners, all the amazing stuff everybody has basically been freaking out about for the last two, three years.
00:10:17.081 -->
00:10:30.509But they don't think, they do not reason and they most definitely do not plan, and so they get stumped on problems that are as simple as how many R's are in the word strawberry, which is why the strawberry name is thrown out there.
00:10:30.509 -->
00:10:33.241So, how many R's are in the word strawberry?
00:10:33.241 -->
00:10:37.604Well, if you said, oh, okay, well, let me look at the word and let me count the R's, it's a pretty easy problem to solve.
00:10:37.604 -->
00:10:49.363But if you're not doing that, if you just are saying what would be the answer to that question based upon the corpus of text that I have, what would be the answer to that question based upon the corpus of text that I have, which would tell me what should I predict the number would be?
00:10:49.363 -->
00:10:59.863It's not based on actually breaking down the problem, saying, well, to figure out how many R's are in the word strawberry, I have to look at the word, I have to count the number of instances of the letter R, and then I have to tell you that number.
00:10:59.863 -->
00:11:01.866That's what a deterministic algorithm would do.
00:11:01.866 -->
00:11:05.114That's what we would do also if we were thinking about the problem first.
00:11:05.620 -->
00:11:15.113But the reason these models, including many of the most recent models, fail at that is because they don't have enough of a corpus of content that has that answer pre-written.
00:11:15.113 -->
00:11:16.826So it's a weird question, right?
00:11:16.826 -->
00:11:18.005How many R's in the word strawberry?
00:11:18.005 -->
00:11:22.384Like how many times has that been asked prior to all of this stuff going on?
00:11:22.384 -->
00:11:26.129So that's why these systems are failing is because the training data doesn't have that.
00:11:26.129 -->
00:11:29.354In comparison, though, like it's a simple computation.
00:11:30.215 -->
00:11:40.774So what if a model was able to determine if it should slow down and think as opposed to immediately react?
00:11:40.774 -->
00:11:56.000Right, don't just jump back from the alligator type of instantaneous reaction, but take a breath, think about it and let your bigger brain basically kick in to do the harder work right the longer range planning, the step-by-step breaking things down into complex parts.
00:11:56.000 -->
00:11:57.645Ai models do not do that right now.
00:11:57.645 -->
00:12:04.822They are about to, and Strawberry is possibly going to be the first model that has this capability.
00:12:04.822 -->
00:12:11.525Openai has been talking about it for a long time, possibly through well-architected leaks and drama or whatever.
00:12:11.525 -->
00:12:22.913You can have whatever theory you want about why they've been doing what they've been doing, but I think they've got something interesting and I think I don't think I know from just reading what these people are talking about across the community generally.
00:12:22.913 -->
00:12:26.515This is the thing researchers are working on in terms of model design.
00:12:26.936 -->
00:12:37.815So if you can bake actual reasoning, planning and all these capabilities into the model.
00:12:37.815 -->
00:12:38.799It's astounding what that's going to do.
00:12:38.799 -->
00:12:42.583Well, I can say I really like the name Strawberry a lot more than QSTAR.
00:12:42.583 -->
00:12:42.924It's more fun.
00:12:42.924 -->
00:12:43.525It's very memorable.
00:12:43.525 -->
00:12:49.446Am I understanding you correctly, saying that strawberry is not a next word predictor, or is there some element of it?
00:12:49.446 -->
00:12:52.581That will be, but there's also these other pieces in the background.
00:12:53.524 -->
00:13:03.261The short answer is we don't know yet, because we don't know what strawberry is or isn't, but my suspicion is that it's an ensemble or a hybrid where you have multiple different models within it.
00:13:03.261 -->
00:13:10.663We've talked on this podcast before Mallory, about mixture of experts models, where we have different kinds of models, that kind of combined.
00:13:10.663 -->
00:13:15.822Like you know, Mistral had the first kind of broadly available model that was open source.
00:13:15.822 -->
00:13:23.427That was an MOE model with eight underlying models, and there's a lot of belief that GPT-4 and its first edition was an MOE model as well.
00:13:23.427 -->
00:13:30.251And a mixture of experts model is basically multiple models that are combined together with an intelligent routing mechanism.
00:13:30.580 -->
00:13:41.232So the question or the prompt comes in and then the first thing that the model is doing is deciding which subcomponent or subcomponents of its mixture of experts it's going to use.
00:13:41.232 -->
00:14:01.821It's like saying, hey, I got a question in my email inbox and my email inbox might answer both questions about our upcoming meeting, but it also might answer questions about our knowledge base within our domain and it might be a complaint, and so each of these types of emails that come in might need to be handled by different people on my team, right?
00:14:01.821 -->
00:14:05.409So I might send, oh well, the domain expertise I'm member services.
00:14:05.409 -->
00:14:11.005That's not my strength, so I'm going to send it to our publications person who comes from the field and they can probably answer this.
00:14:11.005 -->
00:14:13.168That's the idea, like it's a team-based approach.
00:14:13.168 -->
00:14:16.769So model architectures have already embraced this idea.
00:14:16.769 -->
00:14:22.409But the mixture of experts thus far have been mixtures of next token predictors.
00:14:22.840 -->
00:14:28.328So they're just tuned differently, with different capabilities as their strengths and as powerful as that is.
00:14:28.328 -->
00:14:42.052This will probably be a similar kind of architecture, where there's some kind of intelligent routing and then, within that model architecture, you'll have the ability to run longer range planning, do step-by-step reasoning and execute things.
00:14:42.052 -->
00:14:52.583If you give the model more time, you might say, hey, just like I'd work with a team, I might say, mallory, I'd really appreciate it if you could develop a plan for marketing the Digital Now Conference.
00:14:52.583 -->
00:14:56.802Can you do that over the next week and come back and you would put some thought into that.
00:14:56.802 -->
00:15:01.585You'd come back and say what resources do we have, what are our goals, what types of things might we want to do?
00:15:01.585 -->
00:15:03.328And you'll come back with a plan in a week.
00:15:03.389 -->
00:15:14.530If I say, oh, I need it, like in five seconds, well, you're just going to blurt out whatever it is you can immediately think of right, and both of those capabilities have purpose and value.
00:15:14.530 -->
00:15:19.952But if the model is going to completely self-determine should it take more resources or not?
00:15:19.952 -->
00:15:25.695Or perhaps it's something where you have to tell the model take your time thinking about this and do the following right.
00:15:25.695 -->
00:15:28.945But I suspect that it will probably be able to self-determine.
00:15:28.945 -->
00:15:36.676If the nature of the prompt is such that it requires or would benefit from longer range thinking and planning, does that kind of make sense?
00:15:37.201 -->
00:15:38.046It does, it does.
00:15:38.046 -->
00:15:51.400I think the part I kept getting hung up on this was with the system two, the thinking slowly and understanding how a model needs to think slower because, right is technology, it just probably happens as quickly as it can or as a chip can support.
00:15:51.400 -->
00:15:55.511But I the idea of mixture of experts and essentially all these things happening.
00:15:55.511 -->
00:16:06.471You provide a prompt and then all of these experts, let's say consulting, quote unquote, with one another, this routing that you mentioned, it makes sense that that is the slowdown and that is why we're getting the more thoughtful output.
00:16:07.898 -->
00:16:08.320Exactly.
00:16:08.975 -->
00:16:18.210Amit, how does this kind of system if we're talking about mixture of experts architecture here compare to multi-agentic systems which we've discussed at length on the pod?
00:16:19.495 -->
00:16:23.547Yeah, you know, multi-agent systems are basically similar.
00:16:23.547 -->
00:16:30.083If you think about it, it's like you have units of capability that are constructed together and you're kind of like building blocks.
00:16:30.083 -->
00:16:34.791You know, or think of it as Lego blocks, right, that are constructed together and you're kind of like building blocks.
00:16:34.791 -->
00:16:36.452You know, or think of it as Lego blocks, right.
00:16:36.452 -->
00:16:41.350In a multi-agent system, you have prompting, you have memory, you have all these resources which are kind of like Lego blocks, and you construct them up to create an agent that does certain things.
00:16:41.350 -->
00:17:01.287What we're saying here is that the underlying model itself will have the capacity to handle basically a broader range of tasks, and so, because the model itself will be capable of doing this, you end up potentially with much more powerful capabilities.
00:17:01.287 -->
00:17:09.585Because in an agentic system, what you're doing is taking a fairly low-powered model in the grand scheme of things, like a GPT-4.0, a Cloud 3.5, sonnet.
00:17:09.585 -->
00:17:11.491They're fairly low-powered models and we're trying to make them smarter.
00:17:11.491 -->
00:17:21.944What we're trying to make them smarter, what we're doing, is essentially putting a layer of software on top of that to say, oh, mallory's asked for a marketing plan for digital now.
00:17:21.944 -->
00:17:25.419So GPT-4.0 can certainly give you an answer to that question.
00:17:25.419 -->
00:17:37.147But what if we had a multi-agent system that said okay, I'm a marketing planning expert, the first thing I'm going to do when I get a prompt is I'm going to ask a model break this down into a set of tasks.
00:17:37.147 -->
00:17:39.057So that's the first thing we do.
00:17:39.057 -->
00:17:45.136Is we go to like Cloud 35 or Gemini or 405B from Lama and say give us a breakdown of what we should do.
00:17:45.136 -->
00:17:54.355Let's not try to solve the problem, let's like break this down into a set of steps, a chain of thoughts, a chain of tasks, right, a tree of thought.
00:17:54.355 -->
00:18:03.419These are all the different terms that get chucked around like candy in this field and basically all they mean all the same damn thing, which is, essentially, you're going to break it down into a sequence of steps.
00:18:03.419 -->
00:18:05.143It's like the simplest concept in the world.
00:18:05.143 -->
00:18:06.105That's been.
00:18:06.507 -->
00:18:19.761You know, complexity has been added for reasons that make sense, because they're all subtly different, but also because it's helpful for people, I think, to feel like it's really cool and more complex than it really is, but basically it's just breaking the thing down into small pieces.
00:18:19.761 -->
00:18:22.859So the agent, the first step would be break it down into a plan.
00:18:22.859 -->
00:18:24.423And then what would the agent do?
00:18:24.423 -->
00:18:27.699The agent would say okay, step one think about our goals.
00:18:27.699 -->
00:18:29.040Okay, what should our goals be?
00:18:29.040 -->
00:18:31.044Maybe that gets broken down into five other steps.
00:18:31.463 -->
00:18:40.709And so the agent breaks it down into small enough tasks where it thinks that it can actually farm out these tasks either sequentially or in parallel to different LLMs.
00:18:40.709 -->
00:18:51.526And so I might say, okay, now I've broken it down into a series of tasks I want to do paid advertising, I want to do email marketing, I want to create a landing page, I want to do these eight different things.
00:18:51.526 -->
00:18:58.324And then the agent says, okay, I need to do those eight different things.
00:18:58.324 -->
00:19:07.757And it starts firing off those eight different tasks to eight different prompt strategies, to maybe the same LLM or different LLMs, and then it pulls those results back in and it constructs them into one integrated marketing plan.
00:19:07.757 -->
00:19:18.593Right, well, if the model itself had the capacity to do that, it would have a higher level of understanding, because right now the model is only doing one little narrow task.
00:19:18.593 -->
00:19:21.339So the model only knows that little bit that you've given it.
00:19:21.801 -->
00:19:32.269So in theory, if the model itself was doing an entire, broader task, it potentially could be far more robust in its capability because it actually has visibility understanding.
00:19:39.674 -->
00:19:44.092So it's kind of like in this kind of an agentic system, each step in the process has zero understanding of where that step is relative to all the other steps.
00:19:44.132 -->
00:19:55.029That's just kind of the nature of how these things work at the moment, and the way you construct long-term memory, the way you construct state and all this other stuff that you do in agent systems is a software layer above the model.
00:19:55.390 -->
00:20:06.324But if you bake it into the model itself, then in theory that will result in far more powerful emergent capabilities and the model will have the ability to like use tools itself, like researching on the web.
00:20:06.324 -->
00:20:20.338In concept, this could help us do original science, where a model is expressing hypotheses and the hypotheses aren't just predicted token sequences but are based on some kind of reasoning steps and then go experiment.
00:20:20.338 -->
00:20:21.983There's all sorts of cool stuff that could come from it.
00:20:21.983 -->
00:20:37.686But, to answer your question, at surface level it's actually very much the same as what a multi-agent system does, but under the hood it's very different because the model itself has far more sophistication and from a performance perspective it should allow us to scale in an interesting way.
00:20:37.686 -->
00:20:39.278So I'm really pumped about it.
00:20:39.278 -->
00:20:49.278I think OpenAI tends to be the leader in a lot of new research, but there's a lot of other companies working on the exact same problems, so I expect to see a flurry of this this fall.
00:20:50.461 -->
00:20:56.755And to really simplify this, when we're talking about agents or multi-agent systems, we just mean multiple models in one system.
00:20:56.755 -->
00:21:03.023When we're talking about something like Strawberry, we think we're talking about one model with, let's say, experts within it.
00:21:03.023 -->
00:21:06.582So one model versus multiple is that like a simplified way to think about it?
00:21:07.055 -->
00:21:16.469Yeah, you can definitely think of it that way, and the word model in this particular context could be in quotes, in the sense that they might all be the same actual model.
00:21:16.469 -->
00:21:19.684It might be Claude working across all of those.
00:21:19.684 -->
00:21:21.753It's just a different prompting strategy, gotcha.
00:21:21.753 -->
00:21:31.006So, like you know, the way you would prompt Claude to give you an email campaign would be a different prompt than what you would use for creating a web page or a different prompt for creating you know.
00:21:31.006 -->
00:21:35.990So the idea is that Claude may be doing all of those things for you.
00:21:35.990 -->
00:21:40.026It might be one underlying language model, but you're prompting it in different ways.
00:21:40.154 -->
00:21:48.404And what an agent system does, is it essentially is software that does all of these multitude of steps for you and then pulls back those results.
00:21:48.404 -->
00:21:56.152So if you want to simulate what it's an agent to, you could say okay, I'm going to, I as a user, I'm going to break down this complex task.
00:21:56.152 -->
00:22:02.055I'm going to ask an LLM for a set of steps, then I'm going to go ask that same LLM for the output for each of those other things.
00:22:02.055 -->
00:22:10.902Then I'm going to take the response from each of those separate prompts, I'm going to pull them back up and I'm going to go back to the first level and I'm going to say here's the results for all these pieces.
00:22:10.902 -->
00:22:13.383Bring it all together for me and the LLM will do that.
00:22:13.383 -->
00:22:14.884That's basically what agents are.
00:22:14.924 -->
00:22:16.826They're actually very simple pieces of software.
00:22:16.826 -->
00:22:28.759They just kind of, you know, they're kind of like the general contractor that's constructing the house, but the subcontractors, you know, put up the framing, they do the siding, they do the roof, they do the electricity and so forth.
00:22:28.759 -->
00:22:36.601So the agent system is kind of like just overseeing a complex task, but it's super inefficient in a lot of ways.
00:22:36.601 -->
00:22:39.567It's very much lacking in terms of context sharing.
00:22:39.567 -->
00:22:41.479So there's a lot to be desired.
00:22:41.538 -->
00:22:44.715And, by the way, agent systems don't go away If Strawberry's successful.
00:22:44.715 -->
00:22:52.222What happens is agent systems still can be built on top of layers like Strawberry, and this has happened with software since the beginning of time.
00:22:52.222 -->
00:22:54.932Where you build something, it becomes more powerful.
00:22:54.932 -->
00:22:56.356You build another layer on top of it.
00:22:56.356 -->
00:23:07.736So embedding these capabilities into the model would mean that some use cases for agents aren't necessary to be agents anymore, but it also means that you can do so much more with agents.
00:23:07.736 -->
00:23:15.300So agents like Betty and Skip in our ecosystem will become just crazy powerful if we plug in a model like a strawberry into them.
00:23:15.300 -->
00:23:22.047They already have multi-agentic structures like I'm describing, but they'll become far, far more powerful if the underlying model is smarter.
00:23:23.836 -->
00:23:25.761And I was just curious on this last piece of meat.
00:23:25.761 -->
00:23:37.506Well, it sounds like you just kind of answered it, but when strawberry is released, if it's at the price point I read on one article of $2,000 per month Is this something you're going to run and try out and plug into your products?
00:23:37.506 -->
00:23:41.046Do you have any sort of other use cases that you personally would want to use something like this?
00:23:42.018 -->
00:23:51.125for yeah, I mean, as soon as anything comes out, our group will collectively go and all experiment with it pretty much immediately if it's for a major player or if it's something particularly interesting.
00:23:51.125 -->
00:24:06.544So yeah, for sure, I mean, you know, for us it's not going to change, Like the price is not really relevant to whether or not we'll test it, because we'd look at it and say, hey, you know, this is kind of like a fine wine, you know, we'll use it sparingly in certain cases we're not going to, like you know, pour it out of the faucet like water.
00:24:06.544 -->
00:24:12.230So I think that you're still going to have uses for models like GPT-4-0 Mini or Lama 7B.
00:24:12.230 -->
00:24:14.919I would say collectively.
00:24:14.919 -->
00:24:19.299You know I am, I'm super excited about what something like Strawberry is going to bring us.
00:24:19.299 -->
00:24:29.180I'm actually more excited about the scaling of the power in small models and I've said this a number of times in this pod for the last, you know, 47 episodes and a number of different times.
00:24:29.855 -->
00:24:39.536Small models are more exciting because if you look at what small models do, like Lama 3.17b, 7 billion parameter model it's the smallest Lama model.
00:24:39.536 -->
00:24:49.862Or Microsoft Fi, which is PHI their third generation and many others these small open source models are super fast, super cheap and they're open source.
00:24:49.862 -->
00:24:54.451And, most importantly, super fast, super cheap and they're open source.
00:24:54.451 -->
00:25:03.299And, most importantly, the capability of Lama 3.1, their smallest model, is about on par with earlier versions of very high-end models, right?
00:25:03.299 -->
00:25:10.221So the compression of size while retaining functionality and capability from the larger models of the prior generation just keeps happening consistently.
00:25:10.221 -->
00:25:12.386So it's a really exciting time.
00:25:13.375 -->
00:25:14.319And so actually I misspoke.
00:25:14.319 -->
00:25:17.365It's Lama 3.1, 8 billion parameter and then it's 70 billion.
00:25:17.365 -->
00:25:18.880I always flip the two around.
00:25:18.880 -->
00:25:19.844But it's the small model.
00:25:19.844 -->
00:25:29.286It's unbelievably powerful, it's as good as GPT-3.5 Turbo from last summer and Lama 70B is about on par with the first version of GPT-4.
00:25:29.286 -->
00:25:34.717So why do I say that's more exciting?
00:25:34.717 -->
00:25:37.163Well, not that I have to choose, because it's kind of a cool time, we can do both.
00:25:37.163 -->
00:25:45.115But those models you can use at light speed and you can use them all over the place in solutions, and then the fine wine might be strawberry, right?
00:25:45.115 -->
00:25:51.446Or you say, okay, I'm going to use that only in situations where I know I need that layer of true power.
00:25:51.446 -->
00:25:53.390So if it's super expensive, that's fine.
00:25:53.390 -->
00:25:54.416You just use it sparingly.
00:25:55.500 -->
00:25:58.465Next we're talking about HuggingFace Speech-to-Speech.
00:25:58.465 -->
00:26:04.820Huggingface is a leading platform in the field of AI and machine learning, particularly focused on natural language processing.
00:26:04.820 -->
00:26:07.269You can think of it like a model hub.
00:26:07.269 -->
00:26:13.707Huggingface provides a vast repository of pre-trained AI models for NLP or natural language processing tasks.
00:26:13.707 -->
00:26:19.047These models can be easily accessed, downloaded and fine tuned for specific applications.
00:26:19.934 -->
00:26:34.319Hugging Face recently released multilingual speech to speech, so now it's cross platform pipeline to run GPT-4.0 like experiences on device can now seamlessly switch languages mid conversation with onlya 100 second millisecond.
00:26:34.319 -->
00:26:35.182Sorry delay.
00:26:35.182 -->
00:26:54.867They released a short video recently on LinkedIn showing this quick switch in languages between Chinese, english and French and I will say there is a slight lag, but honestly it's pretty imperceptible and I would say I hear longer lags in just phone trees when you're calling for customer service reasons in English in the same language.
00:26:54.867 -->
00:27:03.689And Hugging Face announced that speech-to-speech has been received really well on GitHub and it's working to expand the languages included in speech-to-speech.
00:27:03.689 -->
00:27:10.795Amit, we know LLMs can, or large language models can, interact with users right now in multiple languages.
00:27:10.795 -->
00:27:16.231So what would you say is novel in your opinion about speech-to-speech on Hugging Face?
00:27:16.231 -->
00:27:18.498Is it its ability to detect language?
00:27:18.498 -->
00:27:19.842The reduced latency?
00:27:19.842 -->
00:27:20.864Is it all of those things.
00:27:20.864 -->
00:27:21.886What do you think is impressive?
00:27:22.535 -->
00:27:23.317It's all of those things.
00:27:23.317 -->
00:27:29.702It's open source, it's super inexpensive, you can inference or run it on almost any device and it's going to get faster.
00:27:29.702 -->
00:27:37.339It's going to go from 100 milliseconds probably down to half of that, and then half of that.
00:27:37.339 -->
00:27:38.726It's going to keep progressing at the rate we've been talking about.
00:27:38.726 -->
00:27:43.843The languages will explode and it'll basically have all known languages at some point, right, and these will be available for multiple different vendors.
00:27:44.003 -->
00:27:47.218I think that the other key thing here is that it's speech-to-speech.
00:27:47.218 -->
00:27:49.161It's not speech-to-text-to-text-to-speech right.
00:27:49.161 -->
00:27:56.191It's not speech to text to text to speech, right, where traditionally what we've been doing is essentially using multiple models to communicate via speech.
00:27:56.191 -->
00:28:03.729So if you use ChatGPT's voice mode, they're working on a native GPT-4.0 speech capability.
00:28:03.729 -->
00:28:06.344That means it's out there, but most people don't have access to it yet.
00:28:06.344 -->
00:28:08.540The current version people have access to.
00:28:08.540 -->
00:28:26.906Where you talk to ChatGPT, what it's doing is it's saying first thing is it's talking to you, then it's converting that to text, then it's running the text prompt, then it's getting text back and then it's converting that back to audio, and so the idea here is that speech-to-speech allows you to do translation, but it also allows you to do direct.
00:28:26.906 -->
00:28:30.196You know you're not losing the modality of speech.
00:28:30.257 -->
00:28:33.378So think about it from an information theory perspective for a moment.
00:28:33.378 -->
00:28:37.000You say, okay, where is there higher information density?
00:28:37.000 -->
00:28:38.442In text or in audio?
00:28:38.442 -->
00:28:48.689You say, well, clearly, audio can be converted to text and you lose information because you don't have tone, you don't have pauses, you don't have all the things that make audio a richer modality.
00:28:48.689 -->
00:29:01.298A richer modality when you go from text to audio, of course, then you're upscaling, so to speak, and you're trying to gain information, but at the same time you're losing a lot.
00:29:01.318 -->
00:29:03.784So if I go to a model and I say you know, how should I prepare for the hurricane?
00:29:03.784 -->
00:29:06.877Just like that, you know it might interpret it one way.
00:29:06.877 -->
00:29:09.605But if I said it kind of like, hey, how should I prepare for this hurricane?
00:29:09.605 -->
00:29:10.948You know it's a little bit different.
00:29:10.948 -->
00:29:13.555It's kind of like understanding my tone, same exact words.
00:29:13.555 -->
00:29:24.083The text would not capture that, but the audio input would and therefore the model potentially can take into account a lot more of what it is that I'm thinking, doing, feeling, et cetera.
00:29:24.083 -->
00:29:50.442And the same thing is true also for translation, because literal translations, word for word, one of the reasons they've historically been really unsuccessful compared to human translators is that they don't really capture the context of what's going on in the conversation, whereas these AI-based translators have been really good because they have this broader context window that can understand pretty much the full conversation at this point and be super accurate at translation because of that.
00:29:50.442 -->
00:29:53.277So it's really powerful on a lot of levels.
00:29:53.356 -->
00:30:05.981It actually ties to my comments in the last topic pretty well that small models are where a lot of the action is, because they allow you to do things in ways on device and at low cost or, in some cases, effectively no cost.
00:30:05.981 -->
00:30:07.244That, to me, is exciting.
00:30:07.244 -->
00:30:08.435You can build a lot of applications.
00:30:08.435 -->
00:30:11.563So for associations, you might say, okay, what do we do with this?
00:30:11.563 -->
00:30:15.563What if you had in your meetings app a real-time translator?
00:30:15.563 -->
00:30:19.988Say, you have an international crowd coming to your event and people are speaking whatever language.
00:30:19.988 -->
00:30:30.642So you build a simple tool that basically pipes in real-time audio from the AV system, from the thing to the device, and that's translated in real-time.
00:30:30.642 -->
00:30:42.651It's got 100 millisecond lag or whatever, but it's imperceptible, and so I can keep an earbud in and listen to it in whatever language and it sounds like the person speaking right as opposed to a translation.
00:30:42.651 -->
00:30:45.038So it's a translation in terms of the voice and all that.
00:30:45.038 -->
00:30:49.180That would be amazing, and there's a lot of other applications, but that's just one that comes to mind.
00:30:51.208 -->
00:30:57.071In my mind I was thinking that, even though this is speech-to-speech, that it was still going through that text phase.
00:30:57.071 -->
00:30:58.719So that's really interesting to think about.
00:30:58.719 -->
00:31:06.775So this model we can assume was trained on lots of audio to develop this capability like not text data, I'm assuming.
00:31:06.795 -->
00:31:08.816So Okay, yeah, I haven't read the paper.
00:31:08.816 -->
00:31:12.817They published a paper on it as well, which is another great thing about the open source communities.
00:31:12.817 -->
00:31:19.201You not only get the software and the weights, but you also typically get a paper that explains how did they train, what did they train on.
00:31:19.201 -->
00:31:25.924And sometimes it's interesting just to kind of skim those and see what they talk about, but they'll tell you about their training approach with the content.
00:31:25.924 -->
00:31:26.645Where do they get it?
00:31:26.645 -->
00:31:39.913And these guys are big open source advocates based in France, and they are publishing tons of research all the time, but they are, as you described earlier, a hub for a lot of pretty much everyone posts stuff there as soon as it's ready for consumption.
00:31:39.913 -->
00:31:45.448Have you played around in Hugging Face?
00:31:45.448 -->
00:31:59.255Yeah, I have no-transcript.
00:31:59.255 -->
00:32:07.707Actually, I think there was news, maybe in the last day or so, that the Firefox browser actually includes Hugging Chat, built right in Wow, so that's a cool thing to check out.
00:32:08.476 -->
00:32:08.717Right.
00:32:08.777 -->
00:32:10.604We'll include that in the show notes as well.
00:32:11.709 -->
00:32:22.163Now I know you mentioned translating keynote sessions as one potential use case, and then my mind immediately goes as well to customer service calls and being able to translate those real time.
00:32:22.163 -->
00:32:43.817And this leads me to kind of a broader thought, and I don't know if you know a ton about this, amit, but it's just something I think about a lot, especially given how advanced AI is getting, thinking that we still have to call these major players like Verizon and Delta and we're still working through these phone trees and it seems like you have to say the right keyword right for them to move you on to the customer service rep.
00:32:43.817 -->
00:32:53.671Do you know why we aren't seeing kind of major companies use AI or at least it doesn't seem like they are using AI in their customer service calls?
00:32:54.974 -->
00:32:59.419You know, my suspicion is that for the larger players there's cultural roadblocks.
00:32:59.419 -->
00:33:02.079There's also complexity of systems integration.
00:33:02.079 -->
00:33:09.925For Delta Airlines to implement this would be a lot harder than something like Klarna, who we talked about before, and they implemented a customer service bot.
00:33:09.925 -->
00:33:11.646Smaller company rolling it out.
00:33:11.646 -->
00:33:23.174But it also creates an expectation in the mind of the consumer right.
00:33:23.174 -->
00:33:26.123Think about the mainstream consumer and what their expectations are, which expectations quickly turn into demands by consumers.
00:33:26.123 -->
00:33:32.363You will see adoption of these tools, both because it's better for the consumer but also because it saves a lot of money.
00:33:32.363 -->
00:33:36.599So ultimately, those companies, 100%, are going to be deploying this kind of tech.
00:33:36.599 -->
00:33:41.140I don't know about Verizon, actually, but everyone else probably will.
00:33:41.781 -->
00:33:43.286Don't get me started on Verizon.
00:33:43.875 -->
00:33:44.597Yeah, exactly.
00:33:44.597 -->
00:33:55.175So the way I describe customer service tech and technology in general I think this is a true statement is that the first priority for companies has always been to serve themselves.
00:33:55.175 -->
00:34:00.935Meaning when you think about your integrated voice response, ivr phone tree that you're talking about.
00:34:00.935 -->
00:34:07.288Those systems were never built to optimize for customer experience.
00:34:07.288 -->
00:34:14.436They were built to hopefully minimize how bad the experience was, but they were built to optimize for cost.
00:34:14.436 -->
00:34:27.940And if you think about the original chatbots that have been on the web for 10 plus years right and longer in some cases they're horrendous, but they're not optimized to serve the customer, to improve the experience.
00:34:27.940 -->
00:34:32.617They're optimized to reduce cost primarily right, I mean, there's other motivations that could exist.
00:34:32.617 -->
00:34:40.643It's like, oh, we don't have people on on the clock 24 hours a day and this thing can help in the meantime, but, but generally speaking, they're extremely limited of value.
00:34:40.643 -->
00:34:44.898And, like, the first thing I always do is I say agent and I say give me an actual person.
00:34:45.219 -->
00:34:46.043But they're better now.
00:34:46.043 -->
00:34:49.603I mean, they make you say it like 25 times before you get to the agent.
00:34:49.603 -->
00:34:50.706Oh, so frustrating.
00:34:50.875 -->
00:34:51.735Proving my point right.
00:34:51.735 -->
00:34:59.947It's not about the experience of the customer, it's about reducing costs, because if they do that, I might go away and then I don't have to call them again, or you know, and they don't really care, right?
00:34:59.947 -->
00:35:02.536So I mean, I don't want to be a cynic about all companies.
00:35:02.536 -->
00:35:17.726There's definitely plenty of companies who do value creating a great customer experience, but, generally speaking, over the course of time, the technology has been focused on how do you optimize costs and how do you optimize other priorities internally, which rarely are about the customer experience, with some exceptions, obviously.
00:35:17.726 -->
00:35:28.907What I would say now is that you're entering a world where, fortunately, the alignment of the priorities is similar, where the customer experience is both improved and the company saves a crazy amount of money.
00:35:29.875 -->
00:35:31.559And the Klarna case study is a great one.
00:35:31.559 -->
00:35:34.217We included it in the book and we talked about it in this podcast.
00:35:34.217 -->
00:35:36.561Klarna is a buy now, pay later company.
00:35:36.561 -->
00:35:44.842They power a lot of e-commerce brands people who want to purchase a product but essentially pay for it in installments over time.
00:35:44.842 -->
00:36:06.317They're one of several companies in the BNPL space and they implemented a ChatGPT4-based chatbot, and they found that the chatbot was able to, in its first month of operation answer the equivalent number of inquiries as 700 people, which is, I think, about a third of the number of people they actually employ in that role.
00:36:06.317 -->
00:36:08.663And what was really interesting?
00:36:08.663 -->
00:36:12.902So that's the cost-saving side of it, right, and so that's exciting, that's great.
00:36:13.483 -->
00:36:19.123The flip side of it is that the customers were happier with the outcome of the AI.
00:36:19.123 -->
00:36:19.525Why?
00:36:19.525 -->
00:36:28.402Not necessarily because the bot did a better job responding although over time it most certainly will do a better job of responding but it was about the amount of time it took.
00:36:28.402 -->
00:36:33.632Like when you call Delta, would you rather have your problem solved in 90 seconds or 10 minutes?
00:36:33.632 -->
00:36:44.146Right, you want to be off the phone or off of the chat bot as quickly as possible with your resolution to you know you want the information you went there for or you have your transaction handled or whatever it is.
00:36:44.146 -->
00:36:53.530The chatbot from Klarna was able to resolve cases an average of, I think, about two minutes, if I'm remembering correctly, whereas the human agents took an average of 11 minutes.
00:36:53.530 -->
00:37:12.751So that is a massive improvement in customer experience and to me that's a big opportunity coming back to this topic of speech to speech, because it opens up a new modality and, again, it's a richer form of information coming in audio that isn't downscaled to text and then upscaled later back to audio, doesn't lose the information right.
00:37:12.751 -->
00:37:29.887More likely that a speech-to-speech model or that is a component of these systems, will be able to serve customers in a brilliant way and to do it so that you call Delta Airlines down the road, hopefully in the not-too-distant future, and a quote-unquote person answers which is an AI?
00:37:30.315 -->
00:37:31.340And you just have a conversation.
00:37:31.340 -->
00:37:36.219You're like yeah, I'd like to change my flight, or I'm having a problem, I didn't get miles on my last flight, or whatever it is.
00:37:36.219 -->
00:37:38.885They're like oh, let me look it up, they authenticate you quickly.
00:37:38.885 -->
00:37:42.619And they're like oh, mallory, sure, here you go, we're going to solve that problem for you and you're done.
00:37:42.619 -->
00:37:46.648It's like having your own dedicated Delta expert.
00:37:47.695 -->
00:37:59.956Also, too, when you go through those phone trees, you know very well that you either struck gold, that you got one of their good customer service reps, or every once in a while, you get someone.
00:37:59.956 -->
00:38:00.858They don't know how to spell Delta.
00:38:00.858 -->
00:38:02.101They just have no idea how to serve you.
00:38:02.101 -->
00:38:07.766You're like damn, I'm going to have to hang up and basically get back on this call to roll the dice again.
00:38:07.766 -->
00:38:11.840So if everyone get the best customer service rep instantly.
00:38:11.840 -->
00:38:12.864Wouldn't that be cool?
00:38:12.985 -->
00:38:17.199And this is a technology that will enable exactly that outcome For associations.
00:38:17.199 -->
00:38:40.001You're not Fortune 500 companies, but people expect from you the same that they expect from the largest brands in the world, which may not be fair, but it's reality, and so understanding this technology is important, because it can not only help you play defense where you're thinking about how do you provide a comparable experience to what people are looking for, but allows you to create a broader surface area to serve your customers and members in new ways.
00:38:40.001 -->
00:39:07.798Right, translating your content, like we talked about translating content in the context of a conference, but also think about, like an LMS, for example, being able to dynamically translate and have conversations with an AI agent in voice-to-voice about the content, so that the LMS really becomes like a synchronous platform where the best tutor in the world on your domain of expertise is available side by side with the pre-recorded content that's coming and you'll be able to leverage that from.
00:39:07.798 -->
00:39:11.943Probably all the mainstream LMS providers will have that in the next couple of years, in my estimation.
00:39:13.655 -->
00:39:22.188Lots of use cases there, from your experience, would you say customer service, or we could call it member service, is a big pain point for a lot of associations that you've spoken with me.
00:39:22.994 -->
00:39:23.577Yeah, for sure.
00:39:23.577 -->
00:39:26.385I mean, it's an area of significant time investment.
00:39:26.385 -->
00:39:30.663So if you think about where do associations invest their staff's time?
00:39:30.663 -->
00:39:43.088A big chunk of it is interacting with members and answering questions, and what I get excited about with this type of technology isn't that you're eliminating that, but that you're freeing up your people to actually have meaningful conversations with people.
00:39:43.088 -->
00:39:50.461So, rather than answering an endless stream of emails that are asking you basically the same 20 or 30 questions, you can have an AI.
00:39:50.461 -->
00:39:51.362Just take care of that.
00:39:51.362 -->
00:39:59.952Give people a better experience because it's immediate, and then have your people focus on, first of all, figuring out better things they can do to serve members.
00:39:59.952 -->
00:40:03.175But also, when you do have those interactions, you're not hurried.
00:40:03.175 -->
00:40:09.307You can actually take the time to go deeper with the member and make sure that you're really connecting them with the best possible outcome.
00:40:09.994 -->
00:40:13.061It's kind of like your experience, all of our experiences.
00:40:13.061 -->
00:40:15.467When you go to the doctor's office, I always feel rushed.
00:40:15.467 -->
00:40:17.458It's like, oh, you're going to spend 10 minutes with the doctor.
00:40:17.458 -->
00:40:19.780If you're lucky, they don't remember anything about you.
00:40:19.780 -->
00:40:21.382Maybe they remember your name, maybe not.
00:40:21.382 -->
00:40:25.769They pretend like they do and you have 10 minutes to tell them all about whatever it is that ails you.
00:40:25.769 -->
00:40:34.992Wouldn't it be great if that didn't feel that way and you just had all the time in the world, at your pace, to address whatever the issue is?
00:40:43.855 -->
00:40:45.061And that's a customer service problem in the medical world.
00:40:45.061 -->
00:40:45.402Same thing there.
00:40:45.402 -->
00:40:47.251So I just get pumped about this because I think it can democratize access to great services.
00:40:47.251 -->
00:40:48.034Associations can be a big part of that.
00:40:48.034 -->
00:40:50.956All right.
00:40:50.956 -->
00:40:52.320Topic three AI and grid infrastructure tech.
00:40:52.340 -->
00:40:54.887Recent reports highlight an interesting trend in climate tech investments, particularly related to AI.
00:40:54.887 -->
00:41:03.909While overall investment in climate tech startups has declined since peaking in 2021, one sector is seeing significant growth and that is grid infrastructure technology.
00:41:03.909 -->
00:41:12.268Grid infrastructure startups raised $2.73 billion globally in the first half of 2024, with US deals accounting for almost half of that.
00:41:12.268 -->
00:41:18.278The investment is driven largely by the increasing power demands of AI and electric vehicles, largely by the increasing power demands of AI and electric vehicles.
00:41:18.278 -->
00:41:26.844According to John McDonough, a senior analyst at PitchBook, vc investment in electric grid infrastructure is on track to surpass last year's $4.37 billion.
00:41:26.844 -->
00:41:38.313A significant portion of this funding is going toward battery energy storage, new battery technology, ai and software for grid management and hardware for grid management.
00:41:43.594 -->
00:41:56.865The surge in electricity demand due to AI, particularly for data centers, is actually driving interest in technologies to strengthen and manage the power grid more efficiently, which underscores how AI is a driver of innovation in other sectors, particularly in managing and improving our power infrastructure to meet growing energy needs.
00:41:56.865 -->
00:42:12.099So, amit, when you sent this to me, you mentioned that many associations rightfully so and many association leaders are concerned about the environmental impact of AI, which I don't feel like we definitely haven't spent a whole dedicated topic on, I think, on this podcast.
00:42:12.099 -->
00:42:15.146So can you speak a little bit to those concerns?
00:42:16.414 -->
00:42:21.947Well, first of all, we have to recognize that AI consumes an enormous amount of power.
00:42:21.947 -->
00:42:24.902There's an unbelievable amount of power and growing really rapidly.
00:42:24.902 -->
00:42:34.384Every time you turn around, there's some new data center that's coming up that is going to be the next bigger version, the next supercomputer, the next cluster, the next thing.
00:42:34.384 -->
00:42:42.161So these things are all very, very energy intensive and that is a valid concern, both in terms of the financial costs.
00:42:42.161 -->
00:42:42.945But then it's.
00:42:42.945 -->
00:42:46.217You know, energy right now is a scarce resource.
00:42:46.217 -->
00:42:53.923Effectively, it's expensive and it's scarce in the context, independent of the financial side, in that it produces, generally speaking, is not clean.
00:42:53.923 -->
00:42:55.635You know, most energy is still not clean.
00:42:55.635 -->
00:42:58.222So there's valid concerns there.
00:42:58.222 -->
00:43:26.577At the same time, I think that some of the points you've made about the investment being had in this sector, coupled with the growth in AI itself actually being a massive assistant in solving for these problems, I'm very optimistic that all of this AI investment, in terms of the energy side and the carbon footprint of it, is going to pay massive dividends, probably in next, you know, probably in the next decade, you know, and so you think about what you said of storage.
00:43:26.577 -->
00:43:36.400So a lot of the problems with our energy infrastructure is that when power is available isn't necessarily when it's going to be consumed.
00:43:36.400 -->
00:43:37.923So think about solar.
00:43:37.923 -->
00:43:48.277Solar is generated during the day and, assuming that there are no clouds in the sky and so forth, that isn't necessarily when it needs to be consumed.
00:43:48.277 -->
00:43:49.800Sometimes it is, sometimes it isn't.
00:43:49.800 -->
00:43:59.601So storage is a helpful thing because if I can say I'm going to have a solar farm, I'm going to store the energy and then I can redistribute it when I need to, that's very interesting.
00:44:00.425 -->
00:44:01.748Another problem is transmission.
00:44:01.748 -->
00:44:12.306So we lose a large percentage of energy in transmission because the way we transmit energy is the way we've done it for a long time and there's loss and transmission.
00:44:12.306 -->
00:44:22.978For every additional mile there's a percentage of lossiness and so we can't generate power too far away from the consumption of that power, even though we have a national grid.
00:44:22.978 -->
00:44:28.844It's really a bunch of localized grids that are connected, because the farther you try to transmit the power, the more loss there is.
00:44:28.844 -->
00:44:37.559You know the loss basically generates a lot of heat, so that has its own issues, but really, more than anything, you're just losing a large percentage of it's like you know.
00:44:37.559 -->
00:44:48.135Know you're driving an oil tanker down the road and there's a hole, and so you started off with a million gallons of gas or whatever it is, and by the time it gets to where it's going, it loses a certain amount, right?
00:44:48.135 -->
00:44:51.105That's kind of the analogy that I'd paint in picture in people's minds.
00:44:51.105 -->
00:44:53.458You see transmission wires overhead.
00:44:53.458 -->
00:44:54.159You assume that.
00:44:54.420 -->
00:44:55.581Well, how do you lose energy?
00:44:55.581 -->
00:44:57.364Well, it's just the nature of the materials.
00:44:57.364 -->
00:45:11.387It's not 100% efficient in terms of the way that material is transmitting or conducting the electricity, which is, by the way, one of the reasons there was so much excitement about this idea of a room temperature superconductor, which didn't actually turn out to be true.
00:45:11.387 -->
00:45:14.625But last year we talked about it a little bit on one of our early pod episodes.
00:45:14.625 -->
00:45:17.619The theory behind a room temperature superconductor.
00:45:17.619 -->
00:45:28.628Superconductivity essentially means there's zero loss in transmission of electrical current and so in theory and this has been produced in the lab at like, basically close to absolute zero temperature.
00:45:28.628 -->
00:45:43.000So if you could do superconductivity with materials that are affordable at scale, right in room temperature or like in the real world, then you can transmit power with no loss, and that solves one of the major problems.
00:45:43.000 -->
00:45:49.746Also, it means that the chips that we have, if we have superconductivity at that level, you're going to lose a lot less energy.
00:45:49.746 -->
00:45:53.523Chips can be smaller, faster, more energy efficient, so that's interesting.
00:45:53.864 -->
00:45:54.606Why did I bring that up?
00:45:54.606 -->
00:45:58.307It's not part of the particular investments you're talking about.
00:45:58.307 -->
00:46:08.461That's a material science problem more than anything else, and so material science is an area that's exploding, in large part due to AI, so I'm excited about that.
00:46:08.461 -->
00:46:13.983And then the last thing I'd say is there's a lot of optimizations in terms of the grid management.
00:46:13.983 -->
00:46:20.958You mentioned the software and hardware being able to be smarter about how we manage the power that we do have and how we're transmitting it.
00:46:20.958 -->
00:46:30.742So that's good in terms of better quality of service, fewer brownouts and blackouts, but it's also a good thing from an efficiency perspective, which both saves money and reduces the carbon footprint.
00:46:30.742 -->
00:46:39.358So I'm pumped about this article and the statistics about the investment, because where capital flows, innovation goes, because where capital flows, innovation goes.
00:46:39.378 -->
00:46:44.125What a good segue, I'm thinking for those of our listeners who are not.
00:46:44.125 -->
00:46:54.777I fully understand, right, that there's a huge energy consumption piece to AI, but it's always helpful for me then to go out and like find some stat or data point.
00:46:54.777 -->
00:46:57.922So I felt like this would be helpful to share with listeners as well who might be trying to bring this back down to the ground.
00:46:57.922 -->
00:46:59.824Share with listeners as well who might be trying to bring this back down to the ground.
00:46:59.824 -->
00:47:05.010By 2025, it's estimated that data centers will consume between 3% and 13%, which is a large range.
00:47:05.010 -->
00:47:13.606But think about the 13% Will consume between 3% and 13% of global electricity.
00:47:13.606 -->
00:47:15.112So for me, that definitely helps put this into terms that I can understand.
00:47:15.132 -->
00:47:17.838When thinking about greater environmental impact.
00:47:17.838 -->
00:47:33.726I think if you ask the average person, right, what can you do to help the environment, they might say things like recycle more or eat less meat or maybe drive electric vehicles, even though, amit, I think you told me that they're actually not as good for the environment as you would think.
00:47:33.726 -->
00:47:47.943But I want to know, in terms of this conversation around energy consumption and AI, is there anything an association can do or keep in mind as they're implementing AI, or even an individual, or do we just kind of have to let this play out?
00:47:48.954 -->
00:47:56.864Well, I mean, I'll come back to the analogy of the fine wine versus tap water thing that I talked about earlier and that might have a financial cost.
00:47:56.864 -->
00:48:09.485And you see, you allocate the fine wine more closely and more carefully because you don't want to spend money or you don't want to wait as long for the response and you see built systems that use that resource a lot more sparingly.
00:48:09.925 -->
00:48:11.717The same thing applies to our energy consumption.
00:48:11.717 -->
00:48:14.927If you use the smaller models, they're much more energy efficient.
00:48:14.927 -->
00:48:19.945So both you will save money and also you will have a smaller carbon footprint.
00:48:19.945 -->
00:48:28.304So it's easy to just say, oh yeah, we're going to go with, you know, chachi, pt or Claude, and we're just going to use the most intense model because it's the most powerful.
00:48:28.304 -->
00:48:29.567You know?
00:48:29.567 -->
00:48:35.737Using another metaphor, it's like flying an Airbus A380 double-decker that can seat 600 people and putting one person on it.
00:48:35.737 -->
00:48:37.940You don't need the Airbus 380 for all missions.
00:48:37.940 -->
00:48:41.387You might have a Cessna that works just fine, and so it's the same thing.
00:48:41.467 -->
00:48:50.762Small models have their purpose and I get excited about how small models especially if you inference them locally on your own computer, running them on your phone, right?
00:48:50.762 -->
00:49:00.025Apple had their event this week where they talked about Apple intelligence and the silicon that they have, which is capable of running their own models on device, which are quite capable models.
00:49:00.025 -->
00:49:08.242That excites me because that efficiency is going to result in a lot of the workloads being done by super efficient chips.
00:49:08.242 -->
00:49:16.070I'm also excited about hardware innovations, like we talked about Grok with a Q G-R-O-Q grokcloudcom.
00:49:16.070 -->
00:49:25.007We're big fans of those guys because they have a radically more efficient hardware architecture for AI inference these massive data centers you're talking about.
00:49:25.007 -->
00:49:37.521They do two things they train models and then they serve models, or what people call inference, which is basically when you run the model, and so the ratio of training versus inference is constantly changing.
00:49:37.521 -->
00:49:47.329But as demand grows, it's going to be a massive amount of inference, and so if you're able to be more efficient the way Grok is, it's both way faster, but it's less energy intensive.
00:49:47.795 -->
00:49:52.186So I think an association should be, especially as they're looking to scale their use of AI.
00:49:52.186 -->
00:49:53.860Right, like when you're in the experiment phase.
00:49:53.860 -->
00:50:03.590It's not a big deal, you're not even a rounding error, but if you're going to scale your use of AI in a significant way, you should definitely be thinking about these issues.
00:50:03.590 -->
00:50:15.925It's part of the responsible AI framework, where you're thinking about the ethics of data privacy, you're thinking about models and how they affect people, you're thinking about biases, and you should also be thinking about carbon footprint energy consumption.
00:50:15.925 -->
00:50:26.481The good news is that there is an incentive to save money, which everyone's tuned into around the world, and that same incentive will drive people to use the more efficient models, so that's exciting as well.
00:50:27.923 -->
00:50:35.657So your advice, then, is not to stop using these powerful large language models, but simply to identify opportunities where you can use small models, if possible.
00:50:36.420 -->
00:50:40.777Yeah, last time I checked, horses have a smaller carbon footprint than vehicles that we drive.
00:50:40.777 -->
00:50:43.565But I'm not suggesting you go back in time and ride a horse around.
00:50:43.565 -->
00:50:46.581I mean you could, but I don't even know if that's true.
00:50:46.581 -->
00:50:52.141I think it is, but I'm assuming that the horse's output is lesser than a typical car.
00:50:52.222 -->
00:51:02.260But my point would be that the technology is going to go forward and if you want to be relevant in your field, you have to use AI to be relevant and to be effective in your field.
00:51:02.260 -->
00:51:03.585But you can be smart about it.
00:51:03.585 -->
00:51:10.757You don't need to just say, oh, we've got plenty of money, so let's just turn on the car engine and yeah, I'm not driving it right now.
00:51:10.757 -->
00:51:18.666I'm going to keep my car running in my driveway just because I want the air conditioning on all summer long in New Orleans, because I never want to experience not having AC in my car.
00:51:18.666 -->
00:51:22.083It's like that's ridiculous, like I don't know if anyone hopefully no one does that.
00:51:22.083 -->
00:51:25.806But, like you know, even if I had an unlimited amount of money, I didn't care about wasting the money.
00:51:25.806 -->
00:51:28.764That would be just crazy, like to run your car all the time.
00:51:29.175 -->
00:51:31.364The same thing is true with wasted resources and AI.
00:51:31.364 -->
00:51:42.846If I use GPT-4-0 for every single inference request I had, it's unnecessary, it's wasteful and it has a bigger environmental impact.
00:51:42.846 -->
00:51:49.226So there are a lot of choices and one of the choices you can make is using a mixture of different models to serve your association's needs, and it's not the simplest topic in the world to be thinking through.
00:51:49.226 -->
00:51:55.076So I don't recommend it as a concern for people who are in that early, early phase, which is almost everyone right now.
00:51:55.076 -->
00:51:58.541Still, most people are in their very, very beginning of the journey.
00:51:58.541 -->
00:52:01.487You should flag this as an area of importance.
00:52:01.487 -->
00:52:09.039You should note it as something you should be thinking about actively as you grow your use of AI, but don't let it stop you from getting started in the journey.
00:52:09.681 -->
00:52:10.485All right everyone.
00:52:10.485 -->
00:52:15.521Thank you for tuning in to today's episode For any of our listeners that are in the storm's path.
00:52:15.521 -->
00:52:37.373Stay safe and we will see you all next week the storm's path, stay safe.
00:52:37.793 -->
00:52:43.693Sidecar is here with more resources, from webinars to boot camps, to help you stay ahead in the association world.
00:52:43.693 -->
00:52:45.280We'll catch you in the next episode.
00:52:45.280 -->
00:52:48.980Until then, keep learning, keep growing and keep disrupting.
