Adam Davidson
00:00:00Foreign welcome to the end of 2024 year end feedForward podcast I'm Adam Davidson, one of the co founders of FeedForward and boy, it's been a crazy year, hasn't it?
Adam Davidson
00:00:16I mean, just think where we are right now compared to where we were a year ago.
Adam Davidson
00:00:21And then you start to imagine, well, where are we going to be a year from now?
Adam Davidson
00:00:25We're lucky.
Adam Davidson
00:00:26Here at FeedForward we have some really smart people in this community and I asked a bunch of them and some others people outside of the community, what are you thinking about at this moment?
Adam Davidson
00:00:37What are you looking back at?
Adam Davidson
00:00:38What are you looking forward to?
Adam Davidson
00:00:40Let's start with one of the people I turn to all the time to understand what's going on in these AI models.
Adam Davidson
00:00:48How do we talk to them more effectively?
Adam Davidson
00:00:50How do we learn from them?
Adam Davidson
00:00:51That's one of our co founders, Lilach Malik.
Adam Davidson
00:00:54As you probably know, Lilach is not just a co founder of FeedForward.
Adam Davidson
00:00:59She also runs the Generative AI Lab with Ethan at Wharton, the University of Pennsylvania's business school.
Adam Davidson
00:01:06I think we all know that and has become one of the leading voices on how to prompt AI models.
Adam Davidson
00:01:14More than that, how to talk to them, how to think about them, how to listen to them.
Adam Davidson
00:01:18She's somebody that OpenAI and Anthropic and Google turn to all the time to figure out what can our models do and how do we interact with them better.
Lilach Malik
00:01:29So I wear many hats both inside and outside of FeedForward, including teaching and training how to use AI.
Lilach Malik
00:01:35I'm a researcher who studies prompt engineering and I'm building with AI and I'm on the forefront of how AI is integrated in education.
Lilach Malik
00:01:43So I want to talk about AI using these three hats.
Lilach Malik
00:01:46I've talked to a lot of people this year and I've seen some massive changes in attitude towards AI.
Lilach Malik
00:01:52So just putting on my training hat earlier in the year across educational institutions and across organizations, this is something people were very skeptical about whether they should use it, having mostly not touched it.
Lilach Malik
00:02:05But later in the year there was a shift to where while there's still quite a few people who haven't touched it, the organizational imperatives are there across both education and elsewhere where people have realized this is something they should do.
Lilach Malik
00:02:17They don't know what to do with it yet, but they know they have to do something.
Lilach Malik
00:02:20So there's generally this year been a shift towards a more willingness to adapt.
Lilach Malik
00:02:25Putting on my prompting hat, we still haven't crafted how to distribute prompts or well developed use cases or AI bots with specific useful behaviors to anyone who wants them.
Lilach Malik
00:02:35GPTs were a good start, but not a lot of progress has been made there and the result is that there's still a need to that you see coming up this year that of distributing use cases to non technical people remains a challenge.
Lilach Malik
00:02:48But that's a race against the idea that the AIs are just getting smarter at doing everything and that prompting is getting less useful for many cases.
Lilach Malik
00:02:56It used to be that you'd want everyone to learn prompting and prompting techniques, and now it's just more important to get your hands on these things.
Lilach Malik
00:03:03Develop an intuition for what works in your case and how to work with these Systems as the AIs are getting smarter.
Lilach Malik
00:03:09I've also been watching the improvement in intelligence without the improvement in complexity of using these systems.
Lilach Malik
00:03:15So Zero1 is a much smarter system.
Lilach Malik
00:03:18Cloudsonnet is a much smarter system, but they're easier to use rather than harder because there are fewer prompting tricks to make them do stuff.
Lilach Malik
00:03:26They're simply smarter out of the box.
Lilach Malik
00:03:28Putting on my education hat There is a growing recognition that this could be a breakthrough in training, but also a recognition inside organizations that they're in trouble because they don't understand what to do with these systems to let them solve the problem of the breakdown of the professional mentorship pipeline.
Lilach Malik
00:03:45What I'm looking forward to next year is I think agentic systems will create opportunities for figuring out how humans fit in the loop and how we monitor these systems.
Lilach Malik
00:03:54So I think there'll be lots of opportunities for people to figure that out and how you prompt and work and watch agentic systems will really be very interesting.
Lilach Malik
00:04:03I'm also interested in how these multimodal voice and vision systems change.
Lilach Malik
00:04:07Training and education.
Adam Davidson
00:04:08Moving forward, Lilach has identified one of the core questions about AI.
Adam Davidson
00:04:13I think of it as an almost civilizational question.
Adam Davidson
00:04:16Will AI on balance, enhance human capabilities?
Adam Davidson
00:04:20Will it allow us to learn more, do more, dream more?
Adam Davidson
00:04:24Or will it replace us?
Adam Davidson
00:04:25Will it replace human capabilities?
Adam Davidson
00:04:28One of our feedforward expert voices Matt Beam thinks about little else.
Adam Davidson
00:04:34This is his main question.
Adam Davidson
00:04:36This is his obsession and field of study at University of California, Santa Barbara.
Adam Davidson
00:04:42You should read his book, if you haven't already.
Adam Davidson
00:04:44The Skill how to Save Human Ability in an Age of Intelligent Machines.
Adam Davidson
00:04:50He and Lilac have helped me understand how crucial it is to think about AI's impact from several perspectives.
Adam Davidson
00:04:57The individual using it right now, typing on it right now, as well as how individuals use AI over the course of their lives, over the course of their careers, to increase their skills, to have it do things they don't want to do anymore.
Adam Davidson
00:05:11And also this is very important to look at AI not just through the lens of the individual, but through the lens of the overall organization and all the formal and informal systems that different organizations have to get things done.
Matt Beam
00:05:27I think 2024 was the year where individuals extracted productivity out of generative AI.
Matt Beam
00:05:33Whether they were working in an organization or on their own, that's where the primary gains were being had.
Matt Beam
00:05:402024 was the year that organizations foot the bill for that individual productivity gain.
Matt Beam
00:05:46And I think 2025 will be the year where organizations start to have some significant and reasonable success at extracting productivity out of generative AI at the firm level.
Matt Beam
00:05:57And I think 2025 is also going to be the year where firms and individuals start to realize more concretely, more obviously, that there's a hidden price to be paid for these productivity gains.
Matt Beam
00:06:10At the very least, they will see that the age old problem that comes with new forms of automation is going to rear its head here, which is that we drop a new form of automation, a new kind of intelligent tech, in this case into preexisting processes to try to amp them up and get more juice out of them.
Matt Beam
00:06:29And that's not bad necessarily, but it is in the end not aligned with the best, highest use case for the tech.
Matt Beam
00:06:37Refactoring processes in more wholesale ways to take advantage of the tech is really where we all need to go.
Matt Beam
00:06:44So I think this sort of short run, short and small gain will come out of the tech and then eventually organizations are going to figure it out.
Matt Beam
00:06:52I don't know if 2025 is going to be the year where that refactoring starts to happen in earnest, but I would expect to see some of it.
Matt Beam
00:06:59The second unintended consequence of all this productivity gains is one that you would definitely expect from me, Matt Bean, the author of the Skill Code how to Save Human Ability in an Age of Intelligent Machines.
Matt Beam
00:07:11And that is that we are typically changing workflows as we introduce intelligent technologies in ways that disenfranchise novices in the work, separate them from experts who are doing the work, and thereby break the primary means that humans have for building reliable skill.
Matt Beam
00:07:31So professions, organizations, functional groups and individuals have been paying this price in 2024 and will be increasingly aware of that price in 2025.
Matt Beam
00:07:43And it's not as if this is an either or situation.
Matt Beam
00:07:46I think the savvy organizations and individuals will see that there are some ways of Getting both and skill enhancement by the way we use these tools, but that it's a higher energy state, it's a more unstable equilibrium, it's harder to achieve than just grabbing for the productivity gain.
Matt Beam
00:08:04But savvy well run firms and individuals who are focused on their longer term success and not just short run productivity, are going to start to see data that shows definitively that this is a problem, a serious one, and we'll start to correct for it.
Matt Beam
00:08:21Of course, all of that presumes an incredible explosion of technological advancement.
Matt Beam
00:08:29I just think in 2025 it's going to be the year where anybody who said we've got all we're going to get in terms of generative AI, we had our AI moment and now we have 10 to 20 years to adjust.
Matt Beam
00:08:40Those folks are going to get laughed out of court.
Matt Beam
00:08:42Especially given the quantum computing announcement that just came out of Google a couple of days ago.
Matt Beam
00:08:48I just think we haven't seen anything yet.
Adam Davidson
00:08:51This idea that AI can transform entire organizations, entire ways of working made me think of my friend Rekefet Rusak Aminoach.
Adam Davidson
00:09:02She was CEO of Israel's largest bank, Bank Liyomi.
Adam Davidson
00:09:06Well, she became CEO when it was not Israel's largest bank.
Adam Davidson
00:09:10In fact, it was kind of famously a very old fashioned, bureaucratic, slow moving company that had lost its first place position.
Adam Davidson
00:09:19But Kefet transformed it into a fast moving digital first operation.
Adam Davidson
00:09:24It's, it's a big story in Israel.
Adam Davidson
00:09:27Raket is, you know, she goes into a coffee shop, people are taking pictures, asking for selfies.
Adam Davidson
00:09:32She's sort of a rock star there.
Adam Davidson
00:09:34She told me that she could have done so much more with Bangkl me if she had today's AI.
Adam Davidson
00:09:41In fact, that's what Rick Evatt does now.
Adam Davidson
00:09:43She's managing director of Team8, a major VC firm and venture capital studio where she works in fintech and other fields to help use AI to create really impactful digital transformation.
Rekefet Rusak Aminoach
00:09:59What we can imagine today is far more than what we could have imagined have imagined just a year ago.
Rekefet Rusak Aminoach
00:10:06And I want to give you an example from the world of credit.
Rekefet Rusak Aminoach
00:10:10And I don't know if you remember, but before being the CEO of Bank Lumi, which is the largest financial institution in Israel, I was the chief credit officer of the bank at the time.
Rekefet Rusak Aminoach
00:10:23In order to build the best credit portfolio for the bank risk wise, we needed consultations and credit committees and discussions and we needed people that have the expertise of regulatory requirements and the credit policy of the bank and macroeconomic.
Rekefet Rusak Aminoach
00:10:41Imagine if we had a new customer coming to the bank and ask for $100 million, not a huge amount.
Rekefet Rusak Aminoach
00:10:49Sometimes it took us hours of 20 people in one room in order to reach a decision.
Rekefet Rusak Aminoach
00:10:56It was really blood, sweat, tears and toil.
Rekefet Rusak Aminoach
00:10:59As Churchill said a year ago, when we thought about AI copilot, we could have a copilot of regulation and other requirements and a copilot regarding Greece and the ability to assist us with the credit policy of the bank and with compliance.
Rekefet Rusak Aminoach
00:11:19This in itself was quite amazing.
Rekefet Rusak Aminoach
00:11:24But now when we talk about copilots, this is old news.
Rekefet Rusak Aminoach
00:11:29Now we are talking about a copilot that can mimic the thinking of top credit experts.
Rekefet Rusak Aminoach
00:11:37Can you believe this?
Rekefet Rusak Aminoach
00:11:39We will build a copilot that is mimicking the thinking, the expert level thinking.
Rekefet Rusak Aminoach
00:11:47It's huge.
Rekefet Rusak Aminoach
00:11:49What does this mean?
Rekefet Rusak Aminoach
00:11:50This is the collective wisdom of an organization.
Rekefet Rusak Aminoach
00:11:54When we come back to credit, it means that every junior credit analyst at the bank that doesn't know much, he can actually access all the wisdom of the credit committee.
Rekefet Rusak Aminoach
00:12:09Like having the credit committee in his cell phone.
Rekefet Rusak Aminoach
00:12:14This is the collective wisdom of the organization in his cell phone.
Rekefet Rusak Aminoach
00:12:19And for me it means that we reached a democratization or we are going to reach a democratization of credit underwriting.
Rekefet Rusak Aminoach
00:12:29And now you may ask me, okay, why does this matter?
Rekefet Rusak Aminoach
00:12:35Why should we care?
Rekefet Rusak Aminoach
00:12:37So it matters a lot and for two reasons.
Rekefet Rusak Aminoach
00:12:41First of all, because credit is the engine of economies and it fuels growth, innovation, investment, consumption, everything.
Rekefet Rusak Aminoach
00:12:51And with everything that we discussed now, credit will be more accurate, we will have less human mistakes and the price of credit will go down.
Rekefet Rusak Aminoach
00:13:01And this is huge.
Rekefet Rusak Aminoach
00:13:02But besides, and more importantly, this is just one example, think about what else AI will democratize.
Rekefet Rusak Aminoach
00:13:13Everything I said about credit, you can just say about other domains and in other companies about their domain and their expertise.
Rekefet Rusak Aminoach
00:13:22This is huge.
Adam Davidson
00:13:23Why is the end of 2024 so different from a year ago?
Adam Davidson
00:13:28Ethan Malik, Do I need to introduce him?
Adam Davidson
00:13:30You know who he is, right?
Adam Davidson
00:13:32He's the author of Cointelligence, he's a professor at Wharton and he's most importantly co founder of FeedForward.
Adam Davidson
00:13:38He's probably the reason you're listening to this podcast right now.
Adam Davidson
00:13:42He reminds us that AI today is like an entirely different industry than it was a year ago.
Ethan Malik
00:13:50I think we could divide this into what happened in the technology and business of LLMs over 2024, which was by and large a game of catch up.
Ethan Malik
00:13:59So we started the year with one one GPT4 class model and we ended the year with somewhere between six and nine GPT4 class models and we're beginning to see the new models come out.
Ethan Malik
00:14:12So it was an expansion.
Ethan Malik
00:14:13I think December was a very busy month and something worth thinking more about where there was a release of O1 and all these sort of new kinds of approaches to AI.
Ethan Malik
00:14:20But before that it really was a catch up game.
Ethan Malik
00:14:22At the same time, I think organizationally we saw a catch up game which was people realizing AI is real, it's here to stay, is very capable.
Ethan Malik
00:14:29We saw use spread from relatively low usage rates early on to one third of Americans using AI by August of 2024.
Ethan Malik
00:14:37So I think we saw usage expand and ability expand and access expand over 2024.
Ethan Malik
00:14:42But people still haven't figured out what to do with the system.
Ethan Malik
00:14:45I don't think they're quite ready for what 2025 is going to hold.
Ethan Malik
00:14:48So I think 2025 is going to be about agency, which I think is going to be a very real thing about models outstripping human capacity.
Ethan Malik
00:14:57Not in everything we're not seeing AGI, but in specific areas that used to be human.
Ethan Malik
00:15:01Only the AI will be better than people in, in specific forms of statistics, maybe even math.
Ethan Malik
00:15:07We'll see where things go with that with better the programming than most people, better than writing and most people, not everybody.
Ethan Malik
00:15:14And I think we're going to see this capability game start to become clear to people and that's going to create some real changes.
Ethan Malik
00:15:18And I mentioned agency.
Ethan Malik
00:15:19The idea of AI being able to do things in the world is going to matter a lot.
Ethan Malik
00:15:24And all of the AI companies are racing out models that can do things in the world.
Ethan Malik
00:15:28And I think companionship and connection.
Ethan Malik
00:15:30The idea that all of these systems do real time voice interaction, you can share your screen, you can share a camera with it is going to create new kinds of closeness with AI system.
Ethan Malik
00:15:38So I think AI is going to spread out its tendrils both up the chain.
Ethan Malik
00:15:43In the area where we have increasingly powerful systems doing managerial work, high end scientific work, but also down the chain in terms of being a companion and a connection and something most people turn to to get help with things.
Ethan Malik
00:15:56We'll see what this all means.
Ethan Malik
00:15:57I think that the capabilities are going to continue to have an overhang.
Ethan Malik
00:16:01They're going to do more than we think they can do.
Ethan Malik
00:16:02And in large part the next year is going to unlocking those capabilities for organizations in a way that allows us to start taking advantage of these things and getting a sense of where the future is coming.
Ethan Malik
00:16:13The main thing I don't see is a slowdown.
Ethan Malik
00:16:15I think we are at the early days still of this experience and we still have a ways left to go.
Adam Davidson
00:16:19That point Ethan made about AI companions reminded me of something James Cham told me once.
Adam Davidson
00:16:25James is one of our experts here at FeedForward.
Adam Davidson
00:16:28He runs Bloomberg Beta, an early stage venture fund that focuses on making work more fulfilling for people.
Adam Davidson
00:16:34It's part of the Bloomberg universe and its returns fund, Bloomberg Philanthropies.
Adam Davidson
00:16:40He thinks a lot about this interplay between AI, technology and human flourishing.
James Cham
00:16:49There are probably three things that feel like they're obvious now but go under acknowledged around AI.
James Cham
00:16:58The first is that the LLMs are now probably more emotionally intelligent than the median guy, maybe even more emotionally intelligent than the 90th percentile guy.
James Cham
00:17:15There's a way that if you copied and pasted a chat between multiple people into Claude or ChatGPT and asked it for observations around what was some subtext?
James Cham
00:17:31The LLMs consistently do better than most people, or at least they do consistently better than I do.
James Cham
00:17:40That is also coupled with the fact that if you look at the model card for some of the newer models, when they show you their results around persuasiveness, they're at the 70th or 75th percentile.
James Cham
00:17:55And what that means is that the LLMs are not like Spock or data.
James Cham
00:18:02They're actually much more fluid along the dimension that we think of as human.
James Cham
00:18:07And for some reason that is obviously true.
James Cham
00:18:11And we haven't quite figured out how to look at this clearly you get that uneasy sense, that uneasy feeling when you look at, say, the terrible thing that happened with the kid who killed himself because of character AI.
James Cham
00:18:29And so to me, the interesting opportunity there perhaps is to look clearly at how the LLMs are more emotionally intelligent and persuasive and find unambiguously good use cases for them and find ways to apply that in something that will help rather than hurt humanity.
James Cham
00:18:47So that's the first thing.
James Cham
00:18:49The second thing that feels obviously true is that there's a natural desire right now for us to deploy these AIs towards low risk, low reward at scale tasks.
James Cham
00:19:02And I think that's important.
James Cham
00:19:04The little bits of glue or the customer service thing here and there, I think those are all really valuable.
James Cham
00:19:11But at the same time, and of course I'm speaking now as a venture capitalist, so I'm always looking for the big opportunity or the arbitrage that's going to be around because something's become possible right now.
James Cham
00:19:24It does feel like those sort of high risk, high reward, or those really expensive processes that might Yield big, big rewards.
James Cham
00:19:33It does feel like that's the part that remains underexplored.
James Cham
00:19:37And I think that's because we're reluctant to admit that the models will be better at another thing that we think of as uniquely human.
James Cham
00:19:46But they are quite good at it.
James Cham
00:19:48And there's a way in which just by doing a slight bit of twisting and turning, we can make them surprisingly effective in a few new use cases.
James Cham
00:19:57And then the third thing that I'd say is that the information ecosystem around AI continues to shift in interesting ways.
James Cham
00:20:06If you were to ask me at the beginning of this year what I thought would happen, I would have told you that all of the interesting developments would have happened on the fringe.
James Cham
00:20:17They would happen three blocks away from Berkeley, as some, you know, 26 year old grad student builds a system that uses AI in an original way.
James Cham
00:20:28To be honest, that hasn't happened as much as I thought it would.
James Cham
00:20:31Instead, what's happened is that the development of use cases and the development of finding ways to take advantage of these models, that's actually happened in a much more diverse set of circumstances and all over the place.
James Cham
00:20:49And what that means is that the diffusion of knowledge problem ends up being a little bit harder.
James Cham
00:20:56Rather than being able to rely on San Francisco or Silicon Valley as the origin point of so much interesting stuff, I think we're in this weird place where we are looking instead for signal everywhere.
James Cham
00:21:11And then the hard part there is that so much of the signal is unreliable, which is to say that there's so much bad information and so much slightly askew things that come up as more and more of this gets commercialized.
James Cham
00:21:27And that's of course part of the reason why this entire service is so useful in general, but also so useful to me.
James Cham
00:21:33It's really inspiring to see folks actually building and testing and running and redesigning organizations based on all the stuff that's happening here.
James Cham
00:21:45So I'm glad to be part of it.
James Cham
00:21:46Thanks.
Adam Davidson
00:21:47I absolutely love this last point, that developments in AI are not just happening in a handful of labs somewhere within driving distance of Palo Alto, California.
Adam Davidson
00:21:58No developments are happening all over the globe in a pretty chaotic way.
Adam Davidson
00:22:04There's tens, hundreds of millions of people playing around, trying stuff, stretching AI in ways nobody else had thought of.
Adam Davidson
00:22:11This is a big part of why I love FeedForward.
Adam Davidson
00:22:15We have such a unique vantage point here.
Adam Davidson
00:22:18We have all these companies in different industries trying different things with different approaches, different goals.
Adam Davidson
00:22:24Take the Portland Trailblazers, the NBA team that has won awards for being on the cutting Edge of technology and sport.
Adam Davidson
00:22:32That's in part because of Krista Stout, who leads innovation for the team and is a FeedForward member.
Krista Stout
00:22:38When I think about 2024 in AI, I think the word of the year is probably overwhelming because people like us, who use it regularly for everything, had to keep up with new tools, new releases, new potential use cases, trying to understand how other people are using it and figuring out if we're doing it right, whatever that means.
Krista Stout
00:23:02And people who don't use it seem to be overwhelmed by just AI in general and hearing about it and either still being nervous about it or not knowing how to use it effectively, or trying it a long time ago and not realizing how they could be using it more effectively.
Krista Stout
00:23:17So my hope is that going into 2025, we as a company continue and as a society maybe continue to lean into AI Gen AI in a way that helps all of us individually, both personally and professionally, and then also start discovering more systematically how it's transforming businesses and industries.
Krista Stout
00:23:40I'm still not clear on exactly what the future of the sports industry looks like because of and thanks to Gen AI, but I am quite confident that it will look very different in the coming years.
Krista Stout
00:23:53And again, like, I always go back to how personally Gen can be helpful for me, if it's like taking a picture of my lunch and knowing how much protein is in it and whatever, I think that's still a really important way for people to connect to the technology in a more relatable way that will hopefully unlock ideas for them so that they can apply it to their jobs more usefully.
Krista Stout
00:24:16So going into 2025 for us at the Blazers, I'm really focused on how we continue to use AI to empower people across our organization.
Krista Stout
00:24:25The past year it was very clear that when you give people tools and confidence to experiment, they create solutions that are practical and impactful, whether that's saving time, improving experiences for our guests and fans, or solving new challenges, even just like creating more valuable additions to or more valuable work in whatever they're doing.
Krista Stout
00:24:47And next year I want to keep building on the momentum, scaling what works, getting more people to adopt AI responsibly.
Krista Stout
00:24:55I need to figure out the tool that we're like really going to roll out and access to it in a way that is obviously safe and all of that.
Krista Stout
00:25:04So that's definitely a big focus of ours so that we can see the joy that AI can bring to individuals when they free up time to focus on deep, meaningful work and things that matter.
Krista Stout
00:25:16And also, again, as we Try to start to figure out what this means for our business and our industry more broadly in terms of transformative impact.
Adam Davidson
00:25:25There is a theme developing on this podcast.
Adam Davidson
00:25:28Have you noticed that?
Adam Davidson
00:25:29And it's a great theme.
Adam Davidson
00:25:31It's that AI is at its best when it's aiding people, helping people do their work better, do their work in more satisfying ways.
Adam Davidson
00:25:41That, for me, is one of the big lessons of right this moment.
Adam Davidson
00:25:45A year ago, there was a lot of talk about a future in which human beings are almost unnecessary.
Adam Davidson
00:25:52The idea of a world where AI does everything we do better than we do it.
Adam Davidson
00:25:57Now I hear that talk less.
Adam Davidson
00:25:59I mean, it's still there, but it's not the dominant narrative anymore.
Adam Davidson
00:26:03And you can hear in all of the reflections on this podcast a deeply informed view that AI is best seen as a human enabler, not a human replacer.
Adam Davidson
00:26:17Alex Komorowsky says this particularly well.
Adam Davidson
00:26:20Alex was a legendary product manager at Google, one of the best.
Adam Davidson
00:26:25Then he became head of strategy at Stripe.
Adam Davidson
00:26:28He stepped down not long ago to launch a startup, Common Tools, that has, I think, one of the biggest ideas in AI that I've heard this year.
Adam Davidson
00:26:37And it's, well, you should reach out to him and let him explain it to you.
Adam Davidson
00:26:41But there's a decent chance that he is building the core backbone, the fundamental layer for the future of AI.
Alex Komorowsky
00:26:50I think that 2024 has been all of us in the industry trying to figure out what to use LLMs for, and we're still just starting to crack the surface of what they're Most useful for.
Alex Komorowsky
00:26:59LLMs are their best when you are using them not as an oracle, but as a thought partner where you can bounce ideas off of them like any knowledgeable person, someone who knows a lot about basically any topic, almost certainly more than you on almost any topic.
Alex Komorowsky
00:27:12Not as much as an expert would in that domain, but someone that you can have a great conversation with to bounce ideas back and forth and to come up with a much stronger version of that idea.
Alex Komorowsky
00:27:21I also think of LLMs.
Alex Komorowsky
00:27:23They fill in the gaps of whatever you give them with what I would call like a frog DNA kind of approach.
Alex Komorowsky
00:27:28Like in Jurassic park, they have the dinosaur DNA and it's got gaps in it, and they fill in those gaps in the DNA with frog DNA, just generic reptile, amphibian DNA to make the thing viable.
Alex Komorowsky
00:27:39I think LLMs will give you the most mushy frog DNA answer that is possible to give based on your question.
Alex Komorowsky
00:27:46So the challenge becomes asking it good questions.
Alex Komorowsky
00:27:49How do I encourage it to give you multiple different options to find cross disciplinary insights to tackle problems that you otherwise wouldn't have bothered tackling.
Alex Komorowsky
00:27:59One of the LLM superpowers is their patience.
Alex Komorowsky
00:28:01They don't get tired of doing tasks that real humans would get bored doing.
Alex Komorowsky
00:28:05So if it requires sifting through a whole bunch of information or making connections across material, LLMs are great at finding those insights that a human could find, which has just been taken too much cost and effort and attention to do in the first place.
Alex Komorowsky
00:28:22I think that LLMs are at their best in cross pollination and emergence, helping find the ideas between ideas between different areas of expertise where if you're an expert in any given domain, then you're going to be better at coming up with answers than the LLM can do for sure.
Alex Komorowsky
00:28:37But the vast majority of us are not experts in the vast majority of domains and so having that thought partner along with you I think can be really useful.
Alex Komorowsky
00:28:44The way that organizations work I think will change as more and more people use LLMs to help find ideas and figure out how to sift through them to figure out more efficient ways of communicating inside of an organization.
Alex Komorowsky
00:28:55I think we're just getting started with the LLMs applications and if anything it's going to get weirder and weirder.
Alex Komorowsky
00:29:02And we're just in the early innings today, so I'm extremely excited to see as we get increasingly good performance across increasingly large number of multimodal models.
Alex Komorowsky
00:29:11Again, we're just seeing the beginning and it's pretty exciting right from here, I think.
Adam Davidson
00:29:16Have you noticed how excited everyone on this episode is?
Adam Davidson
00:29:19How positive they are, even recognizing some limitations and some worries?
Adam Davidson
00:29:24That wasn't some plan I had.
Adam Davidson
00:29:26I didn't go out and say, okay, I only want to talk to people who are happily excited.
Adam Davidson
00:29:30I reached out to the smartest minds I know and asked them, well, what's going on?
Adam Davidson
00:29:34What are you thinking about these days?
Adam Davidson
00:29:37A year ago, if I had done this exercise, I think I would have heard a lot more fear and uncertainty.
Adam Davidson
00:29:43There was real fear that AI might slow down and that we had reached the limit of its capabilities or were close to it at the end of 2023.
Adam Davidson
00:29:53And there was even some deep uncertainty about whether or not there's even a real business use case for AI.
Adam Davidson
00:30:00Could AI ever actually move the needle?
Adam Davidson
00:30:02Or is this just another hype cycle that I don't hear anymore?
Adam Davidson
00:30:07I don't hear the idea that it's exhausted its capabilities and I don't hear the idea that there's no business use case, at least rarely I do hear it, I guess every once in a while, but pretty rarely.
Adam Davidson
00:30:20Peter Goldstein is chief product and AI strategist at Hearst, the media company.
Adam Davidson
00:30:25He's also one of the most active and thoughtful members of FeedForward.
Adam Davidson
00:30:29The guy has a PhD in physics, so I don't always understand everything he writes, but I always get something out of it.
Adam Davidson
00:30:36He helped me see in specific ways how AI tools are only going to become more useful for real business use cases in 2025.
Peter Goldstein
00:30:47It's become very, very clear that generative AI can be applied to a variety of use cases that are not just kind of the make my text better, edit what I'm doing, give me better copy, et cetera.
Peter Goldstein
00:31:00That really we're talking about generative AI can be used to solve real business problems, can be used to solve potentially real academic problems, research and the like.
Peter Goldstein
00:31:10That question has been answered pretty affirmatively over the course of 2024.
Peter Goldstein
00:31:15The second thing that I think has been answered pretty affirmatively is that there's no obvious limitation at this point to the capabilities.
Peter Goldstein
00:31:26We continue to see real growth from training time scaling.
Peter Goldstein
00:31:31We see real growth from parameter scaling.
Peter Goldstein
00:31:34I think recent models have made it pretty clear that we have not hit limits on parameter scaling in the sense that we're not using the parameters we have efficiently.
Peter Goldstein
00:31:43And so some of the recent models that have been come out that have much smaller numbers of parameters, down to like single digit billions, are competitive with those with an order of magnitude or more higher number parameters.
Peter Goldstein
00:31:56That strongly indicates that ultimately we'll be able to get much smaller models that are much easier to distribute to a variety of computing platforms.
Peter Goldstein
00:32:05That leads into what I think 2025 will show.
Peter Goldstein
00:32:08I think it'll build on all of those trends.
Peter Goldstein
00:32:11First, I think we will see really extensive rollout of generative AI to applications to real world environment.
Peter Goldstein
00:32:21We'll start seeing it having really obvious effects on how we work, how we live in a way that has only been hinted at in 2024.
Peter Goldstein
00:32:322.
Peter Goldstein
00:32:33I think we will see it really effectively move out to the edge this year.
Peter Goldstein
00:32:38Last year, realistically, if you wanted to use cutting edge generative AI, you had to use an API from one of the labs that was producing it.
Peter Goldstein
00:32:46And I don't think that'll be the case next year, at least by the end of the year.
Peter Goldstein
00:32:51Right now, on my MacBook Pro I can download a model, I can run a model locally, that is, I think, safe to say, sort of a GPT4 level model.
Peter Goldstein
00:33:02That's astounding.
Peter Goldstein
00:33:03So I expect that we will see increased push in that direction to get it onto smaller devices and to be able to do things like frankly run a state of the art model on your phone.
Peter Goldstein
00:33:13I think we'll also see an increased commoditization of the models.
Peter Goldstein
00:33:17In 2024 we saw a bunch of labs the start of the year pretty much it was OpenAI Google, right had state of the art models and Google was a little arguable.
Peter Goldstein
00:33:29At the end of the year we have several additional players, many of which really develop their cutting edge capabilities over the course of less than a year.
Peter Goldstein
00:33:36Xai Amazon both by the end of the year release state of the art models that are competitive with those produced by say OpenAI or anthropic or Google format.
Peter Goldstein
00:33:47And so ultimately I think this commoditization, this drive towards smaller size and the demonstration of these can obviously do real world applications is going to drive a world where Gen AI moves everywhere and over 2025 I don't think that by the end of the year we've completed that, but I think it will become very obvious to everyone that's the future we're rolling towards.
Adam Davidson
00:34:10The future we're rolling towards.
Adam Davidson
00:34:12This is a great place to end.
Adam Davidson
00:34:15We're going to end with one last comment from Jessica Johnston.
Adam Davidson
00:34:18You all surely know Jessica.
Adam Davidson
00:34:21She is one of the co founders of FeedForward and is our primary interactor with our membership.
Adam Davidson
00:34:28Jessica has been thinking a lot about the role of FeedForward in the work and lives of our members in 2025.
Jessica Johnston
00:34:39I have had a slide up on my computer since the early days of feedforward with a few notes that I think have proved valuable not just in the last few months of 2024, but I think will be really important heading into 2025.
Jessica Johnston
00:34:58One is the need to be consistently searching at the edges is going to be crucial for business leaders who are on the hook to lead their companies AI initiatives, AI strategies, not just the thinking in their organizations but ultimately the decisions and the actions and the investments that they take.
Jessica Johnston
00:35:17A couple other notes that that are on on the slide on my computer.
Jessica Johnston
00:35:21The other is that constant sensing for early warning signs is key.
Jessica Johnston
00:35:27There have been times when a new model drops like Sora and I'm going well how important is that really?
Jessica Johnston
00:35:34If you are an executive at jpmort, how do we think through what model updates matter, what to pay attention to and so on.
Jessica Johnston
00:35:45But if I zoom out I think it is about understanding the bigger picture that updates continue to happen, the capabilities continue to grow and it matters when a PELICAN increasingly can ride a bicycle and easily my favorite benchmark for testing the capabilities of new models.
Adam Davidson
00:36:07I'm just going to jump in here quickly to explain.
Adam Davidson
00:36:09Our fabulous feedforward expert Simon Willison has this simple, lovely benchmark he does when any new model comes out.
Adam Davidson
00:36:17He asks it to create an image of a pelican riding a bicycle.
Adam Davidson
00:36:22He does that for the image generators, but he also does it for the text generators.
Adam Davidson
00:36:26He uses text based image generation or asks them to A year ago they were pretty lousy at this.
Adam Davidson
00:36:33No model could do a good job of creating an image of a pelican riding a bicycle.
Adam Davidson
00:36:37Now they almost all do at least an okay job and some will really blow you away.
Adam Davidson
00:36:44I love this benchmark as Jessica does, as everyone does.
Adam Davidson
00:36:48It's just a quick, funny way to instantly get an image that tells you is this model doing something fundamentally more advanced than previous models?
Adam Davidson
00:36:59I think all of us should have our equivalent of a pelican on a bicycle.
Adam Davidson
00:37:05What is our quick and easy way in a couple minutes to find out if a new model is actually going to be able to do things we need it to do better than previous models.
Adam Davidson
00:37:16So come up with your own.
Adam Davidson
00:37:17Or you know, I think Simon's fine if you just steal his idea.
Jessica Johnston
00:37:22And then My last note that that I keep is that questions that the leaders ask will shape what organizations pay attention to.
Jessica Johnston
00:37:31I'm keenly aware of the information that we are exchanging, the insights that we are sharing.
Jessica Johnston
00:37:37How is that informing the questions that we can ask inside of our organizations to help drive the right decisions?
Jessica Johnston
00:37:46Again, regardless if we are a financial services firm, media organization, manufacturing consumer goods, all the things.
Jessica Johnston
00:37:54So wanted to make sure I shared just some of my concluding thoughts in 2024 and the the notes that will continue to underpin a lot of the work that I do with our members in 2025.
Adam Davidson
00:38:08Thanks so much for listening to the last episode of 2024 of the FeedForward podcast.
Adam Davidson
00:38:15We are so thrilled by this community.
Adam Davidson
00:38:18One big thing, maybe the biggest thing that happened to us this year is the feedforward community.
Adam Davidson
00:38:24Earlier in the year we were dreaming of it, hoping for it, and it now is flourishing and just barely getting started.
Adam Davidson
00:38:33We're going to have a whole lot of new members, whole lot of new events, new things to give to you to help you do your job more effectively and we hope, a whole lot more discussion with you, a whole lot of conversations and interactions that help us all better understand AI, better understand how it can contribute to business success and to human flourishing.
Adam Davidson
00:38:56So I hope you have very happy holidays.
Adam Davidson
00:38:59Merry Christmas, happy Hanukkah, happy New Year, happy everything.
Adam Davidson
00:39:02I'm Adam Davidson.
