WEBVTT
00:00:00.833 --> 00:00:04.419
Hi, it's Dustin Lanier.
00:00:04.559 --> 00:00:05.301
Thanks for listening.
00:00:05.883 --> 00:00:16.440
Please find me on LinkedIn for original public sector operations content every week and please reach out to me if I and my team of procurement professionals at Civic Initiatives can help you be a public procurement change agent.
00:00:20.193 --> 00:00:20.474
Okay.
00:00:20.493 --> 00:00:22.736
Hey, thanks everybody for joining us today.
00:00:22.777 --> 00:00:24.239
I'm Dustin Lanier.
00:00:24.318 --> 00:00:25.881
I'm founder of Civic Initiatives.
00:00:26.402 --> 00:00:44.606
If you're really interested in these topics and you want to talk afterwards, either about some of the speaking that I'm doing on these topics at procurement conferences or the potential to do some work for you in the topics that we talked about, please go to procurementagent.ai, which will take you to a page on the Civic website where you can say you'd like to talk.
00:00:45.121 --> 00:00:51.229
With that as preamble out of the way, let me introduce Steve Meyer, who's the head of innovation for Kung Fu AI.
00:00:51.430 --> 00:00:54.234
Steve, why don't you introduce yourself and Kung Fu AI?
00:00:54.335 --> 00:00:56.578
And obviously people want to know about the fun name.
00:00:56.677 --> 00:00:59.241
So maybe talk a little bit about the company as well.
00:00:59.841 --> 00:01:00.343
Hi, everyone.
00:01:00.523 --> 00:01:01.223
Thanks, Dustin.
00:01:01.424 --> 00:01:02.246
It's exciting to be here.
00:01:02.265 --> 00:01:02.826
This will be fun.
00:01:03.267 --> 00:01:04.368
Like you said, my name is Steve Meyer.
00:01:04.388 --> 00:01:06.391
I'm one of the co-founders of Kung Fu AI.
00:01:06.492 --> 00:01:07.974
Non-traditional background.
00:01:08.353 --> 00:01:22.109
I'm a co-founder, I'm a non-technical co-founder, spent about a decade prior to starting Kung Fu AI in the Fortune 500 tech sector, working for a boutique consulting firm, predominantly with IBM and IBM Watson Groups.
00:01:22.150 --> 00:01:31.981
Got to cut my teeth in AI back in 2015 to 2017 period, helping IBM apply IBM Watson across a couple of interesting use cases.
00:01:32.481 --> 00:01:40.453
So fell in love with it so much so that I took a risk and decided to start a consulting practice around artificial intelligence.
00:01:41.474 --> 00:01:43.537
And that's exactly what we do.
00:01:43.759 --> 00:01:47.424
We help our customers both on the strategy and engineering side.
00:01:47.463 --> 00:01:54.754
And the name is an interesting story in that most people confuse Kung Fu with a martial art.
00:01:55.436 --> 00:02:27.204
And while that's true, it's actually a very narrow use of the word where Kung Fu actually is more often used, especially in Eastern cultures, as a verb and a way and a process of acquiring any great skill or knowledge over many years in hard work, in discipline, in practice, which in our estimation, we feel like is the road to artificial intelligence mastery is it's time in practice and like a good Kung Fu master, we can teach it to you and fight for you today.
00:02:27.444 --> 00:02:31.115
But our hope is that you will acquire the skill yourself over a period of time.
00:02:31.215 --> 00:02:33.199
So we thought it was appropriate for the company name.
00:02:33.981 --> 00:02:34.543
Very interesting.
00:02:35.025 --> 00:02:35.526
Great answer.
00:02:35.545 --> 00:02:52.562
I found you through a common friend a couple of months ago when we started collaborating and run up to the speech that I did at NIGP Forum, which was called Generative AI, the new tool in the toolkit, where I was trying to take the topic and present it at layman plus.
00:02:52.942 --> 00:03:03.251
So coming down from you guys as deep implementers, but bringing it higher than just the, sometimes the relatively simplistic stuff that you can see on LinkedIn.
00:03:03.491 --> 00:03:06.993
So had a lot of good feedback off of that presentation.
00:03:07.094 --> 00:03:13.943
I've already updated it twice I'll be doing that at other procurement conferences around the country, both this year and into next year.
00:03:13.984 --> 00:03:15.466
And there's a lot of excitement.
00:03:15.846 --> 00:03:28.564
I would say as much excitement on this topic as anything I've seen in several years of being involved in content, both for the application for people individually and then organizationally, the opportunity that AI can present.
00:03:29.066 --> 00:03:34.813
And so let's take it on the individual basis first, and then we'll talk about the organizational implications first.
00:03:35.394 --> 00:03:47.153
I think what has really astounded me as far as the difference in this piece of technology, as you mentioned Watson and other things that have been there before, is just how unbelievably accessible it is.
00:03:47.193 --> 00:03:59.977
You gave me a graphic that I have in the presentation that talks about this evolution and that right now we're able to provide unstructured data and get an unstructured response, which is new.
00:04:00.296 --> 00:04:08.731
And the barrier to entry is so low for an individual to simply go to one of those pages and to be able to type simple questions or incredibly complex prompts.
00:04:09.212 --> 00:04:16.848
So if anybody on here has not spent time in it, There's a lot that we could talk about, about images and video, but for this conversation, we're primarily talking about text.
00:04:17.389 --> 00:04:19.572
Go to chat.openai.com, play with it.
00:04:19.591 --> 00:04:22.696
There's other tools, but you'll be blown away if you haven't played with it yet.
00:04:22.815 --> 00:04:29.605
So Steve, describe what's happening to somebody who's interested, but hasn't really dipped their toe in their water.
00:04:29.665 --> 00:04:36.975
How would you describe this to your mom or your aunts or your long lost high school buddy who's like, what is all this and what's happening?
00:04:37.716 --> 00:04:38.476
Yeah, it's...
00:04:39.041 --> 00:04:46.711
really profound in the fact that this unstructured data in, unstructured data out, in that that's what we do.
00:04:47.273 --> 00:04:58.666
We take in visual cues, audio data, the world around us, our five senses, and output something unstructured in return, an opinion, our voice, our thoughts.
00:04:59.889 --> 00:05:15.153
And that's in essence why this is becoming more accessible in that we can actually interface in natural language with a artificial intelligent agent and then in return get some sort of intelligent response back.
00:05:15.473 --> 00:06:02.987
And I think what has really put a spotlight on the field of artificial intelligence and the need for artificial intelligence is the fact that any one of us can go to this virtual agent and have what seemingly an intelligent conversation with it ask questions and get a better response than we would ultimately get from a search query and effectively what's happening is there was this aha moment back in 2017 that computers are better with numbers and they are words and that's not really the aha moment but translating the words back to numbers and effectively taking your natural language query and converting it into a point, like you put on a graph.
00:06:04.149 --> 00:06:07.353
You think of your x and your y-axis, you put a point on a graph.
00:06:07.392 --> 00:06:12.139
Well, this graph just happens to have unlimited planes.
00:06:12.920 --> 00:06:18.625
And by doing that, you can start to put words as a point on a graph and start to see patterns.
00:06:19.146 --> 00:06:25.555
And now our words are mathematical input, and sentences can be math equations.
00:06:25.714 --> 00:06:59.033
And all of a sudden, the computer can predict what the correct answer is to your question so all of a sudden your words and response become a prediction problem and it keeps predicting what they think is the natural output and it's getting very very good at that and it can even go so far as start calculating the distance of points to start to understand patterns in new ways and what that means to us is that we can get really sophisticated analysis with very little effort and zero programming and that's just a just door or buster when it comes to this space.
00:06:59.153 --> 00:07:01.838
So it's very, very exciting.
00:07:01.898 --> 00:07:07.706
It's making AI really useful to everyone with a very low barrier to entry for the very first time.
00:07:08.968 --> 00:07:09.730
Great answers again.
00:07:09.769 --> 00:07:14.737
I'm going to have to listen back and pepper in a little bit of that sophistication into my core speech.
00:07:14.817 --> 00:07:18.002
I like the way you talk about translating sentences into equations.
00:07:18.083 --> 00:07:20.225
I've said it similarly, but you said it better.
00:07:20.305 --> 00:07:22.730
So thanks definitely for that viewpoint.
00:07:23.074 --> 00:07:27.346
So let's stay in the individual impact before we go into the organizational impact.
00:07:27.786 --> 00:07:55.151
So I described three things in the speech that I think public procurement, since primarily our audience is public procurement, although public sector in general, I think these also apply, that it's Excellent at synthesis, the ability to take extremely large data sets and to be able to search and remodel and remold it to produce everything from action items to synthesized groupings of information in a way that makes it so much easier to go through gigantic data sets.
00:07:55.572 --> 00:07:57.093
The best brainstormer of all time.
00:07:57.134 --> 00:08:00.036
It means that you never have to start with a blank piece of paper.
00:08:00.076 --> 00:08:10.266
You can always describe a scenario that you're trying to build a document or a piece of content against a requirement set against for from a specification perspective.
00:08:10.286 --> 00:08:17.153
And you get back information that is actionable in a sense that it's at a place where it can be edited.
00:08:17.353 --> 00:08:23.822
And myself personally, I'm a much better editor than I am sitting with a blank piece of paper and writing it out every time.
00:08:23.841 --> 00:08:40.759
And then the third was an on-demand quant that the ability to do certain mathematical things such as Excel formulas on hand for the rest of time, being able to do math within the tool, being able to do things like coding and actual programming and SQL data sets.
00:08:41.200 --> 00:08:51.297
So those were the three areas that I thought had the most impact for individuals who are trying to find where's the edge of this rock that they can pick up from an individual tool perspective.
00:08:51.717 --> 00:09:00.974
So talk about any of those three or things that you would add or how you would kind of say, what is at an individual level, what does it provide as a tool set today?
00:09:01.985 --> 00:09:03.707
No, I think those are really good.
00:09:03.748 --> 00:09:07.753
And where else I may highlight is analyst.
00:09:08.674 --> 00:09:21.350
So what effectively is these large language models, these GPTs, these LMs, they are a magical analyst that can do the work of an army of analysts in many respects.
00:09:22.091 --> 00:09:28.198
So the ability to take a sophisticated question, should I buy from this company?
00:09:28.278 --> 00:09:30.101
Should I invest in this company?
00:09:30.946 --> 00:09:38.738
There is quite a bit of research that is involved in understanding the company's viability, the price of the product.
00:09:38.798 --> 00:09:39.840
Is it a fair price?
00:09:41.201 --> 00:09:43.645
Will they deliver if we purchase?
00:09:44.826 --> 00:09:47.291
Are they a company that's growing and is sound?
00:09:47.971 --> 00:09:50.416
That could be eight hours worth of research.
00:09:50.716 --> 00:10:01.768
You can effectively whittle that down into eight minutes of carefully curated prompts or questions to a language model And that's magical, right?
00:10:01.927 --> 00:10:09.775
It might take an army of analysts to take that level of question and get you a response back in eight to 10 minutes.
00:10:10.255 --> 00:10:15.561
So I think analyst is really important, researcher, and another one I'd add is monitor.
00:10:16.081 --> 00:10:42.881
So if you want to understand if certain phenomenon are present in a large volume of data that's constantly being ingested into your business, think of documents where anomalies or fraud may occur, these models are very good at looking for patterns of phenomenon and being able to detect them where you might have an army of researchers who are constantly looking for changes in regulation or potential fraud issues.
00:10:43.341 --> 00:10:46.926
So I think those are also capabilities that we should be tracking.
00:10:48.849 --> 00:10:49.028
Excellent.
00:10:49.048 --> 00:10:57.145
So moving organizationally, so when Organizations are trying to take on a new piece of technology.
00:10:57.186 --> 00:11:08.668
They're always, of course, going to rightfully try to make sure that they have it set up to be able to be successful at it and that they're not walking into something that they can't manage.
00:11:08.849 --> 00:11:17.037
And so there's things that we can talk about, things like there's an ability for the data to be wrong because it's making estimations.
00:11:17.177 --> 00:11:24.683
So those are called hallucinations, which I find you can tend to see those relatively quickly, but that's been a concern set.
00:11:25.043 --> 00:11:31.931
But probably the number one concern I've heard is just the concept of what happens with the data when I create a query.
00:11:32.270 --> 00:11:41.663
So if I make a query that says, I want to take notes from an evaluation committee and to be able to drop them in in order to pluck out action items.
00:11:42.404 --> 00:11:44.006
Is it possible to query the query?
00:11:44.126 --> 00:11:48.851
Can somebody come in and say, have there been any evaluation meetings from X or Y or Z?
00:11:49.312 --> 00:11:55.577
And I think that that concern is one of the main ones that I think people have to kind of get their heads around.
00:11:55.597 --> 00:12:08.883
I personally feel like from what I've understood that the query is not retained in that specific way in the sense that It is presenting a question that produces the mathematical formula, as you said, and then it produces a mathematical formula back.
00:12:09.264 --> 00:12:15.197
I think we all come from paradigms where we're concerned about a large data set sitting out there that is exposable.
00:12:15.905 --> 00:12:17.327
So let's talk about it in two ways.
00:12:17.828 --> 00:12:21.014
One is just simply going to the open tools that are out there now.
00:12:21.335 --> 00:12:28.645
You know, what is a good data hygiene process on being able to use the tool and not feel like you're creating giant stores of information?
00:12:29.086 --> 00:12:38.480
After you answer that, then let's talk about how we can establish a more limited access environment that maybe amps up the kind of concerns or questions that people may have on that topic.
00:12:39.462 --> 00:12:40.264
Yeah, absolutely.
00:12:40.303 --> 00:12:47.653
Well, at the 10,000 foot level, we get pulled into a lot of LLM advisories.
00:12:47.953 --> 00:12:50.758
Hey, help us understand this model versus that API.
00:12:50.898 --> 00:12:54.001
It's truly a build versus API conversation.
00:12:54.022 --> 00:12:57.966
And the answer is yes, likely you need both.
00:12:58.748 --> 00:13:04.434
And these APIs that are available to us today are, I think are getting often a bad rap.
00:13:04.676 --> 00:13:10.895
It's unfair to call them insecure when it comes to certain types of topics.
00:13:10.975 --> 00:13:19.655
I think the more sensitive the topic and the higher risk the scenario that maybe we would like to leverage a large language model, more oversight is certainly required.
00:13:20.197 --> 00:13:25.528
But there's going to be situations where you can leverage these APIs in many situations.
00:13:25.730 --> 00:13:29.217
And there's going to be scenarios where you're going to want to use open source or something more private.
00:13:29.601 --> 00:13:41.777
But I will say that specifically some of the big guys in this space that are providing great APIs are adding additional guardrails to make things more private and secure if that is a big concern.
00:13:41.917 --> 00:13:55.253
But to your point, Dustin, it is basically scattershotting your input into a bunch of mathematical equations and units and then constructing the best response based on a prediction problem.
00:13:56.001 --> 00:14:06.586
In terms of it, keeping a memory or a log of what exactly you give it isn't quite fair, but there are certainly concerns and reasons why sometimes you may want to be private.
00:14:08.009 --> 00:14:16.378
Before we go to private, let me just sort of pivot the scenario just a little bit, and hopefully you can tell me if my scenario statement is correct.
00:14:17.359 --> 00:14:27.871
I think that government is used to potentially a scenario that says, here's a Word document that includes something that might be lightly sensitive, and I'm going to send it to an individual to do a transcription.
00:14:27.912 --> 00:14:35.182
So I send the document, the document is received by an individual, the individual does a transcription, and then they send me back the transcription.
00:14:35.542 --> 00:14:40.143
So in that scenario, in use case one, There was a full document.
00:14:40.547 --> 00:14:41.355
It went somewhere.
00:14:41.793 --> 00:14:49.260
And it has a possibility to be retained as the document, which is the four corners of the document with the words in the middle.
00:14:49.682 --> 00:14:54.746
So that would generate that concern of the document is existing in the universe.
00:14:55.287 --> 00:15:05.537
In my understanding, when we're doing a query into an AI scenario, I think I joked with you, it's atomizing Captain Kirk and reappearing him on the face of the planet.
00:15:05.616 --> 00:15:06.717
Is it the same Captain Kirk?
00:15:06.817 --> 00:15:09.421
Or did we kill Captain Kirk and rebuild him on the planet?
00:15:09.921 --> 00:15:18.120
Because The document, when it hits the query, it's being atomized into a math equation and then a math equation is coming back.
00:15:18.240 --> 00:15:31.910
And that's what I've been trying to explain to people that if we're thinking we're in the situation where we've sent a full document to somebody for a transcription event and we're trying to make that be an exact equivalent to what happens when we do a query on AI, it's not an equivalent.
00:15:32.451 --> 00:15:42.763
I think what I would add to that, it's definitely a yes and, what I would add to that is when you are training models is when the model is learning off information.
00:15:42.783 --> 00:15:50.096
Oftentimes when you are interfacing with an API, you're using inference or prediction but you're not training the model.
00:15:50.216 --> 00:15:56.342
So basically you are showing it information and saying, Hey, give me a response based on what you see.
00:15:56.403 --> 00:16:01.229
We're not actually putting the information back into the API or the model.
00:16:01.750 --> 00:16:10.500
Now there are certain situations where in the terms of use, you are in fact training the model, but in most instances, the model is pre-trained.
00:16:10.841 --> 00:16:14.706
It has already learned all that it can learn for the time being until the next release comes out.
00:16:15.042 --> 00:16:18.365
And you're basically showing it information and it's making a prediction.
00:16:18.405 --> 00:16:24.975
And that prediction is a response, which is kind of funny to think of a dialogue with a virtual agent as a prediction problem.
00:16:25.054 --> 00:16:28.440
But in actuality, everything we do is a prediction problem.
00:16:28.899 --> 00:16:29.942
Dustin, I hear your question.
00:16:30.121 --> 00:16:32.725
I predict what you and the audience may want to hear.
00:16:32.745 --> 00:16:34.168
And then I give a response.
00:16:34.227 --> 00:16:35.089
And sometimes I'm right.
00:16:35.548 --> 00:16:36.650
Sometimes I'm not so right.
00:16:36.750 --> 00:16:39.914
But everything can be framed fundamentally into a prediction problem.
00:16:39.955 --> 00:16:44.662
And that's effectively what we're doing here, more so when we're using APIs.
00:16:45.153 --> 00:17:06.153
Well, in the speech that I give at the simplest level I can do to talk about how this watch works before we talk about what time it is, I bring up a clip from one of the websites that says if we were to create a prediction formula that says if the words once upon a were in a row, we know that the next word is probably not armadillo and it's probably time.
00:17:06.882 --> 00:17:15.775
but then the ability to build those prediction models are based across a huge level of deep multi-planal learning as you referenced.
00:17:15.875 --> 00:17:27.992
And so to the degree that we're teaching the model something about our specific query, it's teaching it how to refine the future responses, but it's not necessarily teaching it that here's the notes from the evaluation committee.
00:17:28.193 --> 00:17:31.598
It's teaching it how to further refine a mathematical model.
00:17:32.140 --> 00:17:33.582
Let me give you an analogy.
00:17:34.049 --> 00:17:39.758
Think about yourself when you were somewhere between 15 and 18 years old and you were learning to drive a car.
00:17:40.779 --> 00:17:45.826
And you are taking the wheel for the first time, maybe even getting on the interstate or the highway for the first time.
00:17:45.967 --> 00:17:47.148
Your palms are sweaty.
00:17:47.189 --> 00:17:50.053
You're alert to absolutely everything.
00:17:50.093 --> 00:17:53.498
Perhaps you're maybe too alert to your surroundings around you.
00:17:53.890 --> 00:17:58.856
And it's a very tense moment, but you're absolutely learning as you're driving that vehicle.
00:17:58.958 --> 00:18:01.821
And all that processing is creating a lot of the anxiety likely.
00:18:01.862 --> 00:18:06.689
Now, fast forward 20, 30 years and think about driving today.
00:18:06.729 --> 00:18:11.636
You're not necessarily learning anything, but you're doing fundamentally the same action.
00:18:11.656 --> 00:18:15.362
You're identifying, you're predicting, you're reacting.
00:18:15.402 --> 00:18:18.467
But all of that learning has kind of hit its threshold.
00:18:18.487 --> 00:18:20.329
You're no longer training, you're doing.
00:18:20.930 --> 00:18:26.882
I think in some respects, you can think of some of these more prefab models as a similar experience.
00:18:27.022 --> 00:18:30.630
They've gone through the training mechanisms of learning to drive the car.
00:18:31.090 --> 00:18:32.394
Now they're just executing.
00:18:33.015 --> 00:18:40.289
So hopefully that resonates, but it felt good when I was thinking about it.
00:18:40.642 --> 00:18:41.202
No, I like it.
00:18:41.222 --> 00:18:41.663
That's great.
00:18:41.683 --> 00:18:42.684
That's absolutely great.
00:18:43.045 --> 00:18:58.248
So now let's talk about if somebody is going to make an enterprise investment or they want to really say, I want to do this, but I want to do it in an area where I have some controls on the outside edge of the fence line and I want to be able to have this be more secure.
00:18:58.327 --> 00:19:08.126
So what kind of options are there for somebody who, you know, they say it may be all well and good for an individual to use this over here, but what I want to deploy something that I'm confident in.
00:19:08.548 --> 00:19:13.135
What does that even mean from a deployment perspective?
00:19:13.175 --> 00:19:19.448
And what are some of the, how does somebody raise that up a level through something that would be in their own control space?
00:19:20.770 --> 00:19:21.772
Yeah, absolutely.
00:19:21.913 --> 00:19:28.305
So you definitely have the ability to weigh out those trade-offs, build versus API, right?
00:19:28.769 --> 00:19:48.057
And likely you will have API use cases that make a ton of sense because you want the capability, the cost makes sense for you, the reduction to get to production is there, which is always very, very important and useful, especially when you're trying to solve a hard problem as quickly and cost effectively as possible.
00:19:48.538 --> 00:19:53.086
But then there's going to be a lot of instances too, where you've decided that you want to use an open source model.
00:19:53.506 --> 00:20:06.282
You want to host that model inside your environment, and you want to train that model to perform a very specific task at a very, very high level, in which case you will invest more to get that model trained.
00:20:06.863 --> 00:20:14.113
However, the trade-off is you may get additional capability than what might otherwise be available and a private and secure environment.
00:20:15.329 --> 00:20:21.602
The open source community is constantly evolving and the state of the art is moving very, very quickly.
00:20:21.642 --> 00:20:28.213
So you can use a model that may be effective today and then use another one that may be more effective into the future.
00:20:28.674 --> 00:20:29.958
But those are some of the trade-offs, right?
00:20:29.998 --> 00:20:33.845
If you want fast and ready, API is going to be the way to go.
00:20:33.884 --> 00:20:38.374
And there's going to be use cases where it's low risk and privacy is less of a concern.
00:20:38.913 --> 00:20:45.344
And then if you want powerful and private, there's going to be more cost associated and effort associated with getting there.
00:20:45.804 --> 00:20:49.509
But likely the answer is, well, we kind of need both or something in between.
00:20:51.311 --> 00:20:53.315
Well, let's stay kind of within that.
00:20:53.596 --> 00:21:08.415
And organizationally, I know that your organization has had an opportunity to work with some of the federal government partners on different kinds of projects within the DOD zone or some higher ed institutions and laboratories and so forth.
00:21:08.736 --> 00:21:25.729
So whatever you're able to talk about and you can feel free to extrapolate it, you don't have to name individual projects, but maybe share a bit of perspective on the kind of use cases that you've been seeing from governmental clients and where you think some of the opportunity lies for government in this space.
00:21:27.041 --> 00:22:02.255
yeah there's a lot of fascinating work out there we had the opportunity to work with a major university here in the area and work with their career services and help them with labor prediction so the thinking was that their career services think of again we can create this managed magical analyst right these career services folks are spending tons of times looking at labor statistics and forecasts trying to understand what companies and sectors are more likely to grow, so they can invest their time in making the right partnerships, so they can get their students better career service advice.
00:22:02.654 --> 00:22:11.913
Hey, let's steer your curriculum in this direction, get better internships, and ultimately at the end of the day, better employment, which is a lot of what these universities are graded on.
00:22:12.385 --> 00:22:29.651
So being able to use data and machine learning and put a whole bunch of really interesting data sources together and make a prediction that is more effective than what our human brains can do to understand, no, you need to think about this sector and these companies.
00:22:29.692 --> 00:22:39.246
And oh, by the way, there's a shift going on where certain skill sets that existed yesterday that were in high demand are actually moving in another direction entirely.
00:22:39.842 --> 00:23:06.650
that becomes a really interesting one another example especially in the government space we see a bunch of document data problems meaning we have a lot of documents and a lot of data inside those documents and the ability to interrogate that data understand patterns of that data maybe even automate processes that are very paper driven become the concern and getting that information out is very, very difficult.
00:23:06.930 --> 00:23:22.402
So especially using large language models, this becomes a lot easier than how it was when we started working on these problems six years ago, where we can actually drop documents in with very little training and get insights out with simple questioning.
00:23:22.817 --> 00:23:28.567
which becomes very, very powerful, both when you're trying to understand is a phenomenon present in a document?
00:23:28.967 --> 00:23:31.070
What are the trend lines in this document?
00:23:31.590 --> 00:23:37.681
Or perhaps if something happens, we need to move it to this part of the process versus that part of the process.
00:23:37.701 --> 00:23:40.705
So it's really an intelligent triage type of use case.
00:23:40.746 --> 00:23:44.892
And there's plenty of case management issues out there where this becomes really, really right.
00:23:45.172 --> 00:23:56.461
Or if we just want help with research or analysis, another great example of where these things can play You know, we are seeing a tremendous amount of appetite and demand for strategy work.
00:23:56.800 --> 00:24:02.568
We need a position, we need to know the use cases, and we certainly need a lot of governance.
00:24:02.669 --> 00:24:08.457
And we have to understand when these models are performing or not performing, which means we need test and evaluation criteria.
00:24:09.377 --> 00:24:14.766
This is becoming more important for all organizations, especially as we move into generative, because it's inevitable.
00:24:15.201 --> 00:24:17.986
We're all going to have it and use it sooner rather than later.
00:24:18.026 --> 00:24:39.296
But to do this in the best possible way to get the most performance with the least amount of risk, we need to have a plan in place and doing that work, whether you do that work and inform us and people like us go execute, or we inform you and figure out if we or you can go execute, it becomes critical to doing this well.
00:24:40.317 --> 00:25:13.137
And in my speech, there's some areas that We're very interested in, as people who work with public procurement shops around the country, one is we've been deploying in pilot form generative AI chatbots that are trained within a body of knowledge against some of the procurement manual works that we do, which is an interesting alternative way to be able to do searches and generate different kinds of information so that it's more accessible both to people who might be answering questions of end users as well as being able to provide self-service to end users.
00:25:13.538 --> 00:25:23.413
Chatbots have been around a long time, of course, but they tended to be very rigid and very quickly got you to the wrong answer in a way that we've really been finding some success on.
00:25:23.835 --> 00:25:36.295
We're also interested in playbooks around aspects of handling requisition management, some of the solicitation activities, some of the spend activities as something that will be continuing to try to build around.
00:25:36.395 --> 00:25:41.705
So I think there's any number of possible use cases for this kind of tool set.
00:25:41.826 --> 00:25:55.558
And in the speech, I also talked about how one of my closest mentors and friends on overall technology had said he didn't think we would have access to some of this kind of stuff in our lifetimes, which I thought was pretty remarkable.
00:25:55.618 --> 00:25:59.063
So lots of really, really interesting options and opportunities here.
00:25:59.083 --> 00:26:02.986
I might add two others to that because that triggered a memory.
00:26:03.487 --> 00:26:22.785
We've done some work specifically on the commercial side of procurement, great use cases, spend analysis and spend optimization, and being able to look at all of our suppliers and understand perhaps when have we double paid or when are we paying two suppliers for the same thing and we have a surplus?
00:26:23.224 --> 00:26:24.586
Where is the anomalous spending?
00:26:24.727 --> 00:26:27.189
Oh, we don't typically spend that much in this category.
00:26:27.249 --> 00:26:27.970
That's a bit odd.
00:26:28.470 --> 00:26:29.590
These are all things we can catch.
00:26:29.832 --> 00:26:31.512
It just takes a lot of work to do it.
00:26:31.833 --> 00:26:38.952
And we can really reduce that workload and even automate some of those capabilities The other part too is the optimization.
00:26:38.972 --> 00:26:41.436
So the next level of intelligence is great.
00:26:41.797 --> 00:26:42.817
Where are we overspending?
00:26:42.857 --> 00:26:44.820
Then you ask a better question.
00:26:44.980 --> 00:26:46.683
How might we spend more effectively?
00:26:47.265 --> 00:26:51.951
And being able to optimize spend and becomes really important.
00:26:52.070 --> 00:26:56.657
Now, the one thing I want to point out there is that language models aren't the silver bullet.
00:26:56.778 --> 00:27:02.967
There's really no silver bullet in AI, but they do provide a baseline capability, which you can put other AIs on top of.
00:27:03.446 --> 00:27:08.144
So the first question we ask is, where is the anomalous spend happening?
00:27:08.164 --> 00:27:12.308
The next question we might ask is optimize my general spending habit.
00:27:12.490 --> 00:27:17.295
And one AI is built on top of the other to do both levels of sophistication.
00:27:17.615 --> 00:27:21.819
They're using functionally very different artificial intelligence algorithms.
00:27:22.140 --> 00:27:29.428
But when you start to sequence these things together, we can get more and more sophisticated use cases and capabilities.
00:27:30.169 --> 00:27:40.936
We'll sit in that for a minute because I think what's really important about some of these tools is The AI is there, in my opinion, as a helper for human beings to be able to do some of the more strategic pieces of work.
00:27:41.498 --> 00:28:00.859
And specifically in the case of spend analysis, which we've done a lot of, if you have shops that have always been relatively transactional then frankly, sometimes there isn't necessarily even a need for spend analysis because it doesn't affect the order in which you do things or potentially doesn't specifically affect the go-to-market strategy.
00:28:01.280 --> 00:28:10.893
And so a lot of times if you're going into a procurement shop that hasn't had the opportunity to do spend analysis before, what we're trying to do is to democratize the access to data.
00:28:11.354 --> 00:28:18.463
We want it so that it's not that only one person knows how to get the data out of the ERP and he's not even a procurement person, so it didn't matter anyways.
00:28:18.884 --> 00:28:31.720
We're trying to say, we can teach a persona or an agent, in that case, a virtual agent, to be able to say, what are the kind of trends that we're looking for that would affect our go-to-market strategy or our negotiating approach?
00:28:32.561 --> 00:28:38.250
And then being able to feed it with information, and it doesn't necessarily have to go all the way to the end.
00:28:38.611 --> 00:28:42.376
Like you mentioned, having a second AI that could say, well, how do I optimize against that?
00:28:42.849 --> 00:28:53.178
There could be a point where even just simply saying we're trying to democratize the access to data so that we're teaching our buyers how the information that we have in our systems affect our choices.
00:28:53.579 --> 00:29:20.445
And then we don't have to teach them how to do all of the smallest micro list pivot table infinity but we can say here's an ability to have a tool that can provide an answer to this question and here's what you as the human being do with that information in order to be good at procurement that would dramatically expand the role in the nature of spend analysis and procurement in a way that would be very fundamentally impactful a lot of procurement shops And so then that way, that's a great example.
00:29:20.465 --> 00:29:22.328
I mean, do you get asked that kind of question?
00:29:22.589 --> 00:29:34.030
One of the slides from my deck was somebody from the World Economic Forum saying, AI isn't going to take your job, but somebody who knows how to use AI might, right?
00:29:34.050 --> 00:29:51.630
It's like, how do you answer the question about the role of AI in relation to humans and the kind of work displacement that AI may have versus making the work that humans do more creative because it does some of the really deep grungy stuff that we don't really like spending time on anyways.
00:29:51.911 --> 00:29:53.452
How do you generally answer that
00:29:53.952 --> 00:29:54.113
question?
00:29:54.133 --> 00:30:00.641
Yeah, I mean, I think in general, we've spent many decades trying to recreate human level capability in different respects.
00:30:01.161 --> 00:30:05.046
I will tell you that state of the art today is a very low level of human capability.
00:30:05.890 --> 00:30:13.458
You have this army of magical interns and you give all the tasks that you give an intern, you can now potentially give to some of these large language models.
00:30:14.159 --> 00:30:16.121
However, think about the intern scenario.
00:30:16.240 --> 00:30:24.910
Interns need management, they need governance, and the interns can learn and maybe the interns graduate and become a low level analyst.
00:30:24.950 --> 00:30:29.394
And you can trust them in that respect, but there's always going to need to be this management layer.
00:30:29.714 --> 00:30:32.478
At the end of the day, the intern helps extend our capability.
00:30:32.961 --> 00:30:33.221
right?
00:30:33.241 --> 00:30:36.067
They don't exist in a silo usually.
00:30:36.126 --> 00:30:37.750
It's never usually project-based.
00:30:37.809 --> 00:30:47.305
It could be, but typically they augment our ability to get to the things that we need to get to because we don't want to do certain things that really are a time suck.
00:30:47.404 --> 00:30:53.193
So think of the language model or the AI in a very similar fashion is it's an intern.
00:30:53.233 --> 00:30:56.619
It needs all the care and attention and oversight of an intern.
00:30:56.859 --> 00:31:09.684
It's going to take a lot of The grunt work off my plate, thankfully, and maybe over time I can coach it up and it can do a little bit more and a little bit more, but you're always gonna have a similar relationship and that's really what Bleeding Edge means today.
00:31:10.925 --> 00:31:13.228
So I have two more questions and there was also a question in chat.
00:31:13.288 --> 00:31:18.512
I'm gonna pick that one up from somebody who says, this is new and exciting, but it's 100% new.
00:31:19.173 --> 00:31:24.338
What should I, you know, how can I even basically have any impact on that or how can I make use of it?
00:31:24.798 --> 00:31:29.929
And that's, you know, David, my speech is, actually primarily targeted at that.
00:31:29.969 --> 00:31:39.742
If you go to my LinkedIn page, you'll see I have a little carousel on there on three things that AI can do with some examples and use cases of how you would use it.
00:31:40.805 --> 00:31:44.009
Literally, you can go to chat.openai.com.
00:31:44.108 --> 00:31:49.036
There's obviously other competitors to ChatGPT, but it was the first.
00:31:50.178 --> 00:31:58.390
either type very simple prompts about your profession, like tell me about this thing in procurement or tell me about this thing for my hobby.
00:31:58.730 --> 00:32:15.535
And then you can also form extremely complex prompts, basically describing a scenario, as Steve just said, as if you're giving a project to an intern and you'll be astounded by the kind of results and the outputs that it can produce because it's been trained against these extremely large language models.
00:32:16.076 --> 00:32:16.896
Steve, what if...
00:32:17.634 --> 00:32:27.824
you know, if somebody who's 100% not had any touch in this is interested in playing with it, what's the best way just to even get educated on this that you advise people when they ask you about it?
00:32:29.045 --> 00:32:38.333
Yeah, I think there, and this is constantly developing under our feet right now because this kind of breakthrough moment of ChatGPT only happened this past November.
00:32:38.394 --> 00:32:41.076
We're not even a year old and think about how much we've seen.
00:32:41.135 --> 00:32:46.681
So obviously get hands-on with some of the free and open tools at your disposal.
00:32:47.970 --> 00:32:59.808
The bards, the chat GPTs, the clods, there's a handful of them and just play around with it to start to exercise the limits and the capabilities.
00:33:00.730 --> 00:33:04.737
And there's a lot of education out there in terms of what is a prompt?
00:33:05.117 --> 00:33:07.260
How might I prompt more effectively?
00:33:08.066 --> 00:33:12.611
how do I ask these LMs questions and how might I ask them different questions?
00:33:12.711 --> 00:33:20.840
And if I don't get the response I was hoping for, how might I rephrase the question in such a way to get the response I was hoping for?
00:33:20.881 --> 00:33:28.450
There's a lot of education out there that even you could ask a large language model, where might I find access to this type of education?
00:33:28.490 --> 00:33:31.653
You might be surprised and delighted what you find in that respect.
00:33:32.577 --> 00:33:43.276
If you're just getting started and you feel like you need to take some sort of action, it's very important to have a game plan and put together a playbook on how might you actually leverage this across your organization.
00:33:43.817 --> 00:33:44.858
Think about your strategy.
00:33:44.959 --> 00:33:55.817
Think about, take just a scratch pad and write AI magical intern on the top of it and put a one column hopes and one column restraints.
00:33:56.321 --> 00:34:00.428
And your hopes are all the things you wish you could give this AI magical intern.
00:34:00.469 --> 00:34:05.977
And the restraints are, here are the things I would never allow this AI magical intern to do.
00:34:06.017 --> 00:34:08.380
And that's really Mason's strategy.
00:34:08.561 --> 00:34:11.445
And then start building on that and getting your education.
00:34:11.505 --> 00:34:13.849
And you'll see how this starts to fuel itself.
00:34:13.989 --> 00:34:21.661
But to the extent you can, bring in help for people that understand the space, whether you hire that help or you partner with that help.
00:34:22.273 --> 00:34:35.152
take any scenario that you're personally interested and try to stump the computer by asking it to produce a thing that you think is unlikely to get and you will really be taken up with what you can do.
00:34:35.809 --> 00:34:40.518
So let's come back to where you were going, Steve, where you were talking about getting help.
00:34:41.099 --> 00:35:02.907
So I did introduce in that speech that together, Civic and Kung Fu are going to be looking to go and work on some common clients through a variety of things if they're in an exploration mode or to the possible workshops where we're able to be able to help explore, do some hands-on question and answer topics, to be able to produce certain deliverables that are able to be helpful.
00:35:03.507 --> 00:35:16.469
Working on policy development, I know that's a key issue for everyone from the procurement shop to the technology shop to the finance shop, being able to have effective policies that are the correct shaping policy, but not just saying, don't go do anything.
00:35:17.090 --> 00:35:23.923
Horizon assessments where you can say, we want to go, we want to build some of our own platform sets, implement certain playbooks.
00:35:24.264 --> 00:35:35.909
All of these things are concepts for organizations where they are trying to say, I want to establish guardrails, but just like we set up guardrails on the freeway because we intend to go fast, you use the guardrail phrase before.
00:35:36.769 --> 00:35:39.813
guardrails that are intended to protect, but also support going fast.
00:35:40.233 --> 00:35:46.759
So I know some people are seeking to move past the trepidation about what is not known in order to move into the excitement of what can be.
00:35:47.278 --> 00:35:57.248
In my LinkedIn feed yesterday, Sean Naylor, who's the former CIO for Vermont, had a quote that's been on my feed twice that says, it's not about stopping it, it's about managing it.
00:35:57.668 --> 00:36:08.057
So we're looking forward to working with you on some of these projects and helping government and higher education be comfortable with deployment strategies that can let them feel comfortable getting on the front end of the curve.
00:36:08.117 --> 00:36:21.775
So, I mean, thoughts about how you've done some of that kind of work to help people get comfortable with getting out on the curve and the kind of things that we can do together, leveraging civics, public sector, state and local background, and Kung Fu's AI background.
00:36:21.815 --> 00:36:23.557
Like what are some things we can do for people with this?
00:36:24.579 --> 00:36:24.900
Yeah.
00:36:25.099 --> 00:36:27.543
And I mentioned earlier, it's good to have a game plan.
00:36:27.623 --> 00:36:28.885
It's good to develop a playbook.
00:36:29.105 --> 00:36:52.278
In reality, what I'm referring to is assess your opportunities and identify your risks in leveraging this technology and start to understand what might be available to you today in terms of opportunity versus opportunities you'd love to have and think about them two, three years time or 18 months time, if that's a more appropriate horizon.
00:36:52.900 --> 00:36:54.521
But where can all the things go wrong?
00:36:55.101 --> 00:36:58.646
What do we not know that we must know before we get to deployment?
00:36:58.978 --> 00:37:02.545
How do we even build our first environment or model?
00:37:02.905 --> 00:37:04.327
How might we start a sandbox?
00:37:04.367 --> 00:37:15.047
So at least we can, in a safe place, have our people inside our organization be able to bang and test on these models in a way that we're very comfortable with that's private and secure.
00:37:15.086 --> 00:37:29.760
So these are all the things that we guide people towards and help them really lay that out and package it up like a game plan or a playbook with some very precise recommendations on where do we get started right now?
00:37:30.601 --> 00:37:37.188
And opportunistically, where might we go and grow with additional technology and expertise investment?
00:37:37.769 --> 00:37:42.034
So truly crawl, walk, run when it comes to these types of technologies.
00:37:43.036 --> 00:37:47.302
Definitely look before you leap, but that's effectively what we're helping folks do.
00:37:48.994 --> 00:37:49.454
Excellent.
00:37:49.916 --> 00:37:56.887
I'll close as I started by saying, if you're interested in this topic and you want to have a conversation about it, go to procurementagent.ai.
00:37:56.947 --> 00:38:00.974
That'll take you to a direct page on the Civic site, and we're happy to talk to you about it.
00:38:02.157 --> 00:38:07.405
I've gotten a lot of great comments, even while we're sitting here, a couple of texts, and people are really interested in the topic.
00:38:07.465 --> 00:38:13.697
And it's great to have a true expert on who really is sitting in this all day long and being able to work in it deeply.
00:38:13.757 --> 00:38:19.246
And as you said, For all intents and purposes for the rest of us, this fell out of a spaceship in November.
00:38:19.306 --> 00:38:23.414
So it's all new and people are very excited about it.
00:38:23.514 --> 00:38:28.001
So we really appreciate you sharing your knowledge today and look forward to working with you on this in the future.
00:38:28.623 --> 00:38:29.264
It was my pleasure.
00:38:29.304 --> 00:38:30.045
Thanks, Dustin.
00:38:36.297 --> 00:38:36.478
Thank you.
00:00:00.833 --> 00:00:04.419
Hi, it's Dustin Lanier.
00:00:04.559 --> 00:00:05.301
Thanks for listening.
00:00:05.883 --> 00:00:16.440
Please find me on LinkedIn for original public sector operations content every week and please reach out to me if I and my team of procurement professionals at Civic Initiatives can help you be a public procurement change agent.
00:00:20.193 --> 00:00:20.474
Okay.
00:00:20.493 --> 00:00:22.736
Hey, thanks everybody for joining us today.
00:00:22.777 --> 00:00:24.239
I'm Dustin Lanier.
00:00:24.318 --> 00:00:25.881
I'm founder of Civic Initiatives.
00:00:26.402 --> 00:00:44.606
If you're really interested in these topics and you want to talk afterwards, either about some of the speaking that I'm doing on these topics at procurement conferences or the potential to do some work for you in the topics that we talked about, please go to procurementagent.ai, which will take you to a page on the Civic website where you can say you'd like to talk.
00:00:45.121 --> 00:00:51.229
With that as preamble out of the way, let me introduce Steve Meyer, who's the head of innovation for Kung Fu AI.
00:00:51.430 --> 00:00:54.234
Steve, why don't you introduce yourself and Kung Fu AI?
00:00:54.335 --> 00:00:56.578
And obviously people want to know about the fun name.
00:00:56.677 --> 00:00:59.241
So maybe talk a little bit about the company as well.
00:00:59.841 --> 00:01:00.343
Hi, everyone.
00:01:00.523 --> 00:01:01.223
Thanks, Dustin.
00:01:01.424 --> 00:01:02.246
It's exciting to be here.
00:01:02.265 --> 00:01:02.826
This will be fun.
00:01:03.267 --> 00:01:04.368
Like you said, my name is Steve Meyer.
00:01:04.388 --> 00:01:06.391
I'm one of the co-founders of Kung Fu AI.
00:01:06.492 --> 00:01:07.974
Non-traditional background.
00:01:08.353 --> 00:01:22.109
I'm a co-founder, I'm a non-technical co-founder, spent about a decade prior to starting Kung Fu AI in the Fortune 500 tech sector, working for a boutique consulting firm, predominantly with IBM and IBM Watson Groups.
00:01:22.150 --> 00:01:31.981
Got to cut my teeth in AI back in 2015 to 2017 period, helping IBM apply IBM Watson across a couple of interesting use cases.
00:01:32.481 --> 00:01:40.453
So fell in love with it so much so that I took a risk and decided to start a consulting practice around artificial intelligence.
00:01:41.474 --> 00:01:43.537
And that's exactly what we do.
00:01:43.759 --> 00:01:47.424
We help our customers both on the strategy and engineering side.
00:01:47.463 --> 00:01:54.754
And the name is an interesting story in that most people confuse Kung Fu with a martial art.
00:01:55.436 --> 00:02:27.204
And while that's true, it's actually a very narrow use of the word where Kung Fu actually is more often used, especially in Eastern cultures, as a verb and a way and a process of acquiring any great skill or knowledge over many years in hard work, in discipline, in practice, which in our estimation, we feel like is the road to artificial intelligence mastery is it's time in practice and like a good Kung Fu master, we can teach it to you and fight for you today.
00:02:27.444 --> 00:02:31.115
But our hope is that you will acquire the skill yourself over a period of time.
00:02:31.215 --> 00:02:33.199
So we thought it was appropriate for the company name.
00:02:33.981 --> 00:02:34.543
Very interesting.
00:02:35.025 --> 00:02:35.526
Great answer.
00:02:35.545 --> 00:02:52.562
I found you through a common friend a couple of months ago when we started collaborating and run up to the speech that I did at NIGP Forum, which was called Generative AI, the new tool in the toolkit, where I was trying to take the topic and present it at layman plus.
00:02:52.942 --> 00:03:03.251
So coming down from you guys as deep implementers, but bringing it higher than just the, sometimes the relatively simplistic stuff that you can see on LinkedIn.
00:03:03.491 --> 00:03:06.993
So had a lot of good feedback off of that presentation.
00:03:07.094 --> 00:03:13.943
I've already updated it twice I'll be doing that at other procurement conferences around the country, both this year and into next year.
00:03:13.984 --> 00:03:15.466
And there's a lot of excitement.
00:03:15.846 --> 00:03:28.564
I would say as much excitement on this topic as anything I've seen in several years of being involved in content, both for the application for people individually and then organizationally, the opportunity that AI can present.
00:03:29.066 --> 00:03:34.813
And so let's take it on the individual basis first, and then we'll talk about the organizational implications first.
00:03:35.394 --> 00:03:47.153
I think what has really astounded me as far as the difference in this piece of technology, as you mentioned Watson and other things that have been there before, is just how unbelievably accessible it is.
00:03:47.193 --> 00:03:59.977
You gave me a graphic that I have in the presentation that talks about this evolution and that right now we're able to provide unstructured data and get an unstructured response, which is new.
00:04:00.296 --> 00:04:08.731
And the barrier to entry is so low for an individual to simply go to one of those pages and to be able to type simple questions or incredibly complex prompts.
00:04:09.212 --> 00:04:16.848
So if anybody on here has not spent time in it, There's a lot that we could talk about, about images and video, but for this conversation, we're primarily talking about text.
00:04:17.389 --> 00:04:19.572
Go to chat.openai.com, play with it.
00:04:19.591 --> 00:04:22.696
There's other tools, but you'll be blown away if you haven't played with it yet.
00:04:22.815 --> 00:04:29.605
So Steve, describe what's happening to somebody who's interested, but hasn't really dipped their toe in their water.
00:04:29.665 --> 00:04:36.975
How would you describe this to your mom or your aunts or your long lost high school buddy who's like, what is all this and what's happening?
00:04:37.716 --> 00:04:38.476
Yeah, it's...
00:04:39.041 --> 00:04:46.711
really profound in the fact that this unstructured data in, unstructured data out, in that that's what we do.
00:04:47.273 --> 00:04:58.666
We take in visual cues, audio data, the world around us, our five senses, and output something unstructured in return, an opinion, our voice, our thoughts.
00:04:59.889 --> 00:05:15.153
And that's in essence why this is becoming more accessible in that we can actually interface in natural language with a artificial intelligent agent and then in return get some sort of intelligent response back.
00:05:15.473 --> 00:06:02.987
And I think what has really put a spotlight on the field of artificial intelligence and the need for artificial intelligence is the fact that any one of us can go to this virtual agent and have what seemingly an intelligent conversation with it ask questions and get a better response than we would ultimately get from a search query and effectively what's happening is there was this aha moment back in 2017 that computers are better with numbers and they are words and that's not really the aha moment but translating the words back to numbers and effectively taking your natural language query and converting it into a point, like you put on a graph.
00:06:04.149 --> 00:06:07.353
You think of your x and your y-axis, you put a point on a graph.
00:06:07.392 --> 00:06:12.139
Well, this graph just happens to have unlimited planes.
00:06:12.920 --> 00:06:18.625
And by doing that, you can start to put words as a point on a graph and start to see patterns.
00:06:19.146 --> 00:06:25.555
And now our words are mathematical input, and sentences can be math equations.
00:06:25.714 --> 00:06:59.033
And all of a sudden, the computer can predict what the correct answer is to your question so all of a sudden your words and response become a prediction problem and it keeps predicting what they think is the natural output and it's getting very very good at that and it can even go so far as start calculating the distance of points to start to understand patterns in new ways and what that means to us is that we can get really sophisticated analysis with very little effort and zero programming and that's just a just door or buster when it comes to this space.
00:06:59.153 --> 00:07:01.838
So it's very, very exciting.
00:07:01.898 --> 00:07:07.706
It's making AI really useful to everyone with a very low barrier to entry for the very first time.
00:07:08.968 --> 00:07:09.730
Great answers again.
00:07:09.769 --> 00:07:14.737
I'm going to have to listen back and pepper in a little bit of that sophistication into my core speech.
00:07:14.817 --> 00:07:18.002
I like the way you talk about translating sentences into equations.
00:07:18.083 --> 00:07:20.225
I've said it similarly, but you said it better.
00:07:20.305 --> 00:07:22.730
So thanks definitely for that viewpoint.
00:07:23.074 --> 00:07:27.346
So let's stay in the individual impact before we go into the organizational impact.
00:07:27.786 --> 00:07:55.151
So I described three things in the speech that I think public procurement, since primarily our audience is public procurement, although public sector in general, I think these also apply, that it's Excellent at synthesis, the ability to take extremely large data sets and to be able to search and remodel and remold it to produce everything from action items to synthesized groupings of information in a way that makes it so much easier to go through gigantic data sets.
00:07:55.572 --> 00:07:57.093
The best brainstormer of all time.
00:07:57.134 --> 00:08:00.036
It means that you never have to start with a blank piece of paper.
00:08:00.076 --> 00:08:10.266
You can always describe a scenario that you're trying to build a document or a piece of content against a requirement set against for from a specification perspective.
00:08:10.286 --> 00:08:17.153
And you get back information that is actionable in a sense that it's at a place where it can be edited.
00:08:17.353 --> 00:08:23.822
And myself personally, I'm a much better editor than I am sitting with a blank piece of paper and writing it out every time.
00:08:23.841 --> 00:08:40.759
And then the third was an on-demand quant that the ability to do certain mathematical things such as Excel formulas on hand for the rest of time, being able to do math within the tool, being able to do things like coding and actual programming and SQL data sets.
00:08:41.200 --> 00:08:51.297
So those were the three areas that I thought had the most impact for individuals who are trying to find where's the edge of this rock that they can pick up from an individual tool perspective.
00:08:51.717 --> 00:09:00.974
So talk about any of those three or things that you would add or how you would kind of say, what is at an individual level, what does it provide as a tool set today?
00:09:01.985 --> 00:09:03.707
No, I think those are really good.
00:09:03.748 --> 00:09:07.753
And where else I may highlight is analyst.
00:09:08.674 --> 00:09:21.350
So what effectively is these large language models, these GPTs, these LMs, they are a magical analyst that can do the work of an army of analysts in many respects.
00:09:22.091 --> 00:09:28.198
So the ability to take a sophisticated question, should I buy from this company?
00:09:28.278 --> 00:09:30.101
Should I invest in this company?
00:09:30.946 --> 00:09:38.738
There is quite a bit of research that is involved in understanding the company's viability, the price of the product.
00:09:38.798 --> 00:09:39.840
Is it a fair price?
00:09:41.201 --> 00:09:43.645
Will they deliver if we purchase?
00:09:44.826 --> 00:09:47.291
Are they a company that's growing and is sound?
00:09:47.971 --> 00:09:50.416
That could be eight hours worth of research.
00:09:50.716 --> 00:10:01.768
You can effectively whittle that down into eight minutes of carefully curated prompts or questions to a language model And that's magical, right?
00:10:01.927 --> 00:10:09.775
It might take an army of analysts to take that level of question and get you a response back in eight to 10 minutes.
00:10:10.255 --> 00:10:15.561
So I think analyst is really important, researcher, and another one I'd add is monitor.
00:10:16.081 --> 00:10:42.881
So if you want to understand if certain phenomenon are present in a large volume of data that's constantly being ingested into your business, think of documents where anomalies or fraud may occur, these models are very good at looking for patterns of phenomenon and being able to detect them where you might have an army of researchers who are constantly looking for changes in regulation or potential fraud issues.
00:10:43.341 --> 00:10:46.926
So I think those are also capabilities that we should be tracking.
00:10:48.849 --> 00:10:49.028
Excellent.
00:10:49.048 --> 00:10:57.145
So moving organizationally, so when Organizations are trying to take on a new piece of technology.
00:10:57.186 --> 00:11:08.668
They're always, of course, going to rightfully try to make sure that they have it set up to be able to be successful at it and that they're not walking into something that they can't manage.
00:11:08.849 --> 00:11:17.037
And so there's things that we can talk about, things like there's an ability for the data to be wrong because it's making estimations.
00:11:17.177 --> 00:11:24.683
So those are called hallucinations, which I find you can tend to see those relatively quickly, but that's been a concern set.
00:11:25.043 --> 00:11:31.931
But probably the number one concern I've heard is just the concept of what happens with the data when I create a query.
00:11:32.270 --> 00:11:41.663
So if I make a query that says, I want to take notes from an evaluation committee and to be able to drop them in in order to pluck out action items.
00:11:42.404 --> 00:11:44.006
Is it possible to query the query?
00:11:44.126 --> 00:11:48.851
Can somebody come in and say, have there been any evaluation meetings from X or Y or Z?
00:11:49.312 --> 00:11:55.577
And I think that that concern is one of the main ones that I think people have to kind of get their heads around.
00:11:55.597 --> 00:12:08.883
I personally feel like from what I've understood that the query is not retained in that specific way in the sense that It is presenting a question that produces the mathematical formula, as you said, and then it produces a mathematical formula back.
00:12:09.264 --> 00:12:15.197
I think we all come from paradigms where we're concerned about a large data set sitting out there that is exposable.
00:12:15.905 --> 00:12:17.327
So let's talk about it in two ways.
00:12:17.828 --> 00:12:21.014
One is just simply going to the open tools that are out there now.
00:12:21.335 --> 00:12:28.645
You know, what is a good data hygiene process on being able to use the tool and not feel like you're creating giant stores of information?
00:12:29.086 --> 00:12:38.480
After you answer that, then let's talk about how we can establish a more limited access environment that maybe amps up the kind of concerns or questions that people may have on that topic.
00:12:39.462 --> 00:12:40.264
Yeah, absolutely.
00:12:40.303 --> 00:12:47.653
Well, at the 10,000 foot level, we get pulled into a lot of LLM advisories.
00:12:47.953 --> 00:12:50.758
Hey, help us understand this model versus that API.
00:12:50.898 --> 00:12:54.001
It's truly a build versus API conversation.
00:12:54.022 --> 00:12:57.966
And the answer is yes, likely you need both.
00:12:58.748 --> 00:13:04.434
And these APIs that are available to us today are, I think are getting often a bad rap.
00:13:04.676 --> 00:13:10.895
It's unfair to call them insecure when it comes to certain types of topics.
00:13:10.975 --> 00:13:19.655
I think the more sensitive the topic and the higher risk the scenario that maybe we would like to leverage a large language model, more oversight is certainly required.
00:13:20.197 --> 00:13:25.528
But there's going to be situations where you can leverage these APIs in many situations.
00:13:25.730 --> 00:13:29.217
And there's going to be scenarios where you're going to want to use open source or something more private.
00:13:29.601 --> 00:13:41.777
But I will say that specifically some of the big guys in this space that are providing great APIs are adding additional guardrails to make things more private and secure if that is a big concern.
00:13:41.917 --> 00:13:55.253
But to your point, Dustin, it is basically scattershotting your input into a bunch of mathematical equations and units and then constructing the best response based on a prediction problem.
00:13:56.001 --> 00:14:06.586
In terms of it, keeping a memory or a log of what exactly you give it isn't quite fair, but there are certainly concerns and reasons why sometimes you may want to be private.
00:14:08.009 --> 00:14:16.378
Before we go to private, let me just sort of pivot the scenario just a little bit, and hopefully you can tell me if my scenario statement is correct.
00:14:17.359 --> 00:14:27.871
I think that government is used to potentially a scenario that says, here's a Word document that includes something that might be lightly sensitive, and I'm going to send it to an individual to do a transcription.
00:14:27.912 --> 00:14:35.182
So I send the document, the document is received by an individual, the individual does a transcription, and then they send me back the transcription.
00:14:35.542 --> 00:14:40.143
So in that scenario, in use case one, There was a full document.
00:14:40.547 --> 00:14:41.355
It went somewhere.
00:14:41.793 --> 00:14:49.260
And it has a possibility to be retained as the document, which is the four corners of the document with the words in the middle.
00:14:49.682 --> 00:14:54.746
So that would generate that concern of the document is existing in the universe.
00:14:55.287 --> 00:15:05.537
In my understanding, when we're doing a query into an AI scenario, I think I joked with you, it's atomizing Captain Kirk and reappearing him on the face of the planet.
00:15:05.616 --> 00:15:06.717
Is it the same Captain Kirk?
00:15:06.817 --> 00:15:09.421
Or did we kill Captain Kirk and rebuild him on the planet?
00:15:09.921 --> 00:15:18.120
Because The document, when it hits the query, it's being atomized into a math equation and then a math equation is coming back.
00:15:18.240 --> 00:15:31.910
And that's what I've been trying to explain to people that if we're thinking we're in the situation where we've sent a full document to somebody for a transcription event and we're trying to make that be an exact equivalent to what happens when we do a query on AI, it's not an equivalent.
00:15:32.451 --> 00:15:42.763
I think what I would add to that, it's definitely a yes and, what I would add to that is when you are training models is when the model is learning off information.
00:15:42.783 --> 00:15:50.096
Oftentimes when you are interfacing with an API, you're using inference or prediction but you're not training the model.
00:15:50.216 --> 00:15:56.342
So basically you are showing it information and saying, Hey, give me a response based on what you see.
00:15:56.403 --> 00:16:01.229
We're not actually putting the information back into the API or the model.
00:16:01.750 --> 00:16:10.500
Now there are certain situations where in the terms of use, you are in fact training the model, but in most instances, the model is pre-trained.
00:16:10.841 --> 00:16:14.706
It has already learned all that it can learn for the time being until the next release comes out.
00:16:15.042 --> 00:16:18.365
And you're basically showing it information and it's making a prediction.
00:16:18.405 --> 00:16:24.975
And that prediction is a response, which is kind of funny to think of a dialogue with a virtual agent as a prediction problem.
00:16:25.054 --> 00:16:28.440
But in actuality, everything we do is a prediction problem.
00:16:28.899 --> 00:16:29.942
Dustin, I hear your question.
00:16:30.121 --> 00:16:32.725
I predict what you and the audience may want to hear.
00:16:32.745 --> 00:16:34.168
And then I give a response.
00:16:34.227 --> 00:16:35.089
And sometimes I'm right.
00:16:35.548 --> 00:16:36.650
Sometimes I'm not so right.
00:16:36.750 --> 00:16:39.914
But everything can be framed fundamentally into a prediction problem.
00:16:39.955 --> 00:16:44.662
And that's effectively what we're doing here, more so when we're using APIs.
00:16:45.153 --> 00:17:06.153
Well, in the speech that I give at the simplest level I can do to talk about how this watch works before we talk about what time it is, I bring up a clip from one of the websites that says if we were to create a prediction formula that says if the words once upon a were in a row, we know that the next word is probably not armadillo and it's probably time.
00:17:06.882 --> 00:17:15.775
but then the ability to build those prediction models are based across a huge level of deep multi-planal learning as you referenced.
00:17:15.875 --> 00:17:27.992
And so to the degree that we're teaching the model something about our specific query, it's teaching it how to refine the future responses, but it's not necessarily teaching it that here's the notes from the evaluation committee.
00:17:28.193 --> 00:17:31.598
It's teaching it how to further refine a mathematical model.
00:17:32.140 --> 00:17:33.582
Let me give you an analogy.
00:17:34.049 --> 00:17:39.758
Think about yourself when you were somewhere between 15 and 18 years old and you were learning to drive a car.
00:17:40.779 --> 00:17:45.826
And you are taking the wheel for the first time, maybe even getting on the interstate or the highway for the first time.
00:17:45.967 --> 00:17:47.148
Your palms are sweaty.
00:17:47.189 --> 00:17:50.053
You're alert to absolutely everything.
00:17:50.093 --> 00:17:53.498
Perhaps you're maybe too alert to your surroundings around you.
00:17:53.890 --> 00:17:58.856
And it's a very tense moment, but you're absolutely learning as you're driving that vehicle.
00:17:58.958 --> 00:18:01.821
And all that processing is creating a lot of the anxiety likely.
00:18:01.862 --> 00:18:06.689
Now, fast forward 20, 30 years and think about driving today.
00:18:06.729 --> 00:18:11.636
You're not necessarily learning anything, but you're doing fundamentally the same action.
00:18:11.656 --> 00:18:15.362
You're identifying, you're predicting, you're reacting.
00:18:15.402 --> 00:18:18.467
But all of that learning has kind of hit its threshold.
00:18:18.487 --> 00:18:20.329
You're no longer training, you're doing.
00:18:20.930 --> 00:18:26.882
I think in some respects, you can think of some of these more prefab models as a similar experience.
00:18:27.022 --> 00:18:30.630
They've gone through the training mechanisms of learning to drive the car.
00:18:31.090 --> 00:18:32.394
Now they're just executing.
00:18:33.015 --> 00:18:40.289
So hopefully that resonates, but it felt good when I was thinking about it.
00:18:40.642 --> 00:18:41.202
No, I like it.
00:18:41.222 --> 00:18:41.663
That's great.
00:18:41.683 --> 00:18:42.684
That's absolutely great.
00:18:43.045 --> 00:18:58.248
So now let's talk about if somebody is going to make an enterprise investment or they want to really say, I want to do this, but I want to do it in an area where I have some controls on the outside edge of the fence line and I want to be able to have this be more secure.
00:18:58.327 --> 00:19:08.126
So what kind of options are there for somebody who, you know, they say it may be all well and good for an individual to use this over here, but what I want to deploy something that I'm confident in.
00:19:08.548 --> 00:19:13.135
What does that even mean from a deployment perspective?
00:19:13.175 --> 00:19:19.448
And what are some of the, how does somebody raise that up a level through something that would be in their own control space?
00:19:20.770 --> 00:19:21.772
Yeah, absolutely.
00:19:21.913 --> 00:19:28.305
So you definitely have the ability to weigh out those trade-offs, build versus API, right?
00:19:28.769 --> 00:19:48.057
And likely you will have API use cases that make a ton of sense because you want the capability, the cost makes sense for you, the reduction to get to production is there, which is always very, very important and useful, especially when you're trying to solve a hard problem as quickly and cost effectively as possible.
00:19:48.538 --> 00:19:53.086
But then there's going to be a lot of instances too, where you've decided that you want to use an open source model.
00:19:53.506 --> 00:20:06.282
You want to host that model inside your environment, and you want to train that model to perform a very specific task at a very, very high level, in which case you will invest more to get that model trained.
00:20:06.863 --> 00:20:14.113
However, the trade-off is you may get additional capability than what might otherwise be available and a private and secure environment.
00:20:15.329 --> 00:20:21.602
The open source community is constantly evolving and the state of the art is moving very, very quickly.
00:20:21.642 --> 00:20:28.213
So you can use a model that may be effective today and then use another one that may be more effective into the future.
00:20:28.674 --> 00:20:29.958
But those are some of the trade-offs, right?
00:20:29.998 --> 00:20:33.845
If you want fast and ready, API is going to be the way to go.
00:20:33.884 --> 00:20:38.374
And there's going to be use cases where it's low risk and privacy is less of a concern.
00:20:38.913 --> 00:20:45.344
And then if you want powerful and private, there's going to be more cost associated and effort associated with getting there.
00:20:45.804 --> 00:20:49.509
But likely the answer is, well, we kind of need both or something in between.
00:20:51.311 --> 00:20:53.315
Well, let's stay kind of within that.
00:20:53.596 --> 00:21:08.415
And organizationally, I know that your organization has had an opportunity to work with some of the federal government partners on different kinds of projects within the DOD zone or some higher ed institutions and laboratories and so forth.
00:21:08.736 --> 00:21:25.729
So whatever you're able to talk about and you can feel free to extrapolate it, you don't have to name individual projects, but maybe share a bit of perspective on the kind of use cases that you've been seeing from governmental clients and where you think some of the opportunity lies for government in this space.
00:21:27.041 --> 00:22:02.255
yeah there's a lot of fascinating work out there we had the opportunity to work with a major university here in the area and work with their career services and help them with labor prediction so the thinking was that their career services think of again we can create this managed magical analyst right these career services folks are spending tons of times looking at labor statistics and forecasts trying to understand what companies and sectors are more likely to grow, so they can invest their time in making the right partnerships, so they can get their students better career service advice.
00:22:02.654 --> 00:22:11.913
Hey, let's steer your curriculum in this direction, get better internships, and ultimately at the end of the day, better employment, which is a lot of what these universities are graded on.
00:22:12.385 --> 00:22:29.651
So being able to use data and machine learning and put a whole bunch of really interesting data sources together and make a prediction that is more effective than what our human brains can do to understand, no, you need to think about this sector and these companies.
00:22:29.692 --> 00:22:39.246
And oh, by the way, there's a shift going on where certain skill sets that existed yesterday that were in high demand are actually moving in another direction entirely.
00:22:39.842 --> 00:23:06.650
that becomes a really interesting one another example especially in the government space we see a bunch of document data problems meaning we have a lot of documents and a lot of data inside those documents and the ability to interrogate that data understand patterns of that data maybe even automate processes that are very paper driven become the concern and getting that information out is very, very difficult.
00:23:06.930 --> 00:23:22.402
So especially using large language models, this becomes a lot easier than how it was when we started working on these problems six years ago, where we can actually drop documents in with very little training and get insights out with simple questioning.
00:23:22.817 --> 00:23:28.567
which becomes very, very powerful, both when you're trying to understand is a phenomenon present in a document?
00:23:28.967 --> 00:23:31.070
What are the trend lines in this document?
00:23:31.590 --> 00:23:37.681
Or perhaps if something happens, we need to move it to this part of the process versus that part of the process.
00:23:37.701 --> 00:23:40.705
So it's really an intelligent triage type of use case.
00:23:40.746 --> 00:23:44.892
And there's plenty of case management issues out there where this becomes really, really right.
00:23:45.172 --> 00:23:56.461
Or if we just want help with research or analysis, another great example of where these things can play You know, we are seeing a tremendous amount of appetite and demand for strategy work.
00:23:56.800 --> 00:24:02.568
We need a position, we need to know the use cases, and we certainly need a lot of governance.
00:24:02.669 --> 00:24:08.457
And we have to understand when these models are performing or not performing, which means we need test and evaluation criteria.
00:24:09.377 --> 00:24:14.766
This is becoming more important for all organizations, especially as we move into generative, because it's inevitable.
00:24:15.201 --> 00:24:17.986
We're all going to have it and use it sooner rather than later.
00:24:18.026 --> 00:24:39.296
But to do this in the best possible way to get the most performance with the least amount of risk, we need to have a plan in place and doing that work, whether you do that work and inform us and people like us go execute, or we inform you and figure out if we or you can go execute, it becomes critical to doing this well.
00:24:40.317 --> 00:25:13.137
And in my speech, there's some areas that We're very interested in, as people who work with public procurement shops around the country, one is we've been deploying in pilot form generative AI chatbots that are trained within a body of knowledge against some of the procurement manual works that we do, which is an interesting alternative way to be able to do searches and generate different kinds of information so that it's more accessible both to people who might be answering questions of end users as well as being able to provide self-service to end users.
00:25:13.538 --> 00:25:23.413
Chatbots have been around a long time, of course, but they tended to be very rigid and very quickly got you to the wrong answer in a way that we've really been finding some success on.
00:25:23.835 --> 00:25:36.295
We're also interested in playbooks around aspects of handling requisition management, some of the solicitation activities, some of the spend activities as something that will be continuing to try to build around.
00:25:36.395 --> 00:25:41.705
So I think there's any number of possible use cases for this kind of tool set.
00:25:41.826 --> 00:25:55.558
And in the speech, I also talked about how one of my closest mentors and friends on overall technology had said he didn't think we would have access to some of this kind of stuff in our lifetimes, which I thought was pretty remarkable.
00:25:55.618 --> 00:25:59.063
So lots of really, really interesting options and opportunities here.
00:25:59.083 --> 00:26:02.986
I might add two others to that because that triggered a memory.
00:26:03.487 --> 00:26:22.785
We've done some work specifically on the commercial side of procurement, great use cases, spend analysis and spend optimization, and being able to look at all of our suppliers and understand perhaps when have we double paid or when are we paying two suppliers for the same thing and we have a surplus?
00:26:23.224 --> 00:26:24.586
Where is the anomalous spending?
00:26:24.727 --> 00:26:27.189
Oh, we don't typically spend that much in this category.
00:26:27.249 --> 00:26:27.970
That's a bit odd.
00:26:28.470 --> 00:26:29.590
These are all things we can catch.
00:26:29.832 --> 00:26:31.512
It just takes a lot of work to do it.
00:26:31.833 --> 00:26:38.952
And we can really reduce that workload and even automate some of those capabilities The other part too is the optimization.
00:26:38.972 --> 00:26:41.436
So the next level of intelligence is great.
00:26:41.797 --> 00:26:42.817
Where are we overspending?
00:26:42.857 --> 00:26:44.820
Then you ask a better question.
00:26:44.980 --> 00:26:46.683
How might we spend more effectively?
00:26:47.265 --> 00:26:51.951
And being able to optimize spend and becomes really important.
00:26:52.070 --> 00:26:56.657
Now, the one thing I want to point out there is that language models aren't the silver bullet.
00:26:56.778 --> 00:27:02.967
There's really no silver bullet in AI, but they do provide a baseline capability, which you can put other AIs on top of.
00:27:03.446 --> 00:27:08.144
So the first question we ask is, where is the anomalous spend happening?
00:27:08.164 --> 00:27:12.308
The next question we might ask is optimize my general spending habit.
00:27:12.490 --> 00:27:17.295
And one AI is built on top of the other to do both levels of sophistication.
00:27:17.615 --> 00:27:21.819
They're using functionally very different artificial intelligence algorithms.
00:27:22.140 --> 00:27:29.428
But when you start to sequence these things together, we can get more and more sophisticated use cases and capabilities.
00:27:30.169 --> 00:27:40.936
We'll sit in that for a minute because I think what's really important about some of these tools is The AI is there, in my opinion, as a helper for human beings to be able to do some of the more strategic pieces of work.
00:27:41.498 --> 00:28:00.859
And specifically in the case of spend analysis, which we've done a lot of, if you have shops that have always been relatively transactional then frankly, sometimes there isn't necessarily even a need for spend analysis because it doesn't affect the order in which you do things or potentially doesn't specifically affect the go-to-market strategy.
00:28:01.280 --> 00:28:10.893
And so a lot of times if you're going into a procurement shop that hasn't had the opportunity to do spend analysis before, what we're trying to do is to democratize the access to data.
00:28:11.354 --> 00:28:18.463
We want it so that it's not that only one person knows how to get the data out of the ERP and he's not even a procurement person, so it didn't matter anyways.
00:28:18.884 --> 00:28:31.720
We're trying to say, we can teach a persona or an agent, in that case, a virtual agent, to be able to say, what are the kind of trends that we're looking for that would affect our go-to-market strategy or our negotiating approach?
00:28:32.561 --> 00:28:38.250
And then being able to feed it with information, and it doesn't necessarily have to go all the way to the end.
00:28:38.611 --> 00:28:42.376
Like you mentioned, having a second AI that could say, well, how do I optimize against that?
00:28:42.849 --> 00:28:53.178
There could be a point where even just simply saying we're trying to democratize the access to data so that we're teaching our buyers how the information that we have in our systems affect our choices.
00:28:53.579 --> 00:29:20.445
And then we don't have to teach them how to do all of the smallest micro list pivot table infinity but we can say here's an ability to have a tool that can provide an answer to this question and here's what you as the human being do with that information in order to be good at procurement that would dramatically expand the role in the nature of spend analysis and procurement in a way that would be very fundamentally impactful a lot of procurement shops And so then that way, that's a great example.
00:29:20.465 --> 00:29:22.328
I mean, do you get asked that kind of question?
00:29:22.589 --> 00:29:34.030
One of the slides from my deck was somebody from the World Economic Forum saying, AI isn't going to take your job, but somebody who knows how to use AI might, right?
00:29:34.050 --> 00:29:51.630
It's like, how do you answer the question about the role of AI in relation to humans and the kind of work displacement that AI may have versus making the work that humans do more creative because it does some of the really deep grungy stuff that we don't really like spending time on anyways.
00:29:51.911 --> 00:29:53.452
How do you generally answer that
00:29:53.952 --> 00:29:54.113
question?
00:29:54.133 --> 00:30:00.641
Yeah, I mean, I think in general, we've spent many decades trying to recreate human level capability in different respects.
00:30:01.161 --> 00:30:05.046
I will tell you that state of the art today is a very low level of human capability.
00:30:05.890 --> 00:30:13.458
You have this army of magical interns and you give all the tasks that you give an intern, you can now potentially give to some of these large language models.
00:30:14.159 --> 00:30:16.121
However, think about the intern scenario.
00:30:16.240 --> 00:30:24.910
Interns need management, they need governance, and the interns can learn and maybe the interns graduate and become a low level analyst.
00:30:24.950 --> 00:30:29.394
And you can trust them in that respect, but there's always going to need to be this management layer.
00:30:29.714 --> 00:30:32.478
At the end of the day, the intern helps extend our capability.
00:30:32.961 --> 00:30:33.221
right?
00:30:33.241 --> 00:30:36.067
They don't exist in a silo usually.
00:30:36.126 --> 00:30:37.750
It's never usually project-based.
00:30:37.809 --> 00:30:47.305
It could be, but typically they augment our ability to get to the things that we need to get to because we don't want to do certain things that really are a time suck.
00:30:47.404 --> 00:30:53.193
So think of the language model or the AI in a very similar fashion is it's an intern.
00:30:53.233 --> 00:30:56.619
It needs all the care and attention and oversight of an intern.
00:30:56.859 --> 00:31:09.684
It's going to take a lot of The grunt work off my plate, thankfully, and maybe over time I can coach it up and it can do a little bit more and a little bit more, but you're always gonna have a similar relationship and that's really what Bleeding Edge means today.
00:31:10.925 --> 00:31:13.228
So I have two more questions and there was also a question in chat.
00:31:13.288 --> 00:31:18.512
I'm gonna pick that one up from somebody who says, this is new and exciting, but it's 100% new.
00:31:19.173 --> 00:31:24.338
What should I, you know, how can I even basically have any impact on that or how can I make use of it?
00:31:24.798 --> 00:31:29.929
And that's, you know, David, my speech is, actually primarily targeted at that.
00:31:29.969 --> 00:31:39.742
If you go to my LinkedIn page, you'll see I have a little carousel on there on three things that AI can do with some examples and use cases of how you would use it.
00:31:40.805 --> 00:31:44.009
Literally, you can go to chat.openai.com.
00:31:44.108 --> 00:31:49.036
There's obviously other competitors to ChatGPT, but it was the first.
00:31:50.178 --> 00:31:58.390
either type very simple prompts about your profession, like tell me about this thing in procurement or tell me about this thing for my hobby.
00:31:58.730 --> 00:32:15.535
And then you can also form extremely complex prompts, basically describing a scenario, as Steve just said, as if you're giving a project to an intern and you'll be astounded by the kind of results and the outputs that it can produce because it's been trained against these extremely large language models.
00:32:16.076 --> 00:32:16.896
Steve, what if...
00:32:17.634 --> 00:32:27.824
you know, if somebody who's 100% not had any touch in this is interested in playing with it, what's the best way just to even get educated on this that you advise people when they ask you about it?
00:32:29.045 --> 00:32:38.333
Yeah, I think there, and this is constantly developing under our feet right now because this kind of breakthrough moment of ChatGPT only happened this past November.
00:32:38.394 --> 00:32:41.076
We're not even a year old and think about how much we've seen.
00:32:41.135 --> 00:32:46.681
So obviously get hands-on with some of the free and open tools at your disposal.
00:32:47.970 --> 00:32:59.808
The bards, the chat GPTs, the clods, there's a handful of them and just play around with it to start to exercise the limits and the capabilities.
00:33:00.730 --> 00:33:04.737
And there's a lot of education out there in terms of what is a prompt?
00:33:05.117 --> 00:33:07.260
How might I prompt more effectively?
00:33:08.066 --> 00:33:12.611
how do I ask these LMs questions and how might I ask them different questions?
00:33:12.711 --> 00:33:20.840
And if I don't get the response I was hoping for, how might I rephrase the question in such a way to get the response I was hoping for?
00:33:20.881 --> 00:33:28.450
There's a lot of education out there that even you could ask a large language model, where might I find access to this type of education?
00:33:28.490 --> 00:33:31.653
You might be surprised and delighted what you find in that respect.
00:33:32.577 --> 00:33:43.276
If you're just getting started and you feel like you need to take some sort of action, it's very important to have a game plan and put together a playbook on how might you actually leverage this across your organization.
00:33:43.817 --> 00:33:44.858
Think about your strategy.
00:33:44.959 --> 00:33:55.817
Think about, take just a scratch pad and write AI magical intern on the top of it and put a one column hopes and one column restraints.
00:33:56.321 --> 00:34:00.428
And your hopes are all the things you wish you could give this AI magical intern.
00:34:00.469 --> 00:34:05.977
And the restraints are, here are the things I would never allow this AI magical intern to do.
00:34:06.017 --> 00:34:08.380
And that's really Mason's strategy.
00:34:08.561 --> 00:34:11.445
And then start building on that and getting your education.
00:34:11.505 --> 00:34:13.849
And you'll see how this starts to fuel itself.
00:34:13.989 --> 00:34:21.661
But to the extent you can, bring in help for people that understand the space, whether you hire that help or you partner with that help.
00:34:22.273 --> 00:34:35.152
take any scenario that you're personally interested and try to stump the computer by asking it to produce a thing that you think is unlikely to get and you will really be taken up with what you can do.
00:34:35.809 --> 00:34:40.518
So let's come back to where you were going, Steve, where you were talking about getting help.
00:34:41.099 --> 00:35:02.907
So I did introduce in that speech that together, Civic and Kung Fu are going to be looking to go and work on some common clients through a variety of things if they're in an exploration mode or to the possible workshops where we're able to be able to help explore, do some hands-on question and answer topics, to be able to produce certain deliverables that are able to be helpful.
00:35:03.507 --> 00:35:16.469
Working on policy development, I know that's a key issue for everyone from the procurement shop to the technology shop to the finance shop, being able to have effective policies that are the correct shaping policy, but not just saying, don't go do anything.
00:35:17.090 --> 00:35:23.923
Horizon assessments where you can say, we want to go, we want to build some of our own platform sets, implement certain playbooks.
00:35:24.264 --> 00:35:35.909
All of these things are concepts for organizations where they are trying to say, I want to establish guardrails, but just like we set up guardrails on the freeway because we intend to go fast, you use the guardrail phrase before.
00:35:36.769 --> 00:35:39.813
guardrails that are intended to protect, but also support going fast.
00:35:40.233 --> 00:35:46.759
So I know some people are seeking to move past the trepidation about what is not known in order to move into the excitement of what can be.
00:35:47.278 --> 00:35:57.248
In my LinkedIn feed yesterday, Sean Naylor, who's the former CIO for Vermont, had a quote that's been on my feed twice that says, it's not about stopping it, it's about managing it.
00:35:57.668 --> 00:36:08.057
So we're looking forward to working with you on some of these projects and helping government and higher education be comfortable with deployment strategies that can let them feel comfortable getting on the front end of the curve.
00:36:08.117 --> 00:36:21.775
So, I mean, thoughts about how you've done some of that kind of work to help people get comfortable with getting out on the curve and the kind of things that we can do together, leveraging civics, public sector, state and local background, and Kung Fu's AI background.
00:36:21.815 --> 00:36:23.557
Like what are some things we can do for people with this?
00:36:24.579 --> 00:36:24.900
Yeah.
00:36:25.099 --> 00:36:27.543
And I mentioned earlier, it's good to have a game plan.
00:36:27.623 --> 00:36:28.885
It's good to develop a playbook.
00:36:29.105 --> 00:36:52.278
In reality, what I'm referring to is assess your opportunities and identify your risks in leveraging this technology and start to understand what might be available to you today in terms of opportunity versus opportunities you'd love to have and think about them two, three years time or 18 months time, if that's a more appropriate horizon.
00:36:52.900 --> 00:36:54.521
But where can all the things go wrong?
00:36:55.101 --> 00:36:58.646
What do we not know that we must know before we get to deployment?
00:36:58.978 --> 00:37:02.545
How do we even build our first environment or model?
00:37:02.905 --> 00:37:04.327
How might we start a sandbox?
00:37:04.367 --> 00:37:15.047
So at least we can, in a safe place, have our people inside our organization be able to bang and test on these models in a way that we're very comfortable with that's private and secure.
00:37:15.086 --> 00:37:29.760
So these are all the things that we guide people towards and help them really lay that out and package it up like a game plan or a playbook with some very precise recommendations on where do we get started right now?
00:37:30.601 --> 00:37:37.188
And opportunistically, where might we go and grow with additional technology and expertise investment?
00:37:37.769 --> 00:37:42.034
So truly crawl, walk, run when it comes to these types of technologies.
00:37:43.036 --> 00:37:47.302
Definitely look before you leap, but that's effectively what we're helping folks do.
00:37:48.994 --> 00:37:49.454
Excellent.
00:37:49.916 --> 00:37:56.887
I'll close as I started by saying, if you're interested in this topic and you want to have a conversation about it, go to procurementagent.ai.
00:37:56.947 --> 00:38:00.974
That'll take you to a direct page on the Civic site, and we're happy to talk to you about it.
00:38:02.157 --> 00:38:07.405
I've gotten a lot of great comments, even while we're sitting here, a couple of texts, and people are really interested in the topic.
00:38:07.465 --> 00:38:13.697
And it's great to have a true expert on who really is sitting in this all day long and being able to work in it deeply.
00:38:13.757 --> 00:38:19.246
And as you said, For all intents and purposes for the rest of us, this fell out of a spaceship in November.
00:38:19.306 --> 00:38:23.414
So it's all new and people are very excited about it.
00:38:23.514 --> 00:38:28.001
So we really appreciate you sharing your knowledge today and look forward to working with you on this in the future.
00:38:28.623 --> 00:38:29.264
It was my pleasure.
00:38:29.304 --> 00:38:30.045
Thanks, Dustin.
00:38:36.297 --> 00:38:36.478
Thank you.
