1
00:00:06,507.6914110414 -->
00:00:10,557.691411041
Welcome and thank you for joining us on today's AI Explained.
2
00:00:10,747.691411041 -->
00:00:11,977.691411041
I'm Krishna Gade.
3
00:00:11,977.691411041 -->
00:00:14,297.691411041
I'm one of Co-founder and CEO of Fiddler AI.
4
00:00:15,127.690411041 -->
00:00:16,677.691411041
I will be your host today.
5
00:00:16,897.691411041 -->
00:00:18,627.690411041
We have a very special guest today.
6
00:00:18,857.692411041 -->
00:00:27,817.689411041
Uh, Gary Kotovets is the Chief Data and Analytics Officer at Dun Bradstreet, where he leads the company's data and AI strategy.
7
00:00:27,967.692411041 -->
00:00:34,987.692411041
Uh, before joining D&B, he spent nearly two decades at Bloomberg as the global head of data acquisition and management.
8
00:00:35,767.691411041 -->
00:00:37,877.689411041
Um, you know, welcome, Gary.
9
00:00:37,967.692411041 -->
00:00:42,587.690411041
Um, you know, we're very excited to host you on this webinar.
10
00:00:44,167.689411041 -->
00:00:44,697.690411041
Oh, hi.
11
00:00:44,747.691411041 -->
00:00:45,217.689411041
Hi, Krishna.
12
00:00:45,447.692411041 -->
00:00:46,167.690411041
Nice to meet you.
13
00:00:46,677.691411041 -->
00:00:46,797.691411041
Awesome.
14
00:00:46,857.692411041 -->
00:00:47,867.691411041
Thanks for having me.
15
00:00:48,527.690411041 -->
00:00:48,877.690411041
Absolutely.
16
00:00:48,907.691411041 -->
00:00:51,713.692411041
Hello, everybody Awesome.
17
00:00:51,743.692411041 -->
00:00:58,333.692411041
Gary, before we get into it, right, uh, tell us a little bit about who you are and what you're focused on at D- D&B.
18
00:00:58,943.692411041 -->
00:01:01,393.692411041
Um, you know, it'd be great to learn- Yeah about your background.
19
00:01:01,413.692411041 -->
00:01:01,763.692411041
Yeah.
20
00:01:01,953.692411041 -->
00:01:02,743.691411041
Yeah, absolutely.
21
00:01:02,813.692411041 -->
00:01:09,683.691411041
Uh, so, um, I lead, uh, our data analytics and AI strategy.
22
00:01:10,133.691411041 -->
00:01:20,213.692411041
Um, and, uh, you know, the, the data that we're responsible for is essentially all of our commercial data and what we call right now our commercial graph.
23
00:01:20,803.692411041 -->
00:01:36,63.690411041
Uh, so this is data across, uh, uh, 650-plus million businesses, uh, that we, um, uh, uh, acquire information on and manage on a, on a basically daily and intraday basis.
24
00:01:36,523.693411041 -->
00:01:47,903.69241104
Uh, we collect over 11,000 different, uh, data elements or fields, uh, of information o- on any one of those businesses at any point in time across 200-plus, uh, markets globally.
25
00:01:48,503.69241104 -->
00:02:05,373.69341104
Um, and then we produce, uh, you know, analytics and insights, and a lot of people, I'm sure, are familiar with some of our, um, capabilities, uh, such as our Paydex score or, or failure delinquency scores, which basically give you a, a sort of a financial stability insight.
26
00:02:05,403.69441104 -->
00:02:09,623.68741104
Uh, uh, our supply chain analytics, our sales and marketing insights.
27
00:02:10,63.69041104 -->
00:02:20,323.69141104
Um, and then, you know, we serve, uh, the global, uh, I would say 95% of the Global 500, uh, corporates and financial institutions in the world.
28
00:02:20,323.69241104 -->
00:02:21,983.69241104
It's about 200,000 customers.
29
00:02:22,863.69341104 -->
00:02:23,183.69341104
Awesome.
30
00:02:24,173.69041104 -->
00:02:26,203.68741104
You spent almost two decades at Bloomberg.
31
00:02:26,333.68741104 -->
00:02:37,153.69241104
Um, you know, two companies where data is the entire product, right? How did that shape the way you think about AI and, you know, and, you know, build on especially on top of verified data? Yeah.
32
00:02:37,213.69141104 -->
00:03:17,663.69441104
Um, you know, if you, if you think about the data that, uh, Bloomberg provides to its customers, um, it's, uh, you know, millisecond, uh, you know, high frequency, extremely sensitive content, right? So you're talking about exchange, uh, pricing, uh, right, real-time pricing, or you're talking about, uh, bond pricing from banks or, or whether it's, uh, index data from, uh, you know, MSCI or, or S&P that, uh, that you use to, uh, benchmark your portfolio against and make some of the most, you know, critical financial decisions as a, as a hedge fund or a large pension fund.
33
00:03:18,193.68741104 -->
00:03:59,457.69141104
Uh, and so the criticality and the sensitivity, uh, and the quality of the data, right, is extremely important for those customers, right? And I think that kind of, um- Uh, uh, you know, I guess, uh, the perspective, right, uh, on data and, and that kind of, I would say constant, uh, uh, it's almost like embedded in my blood at this point, right? Uh, to ensure that, uh, the-- our customers get the highest quality, uh, content, uh, because it's so sensitive, because their jobs are so sensitive to the quality of the information that they receive and the decisions that they make.
34
00:03:59,867.69241104 -->
00:04:03,457.69141104
And so this is very similar if you think about now from an agentic perspective.
35
00:04:03,947.69041104 -->
00:04:23,307.69241104
Um, you know, as, as more and more customers, uh, start to implement, uh, and, and, you know, orchestrate AI workflows, uh, AI-driven workflows that are becoming more and more autonomous and the decisions that are being made, uh, with, uh, you know, less and less humans in, in, in the middle.
36
00:04:23,787.69041104 -->
00:04:30,247.68941104
They have to trust the decisions those agents, uh, uh, make inside those workflows.
37
00:04:30,647.69141104 -->
00:05:05,857.68941104
And, uh, looking at the data and relying on the quality data is, is the most important critical step in that deci- in that process or in that design process, right? So, uh, you know, the way we think about D&B data, uh, is very much through that lens, right? Is, is, is saying we want the customer to be able to, you know, improve their productivity, improve their efficiencies, right? And we want to improve time to value for them to get that, uh, value out of their R- uh, AI investment.
38
00:05:06,567.68941104 -->
00:05:19,891.69241104
And so, uh, you know, our job is to ensure that we give them the, the, the best, uh, quality and the-- to serve as that foundation so they can start their journey and, and build their capabilities as quickly as possible Awesome.
39
00:05:20,51.69241104 -->
00:05:24,391.69241104
I think, uh, you mentioned two things, right? Agentic trust and data quality.
40
00:05:24,491.69241104 -->
00:05:31,591.69241104
Um, D&B's data already lives in these enterprise workflows, you know, long before agentic AI, AI was a thing.
41
00:05:32,131.69141104 -->
00:05:32,151.69141104
Yeah.
42
00:05:32,291.69241104 -->
00:05:47,871.69141104
And now when you're starting to build agents, you know, on top of your data sets, you know, what, what are the foundational elements that you're putting together to make this change happen? Uh, so, well, you know, we have, um...
43
00:05:48,651.69141104 -->
00:05:52,21.69141104
There's sort of, kind of multiple dimensions to this.
44
00:05:52,101.69041104 -->
00:06:53,71.68941104
Uh, one is you have customers who already embed our data into their, like you said, right, into their risk applications, into their, uh, procurement applications like SAP or into their CRMs, right? So the, the data is already living there, right? Uh, however, one of the key aspects as we all know, right, as part of that, uh, sort of AI development journey, you need the context, right? And so how do you become that context layer? And so we had to, uh, uh, you know, ensure that our data is available via MCP, right? Uh, we've created a lot of, uh, you know, skills and tools, uh, that, uh, help, that are available in sort of, uh, as a markdown file for our customers, and that allows them to quickly adapt our, our, our data and capabilities for those agents that are already sitting inside, uh, those CRM, for example, or is already interacting with their first party data.
45
00:06:53,501.68741104 -->
00:07:19,515.02407771
Uh, it may be sitting in their Databricks environment, right, along with their data, or maybe sitting in their, um, you know, or they may be using, you know, Codex or Cursor to develop those capabilities, right? And as they're developing those agentic workflows, uh, they need access, access to, uh, our content in that, in a way that, uh, makes it, you know, obviously consumable, right, uh, by those, uh, agentic capabilities.
46
00:07:19,515.02407771 -->
00:07:21,641.68741104
So MCP is the way to do that.
47
00:07:22,71.68741104 -->
00:07:44,401.69041104
We've also released, uh, A2A as, as another, uh, way and, uh, you know, we, we have a, a business verification match agent that we made available, uh, to our customers, uh, that, that could allows them to easily find, uh, through that protocol on Google Marketplace, uh, our, our, our agents and then deploy them into their, uh, uh, workflows.
48
00:07:45,465.69241104 -->
00:07:45,915.69241104
Got it.
49
00:07:46,485.69241104 -->
00:07:57,65.69241104
So when you actually go back to that data quality, when you say verified data, what does that actually mean to build in practice? You know, where does it go beyond just having clean- Yeah data records? Yeah.
50
00:07:57,915.69241104 -->
00:07:58,295.69241104
Yeah.
51
00:07:58,305.69241104 -->
00:08:04,275.69141104
I mean, uh, we think about, uh, data lineage, uh, right, and provenance.
52
00:08:04,375.69141104 -->
00:08:13,665.69241104
Uh, and we think about, you know, ensuring that the sources from which we obtain the data, right, are verified and validated.
53
00:08:14,185.69241104 -->
00:08:49,383.69241104
And then we have, you know, several steps after that where we also ensure that the values that are, uh, you know, extracted from those sources, right, are accurate, timely, um, and, uum- consistent, right? Uh, and, you know, we, we probably do around 100 billion data quality checks on a monthly basis across that in our entire, uh, you know, all those refreshes that are coming in through our- Mm-hmm Uh, through our data supply chain go through about 100 billion checks, uh, a day.
54
00:08:50,103.69141104 -->
00:09:12,133.69041104
So that's pretty significant, uh, right? I mean, and so, you know, we're continuously, uh, uh, tweaking and monitoring, tweaking and modifying the data quality rules, uh, as the changes, uh, to the data occur from the source, uh, to make sure that we're validating and, and ensuring that we're, we're, we're feeding to the customer the best data there is.
55
00:09:13,83.69241104 -->
00:09:14,903.69241104
That kind of goes back to that trust factor.
56
00:09:15,703.69141104 -->
00:09:16,33.69241104
Got it.
57
00:09:16,213.69041104 -->
00:09:29,633.68941104
And when you're building the agentic layer on these data sets, you know, whether it's for credit analysis or, you know- Mm-hmm compliance use cases, how do you make sure that there is full lineage behind those decisions, you know, when agents are acting on their own? Yeah.
58
00:09:29,853.68941104 -->
00:09:30,303.68941104
Yeah.
59
00:09:30,373.68941104 -->
00:09:38,23.69041104
So o- our, I mean, right from the start, I mean, we started our AI journey probably in '23, uh, ish.
60
00:09:38,133.68841104 -->
00:09:50,553.69341104
Uh, you know, we built a platform, a centralized platform that allows us to build, uh, our, our capabil- our AI capabilities, uh, and, and deploy them, uh, quite rapidly.
61
00:09:51,33.68941104 -->
00:10:05,53.69241104
Uh, and as part of that platform, uh, we've, uh, incorporated the, all of the data entitlements and the controls, and that also allows us to then create...
62
00:10:05,63.68841104 -->
00:10:25,113.69041104
You know, that sort of followed with, um, or followed up with a, a, a clear, concise, I would say, pretty rigid policy that says anything anybody builds, uh, in our company, uh, has to be able to demonstrate where the answer came from.
63
00:10:25,433.69141104 -->
00:10:25,823.69141104
Hmm.
64
00:10:26,173.68941104 -->
00:10:32,643.68941104
Um, and so that lineage is extremely critical, and it's embedded in all of our, uh, products today.
65
00:10:33,63.68841104 -->
00:10:33,603.69041104
Got it.
66
00:10:34,53.68641104 -->
00:10:43,673.69341104
Apart from lineage, you know, one of the problems that, uh, data companies like you might be dealing with is this non-determinism of agents, right? Where agents can hallucinate and, you know.
67
00:10:44,263.69341104 -->
00:10:59,935.69141104
You know, what-- how are you sort of, uh, addressing those problems? Is it a data problem or a model problem for you? So it's definitely not a data problem- Mm-hmm because, uh, you know, it, it, we're, we, we are that grounded truth, right? Yeah.
68
00:10:59,945.69241104 -->
00:11:14,155.69141104
So it's been, it's been easier, I would say, for us and- Mm-hmm a lot of our customers because we're sort of serving up to you what you need to, uh, uh, we're serving up to you what you need to, um, you know, validate, right? Mm-hmm.
69
00:11:14,155.69141104 -->
00:11:15,105.69141104
Uh, against.
70
00:11:15,465.69141104 -->
00:11:15,695.69141104
Hmm.
71
00:11:15,785.69241104 -->
00:11:21,185.69241104
Uh, so it really becomes ensuring that the model serves up the right answer.
72
00:11:21,545.69141104 -->
00:11:21,905.69141104
Right.
73
00:11:21,965.69141104 -->
00:11:29,735.69041104
And so, you know, we, we've experiment a lot with many different, uh, small language models, large language models.
74
00:11:29,915.69041104 -->
00:11:29,945.69041104
Mm-hmm.
75
00:11:29,945.69041104 -->
00:11:29,955.69041104
Mm-hmm.
76
00:11:30,305.69041104 -->
00:11:57,385.69041104
Uh, w- I mean, we literally have a continuous sort of pipeline- Mm-hmm through which we, once we identify something new- Mm-hmm right, that we should look at, open source or, or frontier, uh, version or new frontier model version, we automatically start to test validate it, and then- Mm-hmm Figure out, you know, look at how it responds, right, uh, to the different data sets and how it serves them up to our customers.
77
00:11:57,395.69341104 -->
00:11:57,415.69341104
Yeah.
78
00:11:57,785.69241104 -->
00:12:20,667.69241104
And ex- and how it also executes agentic, you know, as agents execute those tasks, how are they executing them, and are they doing it the right way? So is there a problem- So you Probably are running, like, a lot of agentic evals as you're trying To We do a lot of agentic evals, and then we also, and then we also, um- created monitoring capabilities, right? That watch for, uh, behavior.
79
00:12:20,997.69241104 -->
00:12:21,117.69241104
Right.
80
00:12:21,117.69241104 -->
00:12:28,637.69241104
That monitor for behaviors, right? So to ensure that the right steps were taken, uh, by the agent to produce that, uh, that answer.
81
00:12:28,977.69241104 -->
00:12:35,897.69241104
And so we run those, uh, periodically- Mm uh, To, to ensure that everything is working the way as designed.
82
00:12:36,867.69241104 -->
00:12:37,37.69241104
Yeah.
83
00:12:37,37.69241104 -->
00:12:42,457.69241104
One of the things that we work with on our, our sales team, we work with some of these, uh, data companies like Nielsen.
84
00:12:42,457.69241104 -->
00:12:52,57.69241104
One of the things that we get reported is when agents query two trusted data sources, and then they, they start to disagree, you know, and then the agent is kind of conflicted.
85
00:12:52,467.69041104 -->
00:12:58,207.69241104
Have you run into some situations like that where, you know, anything that you can share, um, uh, to, to our audience? Yeah.
86
00:12:58,547.69241104 -->
00:13:04,497.68941104
Yeah, I mean, yeah, that happens, uh, you know, occasionally- Mm uh, right, for us.
87
00:13:04,547.69241104 -->
00:13:15,537.69141104
Uh, obviously, you know, we also, uh, you know, we, we kind of pipe everything into the single source, but obviously there's tens of thousands of sources from which we collect our data.
88
00:13:16,37.69041104 -->
00:13:16,267.69041104
Right.
89
00:13:16,457.69241104 -->
00:13:19,157.69241104
Uh, or, a- and/or produce it ourselves.
90
00:13:19,227.68941104 -->
00:13:21,317.69141104
And so yes, that does happen.
91
00:13:21,747.68941104 -->
00:13:26,347.69241104
You know, one of the things that we do is a little bit of that sort of what I just said.
92
00:13:26,347.69241104 -->
00:13:36,787.68941104
We have this sort of generic tools library- Yeah It, that allows the agents to, uh, that has very rigid, uh, r- rules, right? Yeah.
93
00:13:36,847.68841104 -->
00:13:37,257.68841104
Embedded.
94
00:13:38,37.69041104 -->
00:13:49,537.69041104
And so when we, uh, uh, when anybody within the organization develops, uh, an, an agent they must follow or they must call out those tools, right? Mm.
95
00:13:49,537.69041104 -->
00:13:50,327.68741104
That we've developed.
96
00:13:50,327.69141104 -->
00:13:59,657.68741104
And those are the rules that Those are the tools that have the rules- Got it that The specific rigid guidelines, uh, that enforce that, uh, those people to follow.
97
00:13:59,677.69041104 -->
00:14:08,747.68941104
So, uh, we, um, you know, we, we've seen still, right, uh, some agents kind of break those rules over time, right? Mm.
98
00:14:08,747.68941104 -->
00:14:19,107.69141104
Uh, but that's where some of that monitoring comes in, right? To make sure that, hey, like, you know, did they follow the same steps as, as, as they were designed to do or not? Yeah.
99
00:14:19,627.68741104 -->
00:14:35,685.69241104
So I mean, these days we are moving from single turn agents that were more chatbot-like to like multi-turn agents that are taking- Yeah more actions, right? Have you seen the change in how you're evaluating and monitoring agents as you probably going through that journey? Yeah.
100
00:14:35,685.69241104 -->
00:14:36,395.69241104
I mean, yes.
101
00:14:36,435.69241104 -->
00:14:57,875.69141104
I mean, you know, we started ourselves, right? With, uh, we have our own sort of chat interface- Yeah that talks to all our data, right? And, uh, and interestingly, that also became p- in part our framework that we use now to, uh, develop all the other, uh, kind of agents that exec- you know, that, that develop, that execute tasks, right, uh, inside our particular workflow.
102
00:14:57,875.69141104 -->
00:14:59,775.69141104
Becomes your co-pilot somewhat, Yeah.
103
00:14:59,815.69141104 -->
00:15:00,285.69141104
Yeah, yeah.
104
00:15:01,115.69241104 -->
00:15:12,365.02574437
And so yeah, there's, there's, you know, like our risk platform, for example, has, um- You know, uh, a KYC agent, right? Yeah.
105
00:15:12,365.02574437 -->
00:15:14,211.69241104
And a, and a credit check agent.
106
00:15:14,271.69241104 -->
00:15:48,421.69141104
And, uh, u- underneath that, there's other kind of sub-agents that perform the specific subtasks, right, that are required to, to do the full KYC check- Right on that customer, right? Um, or a, a, a sub-agent, uh, sub-agents that perform a full credit check- Mm On a customer, right? Um, and, uh, and, and then, you know, once those two steps are done, then there's some kind of a, a decision that's then produced and, you know, the, the our customer then makes a decision, "Hey, I want to, you know, press the button," so to speak, right? To- Mm.
107
00:15:48,421.69141104 -->
00:15:54,441.69141104
To provide that, uh, uh, cust- that their customer with the loan, right? Yeah.
108
00:15:54,451.69041104 -->
00:16:04,81.69041104
Or reject it, right? And then eventually those steps would probably be automated as well, but we kind of leave it up to the client to make those final- Yeah step decisions, uh, themselves.
109
00:16:04,661.68941104 -->
00:16:14,871.68841104
Um, and so, uh, you know, th- those flows, right, follow, like as I mentioned, those kind of tools and skills that we've created.
110
00:16:15,271.69041104 -->
00:16:15,511.69041104
Right.
111
00:16:15,541.69441104 -->
00:16:44,321.690411
Uh, but they also follow the standard logic that we have already created inside the, our risk analytics software platform that we- Mm you know, already have a lot of customers using, right? So for those agents it's sort of dif- it's easier to follow a specific path- Mm Uh, because those paths have been predetermined- Yeah Uh, for them already, right? Uh- You're layering the agentic skin on your existing applications and taking- Yeah advantage of years of work that you Already have.
112
00:16:44,321.690411 -->
00:16:44,551.688411
Yeah, yeah.
113
00:16:44,551.688411 -->
00:16:51,281.692411
So we recreated the agents actually to mimic- Ah, okay The workflow of the software, right? That's Really cool.
114
00:16:51,281.692411 -->
00:16:51,441.692411
Yeah.
115
00:16:51,491.691411 -->
00:16:51,761.691411
Yeah.
116
00:16:51,761.691411 -->
00:17:06,961.688411
And then that, those And then those, uh So it's the And, and then what we did was just, we just embedded the, the steps the, the, the software took- Uh-huh When it did the analysis, right, o- o- of a KYC step or a, or a credit check step.
117
00:17:06,971.687411 -->
00:17:11,131.691411
And so they, the logic in those applications becomes like the, the checks and balances- The agent.
118
00:17:11,291.686411 -->
00:17:12,241.691411
Yeah, exactly that's doing that work.
119
00:17:12,241.691411 -->
00:17:17,721.687411
The logic of those applications essentially become the prompts- Right For the agents that we built, right? Yeah.
120
00:17:17,721.687411 -->
00:17:25,138.3600777
And then, and then we, we made those prompts into these, what we call the skills, right? Uh-huh.
121
00:17:25,138.3600777 -->
00:17:28,981.696411
And the skills are like there's a skill that does matching, right? Mm.
122
00:17:28,981.696411 -->
00:17:57,675.692411
So first let me verify that the entity you're questioning is the one that you intend to question, right, me about, right? Then let me then, uh, double check to give you here's the, you know, another skill that looks at photographic information and pulls that down, right, to give you the basic- like, knowledge that you need to know before you make that next step, right? Which is, okay, let me go and check, uh, beneficial ownership, uh, information.
123
00:17:57,675.692411 -->
00:18:07,355.693411
Let me go check corporate linkage and understand where that company belongs on that, uh, tree, right? Let me check the CEO name, right, and the corporate officers.
124
00:18:07,395.692411 -->
00:18:23,555.692411
Uh, let me do, um, uh, uh, you know, a, a check around physical locations of that business, right, and where they're You know, do they have any suppliers in, you know, geopolitically or i- in the climate risk, uh, zones, right? Things like that.
125
00:18:23,555.692411 -->
00:18:30,525.690411
So those are all- Yeah Standard steps that our software took that are now- Mm you know, an agent does, performs.
126
00:18:30,945.693411 -->
00:18:31,105.693411
Mm, Mm.
127
00:18:31,105.693411 -->
00:18:42,85.690411
Uh, and then the skills are those individual sort of micro tasks that we require the agent to, to go and get, right? A- and, and look up or interact with, rather.
128
00:18:42,645.690411 -->
00:18:43,85.690411
Awesome.
129
00:18:43,95.691411 -->
00:18:43,735.690411
That's really cool.
130
00:18:44,85.692411 -->
00:18:57,525.688411
So when, when, when you have so many of these, uh, s- agentic setup, like multi-agent systems, you know, agents and sub-agents coming together, you probably need a governance structure, right? So that covers all the- Yeah legal compliance, ethics, cybersecurity.
131
00:18:57,705.687411 -->
00:18:57,985.687411
Yeah.
132
00:18:57,995.688411 -->
00:19:01,835.693411
Can you share more about how you're thinking about it? What have, what have you set up or? Yeah, yeah.
133
00:19:01,835.693411 -->
00:19:10,445.693411
I mean, we, we, we, we established our, our governance la- uh, you know, framework, uh, right from the beginning, right? Mm-hmm.
134
00:19:10,445.693411 -->
00:19:25,935.691411
And, uh, you know, it starts with, uh, we call, you know, LCE, which is our legal compliance ethical, uh, right? And, uh, w- we, we, uh And then there's kind of the product and kind of technology overview, uh, steps or pieces.
135
00:19:26,435.691411 -->
00:19:46,615.686411
Uh, the legal compliance ethical goes through, you know, making sure we understand the, the terms and conditions of the models that we're using, uh, right? Uh, we understand the, the specific, uh, data privacy and data transfer rules, which we already know anyway, so those rules are kind of already embedded by default.
136
00:19:47,255.690411 -->
00:19:59,435.682411
Uh, we understand the, um, uh, you know, the ethical use of s- of some of those use cases and, and, and ensuring that, for example, PII data is protected and it's not going to the right places.
137
00:19:59,445.685411 -->
00:20:25,685.695411
So those things are very rigid for us and, and very, uh, robust, right? Uh, where we have, you know, people reviewing every model before we decide to deploy it, and then people reviewing every use case with the workflow before we decide to deploy it, and those are people from our legal team and our risk, uh, chief risk officer's team- Got it um, along with our technology and, uh, product groups.
138
00:20:26,575.690411 -->
00:20:39,25.690411
And where does humans remain in the loop? You know, where, you know, where, like when, as you move into more, you know, autonomous agents, is that, like, a risk decision or a regulatory decision, you know? Yeah, I think it's all of those.
139
00:20:39,95.694411 -->
00:20:47,983.692411
Uh, you know, w- w- we- Allow you to dec- we, we let you make that decision on your own, right? Yeah.
140
00:20:48,63.692411 -->
00:20:56,993.692411
Uh, a- as you, you know, we can give you pre-canned, uh, workflows as I described- Mm-hmm uh, you know, to, for autonomous decisions.
141
00:20:57,213.692411 -->
00:20:57,453.692411
Yeah.
142
00:20:57,553.691411 -->
00:21:08,383.691411
Uh, or, uh, or, or and, and even in those workflows you can insert yourself in any of the steps- Mm that, you know, we decided an agent could perform, right? Yeah.
143
00:21:09,3.692411 -->
00:21:27,653.690411
Uh, or you can sit all the way at, at the end and, and make that final decision, like I said, to issue credit or to onboard that supplier, uh, or to issue, you know, or send a, a, a marketing email to a particular customer, right? So we're, we're letting you do that final step yourselves.
144
00:21:27,663.692411 -->
00:21:27,683.692411
Yeah.
145
00:21:27,703.693411 -->
00:21:29,663.689411
Uh, but or we can help you automate it.
146
00:21:30,223.689411 -->
00:21:30,543.689411
Mm.
147
00:21:30,723.689411 -->
00:21:31,53.691411
Got it.
148
00:21:31,573.692411 -->
00:21:39,323.691411
So as you g- go on the agentic journey, as many companies are going on that there's this token maxing problem people are running to, right? Mm-hmm.
149
00:21:40,553.692411 -->
00:21:44,853.692411
Lot of interest right now in small language models as a way to drive down costs, you know.
150
00:21:44,863.692411 -->
00:21:45,203.692411
Yeah.
151
00:21:45,293.692411 -->
00:21:48,393.692411
How has D&B approached this internally, you know? Yeah.
152
00:21:48,473.692411 -->
00:21:58,453.692411
I mean, so we've, we've, uh, you know, as I mentioned, you know, we, we have sort of a- an ongoing pipeline of continuous sort of discovery and qualification of different models.
153
00:21:58,643.691411 -->
00:21:58,903.691411
Mm.
154
00:21:59,183.692411 -->
00:22:06,913.691411
Uh, uh, small language models, open sourced, uh, small language models are definitely on our list.
155
00:22:07,33.692411 -->
00:22:12,553.691411
Uh, we've already deployed, uh, them at least, uh, across at least seven different, uh, use cases.
156
00:22:12,773.690411 -->
00:22:13,53.690411
Mm.
157
00:22:13,483.693411 -->
00:22:27,953.690411
Um, and you know, we've seen literally, like 95, 97% reduction in token costs- Mm For some of those tasks, uh, where we use them- Wow which is, uh, s- very significant.
158
00:22:28,63.691411 -->
00:22:28,513.690411
Yeah, yeah.
159
00:22:28,563.691411 -->
00:22:53,303.687411
Not just that, I mean, it's also letting us improve, uh, dramatically the quality, uh- Yeah uh, or accuracy of the results, right? Because you've got sort of these small language models that are very focused, right? Uh, much more focused on specific, um, you know, steps, right? Or tasks- Yeah They're performing and, and the results we've, we've seen are, are, are, uh, very significant.
160
00:22:53,313.687411 -->
00:22:57,483.688411
Like, we have one product that we released, we call it, uh, company summaries.
161
00:22:57,873.692411 -->
00:22:58,83.692411
Mm.
162
00:22:58,133.687411 -->
00:23:14,853.687411
Um, and it's, it's very s- you would think it's very simple, uh, but it's actually, uh, you know, from a customer lens perspective, one of the most complex things, which is really understanding mu- on a much deeper level what does a company actually do for a living? Mm.
163
00:23:14,863.690411 -->
00:23:21,703.694411
So if you think about from an insurance underwriting perspective, that's a very in- that's a very difficult, uh, thing to understand.
164
00:23:21,713.689411 -->
00:23:32,453.686411
Customers are very, you know, it's self-reported- Mm Usually, right? So, uh, you know, if you have a landscaping company that, uh, tells you that they do only...
165
00:23:32,483.693411 -->
00:23:49,695.692411
And they're supplying for workers' comp insurance, and they tell you that they do, um, uh, they only cut grass, right? Uh, and uh, the SIC codes and the industry codes that we provide you- Basically confirmed that that's a landscaping company, and that's all you need to know.
166
00:23:50,225.692411 -->
00:24:00,255.692411
What we've been able to do is look at the company sort of website plus the different other kind of sources of information that we can collect about that company.
167
00:24:00,905.692411 -->
00:24:21,185.691411
And then we use, uh, SLMs across the different tasks, of like identifying the data, the specific information across the different websites, pulling it down, uh, then, uh, filtering and prioritizing the information that's needed, uh, that is, you know, that's needed to be distilled down and trans- translated, so to speak.
168
00:24:21,195.691411 -->
00:24:21,205.691411
Mm.
169
00:24:21,585.690411 -->
00:24:24,575.692411
And then we feed it into a, a, a large language model.
170
00:24:24,945.691411 -->
00:24:25,215.691411
Mm.
171
00:24:25,275.692411 -->
00:24:28,775.691411
And that creates a much deeper insight into that company.
172
00:24:28,775.692411 -->
00:24:35,45.690411
And so in the example of the landscaping company, we were able to also decipher that they also do tree pruning.
173
00:24:35,815.690411 -->
00:24:36,115.692411
Ah, Interesting.
174
00:24:36,115.693411 -->
00:24:41,315.692411
Right? And that gives you a very different, obviously, workers' comp insurance premium than than just cutting grass.
175
00:24:41,815.690411 -->
00:24:42,305.690411
Very Cool.
176
00:24:42,305.690411 -->
00:24:42,345.689911
Very cool.
177
00:24:42,345.689911 -->
00:24:45,965.689411
We have a lot of insurance companies that are very interested in that kind of a capability.
178
00:24:45,965.691411 -->
00:24:55,525.687411
And then, and, uh, like I said, the production of that, uh, summarization went down by 95% as a result of use of the, uh, small language models.
179
00:24:56,35.691411 -->
00:25:03,495.693411
And this is where like, you know, models and LLMs can actually help beyond just simple rules, right? Because they can explore- Yeah the whole related concept space.
180
00:25:03,525.689411 -->
00:25:04,225.689411
Exactly.
181
00:25:04,295.693411 -->
00:25:04,895.693411
Exactly.
182
00:25:04,895.693411 -->
00:25:20,795.687411
And that's, that's what's amazing about this technology, right? It's like, you know, you, you, you, you continuously discover, you know, a lot, you know, a lot of new things that you would never have thought, even for customers never have thought that they would be even interested in.
183
00:25:20,835.686411 -->
00:25:27,115.689411
And then we, you know, we created it and then it's like, oh my God, that's all, it's all we needed, right? It's like the hottest selling product.
184
00:25:27,775.689411 -->
00:25:28,235.689411
Awesome.
185
00:25:28,355.690411 -->
00:25:28,895.687411
That's amazing.
186
00:25:29,365.693411 -->
00:26:11,119.692411
So what is the, you know, practical difference in compute cost between grounding agents in a verified deterministic data source versus sending out to, uh, to collect open information on the web? You know, what does that look like- Yeah in this for you? So we've seen that, uh, you know, as, uh, as we think of, as you think about, uh- You know, Dun Bradstreet basically creating, uh, uh, pre-calculated answers, right? Uh, so if you think about, I don't know, a credit score that I just mentioned, right? Or a corporate hierarchy, uh, that is predetermined that, you know, where we verify that this company is a subsidiary of this other company- Mm.
187
00:26:11,169.691411 -->
00:26:52,199.691411
Right? That kind of deterministic answers, so to speak, right, uh, are, you know, are extremely efficient when you talk a- when you think about, you know, agentic, uh, agents that would have to otherwise go on the web, you know, scrape the web, understand the different companies that are relevant to this company, figure out how they belong to each other on the tree or, or actually if you're asking, uh, you know, for, uh, an agent to perform a risk assessment, uh, right, just based on, you know, loosely available information that's, that's out there and having it connect the dots together and produce the answer.
188
00:26:52,199.692411 -->
00:27:01,259.691411
So just logically, right, the reduction in costs would be, you know, quite significant, right? And, and so w- we've seen the ranges.
189
00:27:01,259.693411 -->
00:27:09,879.688411
I mean, they're, you know, anything from like 50 to 70 to 80% savings- Mm in terms of token costs because you already have the answer served up to you.
190
00:27:09,879.688411 -->
00:27:13,69.690411
You don't need the agent to do the compute, to perform the compute to calculate it.
191
00:27:13,939.694411 -->
00:27:14,309.694411
Awesome.
192
00:27:14,819.691411 -->
00:27:16,99.688411
Let's take some audience questions.
193
00:27:16,99.690411 -->
00:27:18,289.690411
There is a related question here from Sridhar.
194
00:27:18,349.690411 -->
00:27:34,969.692411
Uh, what are the most significant risks enterprises face in proving provenance and lawful acquisition of data used by AI systems while also complying with rapidly evolving AI regulations across the US and globally? Yeah, so that's a big question.
195
00:27:35,9.692411 -->
00:27:52,979.691411
Uh, I think that, um the d- it, it depends on the size of the enterprise- Yeah uh, obviously, right? Uh, but the risks, uh, you know, are, are The risks are very much evolving, right? Right.
196
00:27:52,979.691411 -->
00:28:40,619.692411
So if you think about, um, the systems of record, legacy systems of record i- in a bank, right, uh, that are so, like, dispersed across the organization, um, y- y- you have a lot of companies, a lot of those large organizations, uh, under the pressure of their CEO saying, "We must go AI," right? Are all running super fast towards- Yeah deployment, right? And then they s- have to rapidly stop because they realize that they have to first figure out what's in those systems of record, right? A- and then they have to decipher what's PII, what's, uh, what's not, right? Then it's like, once they figured that out, then they have to extract that.
197
00:28:41,59.692411 -->
00:28:45,679.691411
Uh, they then have to standardize it in some way, right? Mm.
198
00:28:45,679.691411 -->
00:29:00,989.692411
Uh, to make it AI ready or, you know, consumable, right? Uh, then they have to make sure that it follows all the, uh, you know, data residency rules, and it follows the And then you also have to figure out if you're deploying the right model, right? Yeah.
199
00:29:00,999.691411 -->
00:29:10,279.694411
Like the model itself, whether you're open source model from China or you're using a, a, a, you know, a, a, an open source model from US.
200
00:29:10,479.687411 -->
00:29:13,869.692411
I mean, all those things have to be, uh, ver- verified, validated.
201
00:29:13,869.692411 -->
00:29:18,969.692411
Uh, and so the job is quite significant, right? Mm-hmm.
202
00:29:18,969.692411 -->
00:29:43,9.688411
And the risks are there, right? And then on top of that, once you deploy, you then have to make sure you're monitoring to, to understand that things are not being done by these agents that you haven't authorized them to do, right? So you have to create a lot of this entitlement layers and risk management layers to make sure that you, you, you, uh, you know, create like rigid, rigid controls around these things.
203
00:29:43,579.693411 -->
00:29:43,939.691411
Yeah, absolutely.
204
00:29:43,969.692411 -->
00:29:46,839.690411
Um, so there's a lot of, there's a lot of work there, I think.
205
00:29:46,889.690411 -->
00:29:47,279.690411
Yeah.
206
00:29:47,339.687411 -->
00:29:50,999.694411
We call it like the sandwich situation between the FOMO and the FOMO.
207
00:29:51,59.694411 -->
00:29:56,429.682411
You know, everyone has- Yes, yes FOMO AI and the FOMO- Yeah is the fear of messing up, you know? And, and that- Yeah.
208
00:29:56,429.691411 -->
00:29:59,619.685411
Yes, exactly every enterprise AI leader is facing right now.
209
00:29:59,979.694411 -->
00:30:12,959.691411
So when it comes to D&B, um, are you, like, sort of looking at, like, NAIC and EU AI Act and all of these regulations closely? You know, how do you think about the, like, regulatory landscape evolving, you know? Oh my God.
210
00:30:13,279.682411 -->
00:30:13,749.682411
Yeah.
211
00:30:13,759.693411 -->
00:30:25,249.693411
I mean, if you talk to our chief risk, uh, chief compliance officer, you know, she's gonna She monitors how many new regulations she reviews on, like, a, you know, quarterly basis, right? Yeah, yeah.
212
00:30:25,369.693411 -->
00:30:42,399.692411
Uh, and so what we've done is we, we, you know, we have kind of a forward-looking Like, she does all the pre-work- Yeah Uh, for us, uh, not just what's out there already and been established, but what's been, what's also coming, right? Yeah.
213
00:30:42,399.692411 -->
00:30:46,699.692411
And then so also what's, like, kind of horizon three, what, what may be coming, right? Right.
214
00:30:46,719.692411 -->
00:30:52,179.691411
At some point because there are rumors and whispers about it, right? And, and, uh, and that, uh...
215
00:30:52,279.691411 -->
00:31:10,539.693411
So I'm kind of answering more generically, right? You know, and, and that, uh, pipe of that sort of view that they provide us- Yeah Allows us to then build, uh, and design things in accordance, right? Uh, to make sure that we're accommodating for all those different, you know, changes.
216
00:31:11,149.690411 -->
00:31:11,469.690411
Yeah.
217
00:31:12,9.691411 -->
00:31:19,889.691411
One of the things that is on the rise right now is this multi-agent setups, right? So where agents can communicate with other agents, A2A protocols, MCP, you know, these are all...
218
00:31:20,309.690411 -->
00:31:30,699.689411
So when you are seeing, uh, you know, multi-agent setups, when systems from different vendors hand off each other- Mm-hmm how do you keep data verification intact across the whole chain? Yeah.
219
00:31:31,349.691411 -->
00:31:33,109.692411
So there's a, there's, kind of, a few things.
220
00:31:33,139.692411 -->
00:31:35,499.690411
You know, there is, uh...
221
00:31:35,499.692411 -->
00:31:43,983.692411
You know, we, we built this, uh, um- Our match and business verification agent- Mm.
222
00:31:43,993.692411 -->
00:31:46,483.692411
Right? That kind of goes horizontally, right? Mm.
223
00:31:47,343.692411 -->
00:31:55,93.692411
Uh, we think about that as almost like a digital handshake between the different vendor, uh, between the different agents.
224
00:31:55,383.692411 -->
00:31:55,683.692411
Mm.
225
00:31:55,793.692411 -->
00:32:09,213.691411
So as agent, you know, one performs a particular task, uh, on a particular entity, right? It can then call out, uh, uh, and it passes that task over to the next agent.
226
00:32:09,483.691411 -->
00:32:09,773.691411
Mm.
227
00:32:09,823.691411 -->
00:32:20,833.689411
Right? That next agent can call out our, our entity verification agent basically match or business verification- Mm to make sure that it's still talking about the same entity, right? Mm.
228
00:32:20,843.689411 -->
00:32:20,853.689411
Mm.
229
00:32:21,33.689411 -->
00:32:22,943.693411
It's still performing the task of the same entity.
230
00:32:23,293.691411 -->
00:32:23,573.691411
Mm.
231
00:32:23,633.691411 -->
00:32:24,223.691411
So I, I...
232
00:32:24,243.691411 -->
00:32:39,223.691411
That's definitely, uh, an area that, uh, is, you know, we've seen people, you know, kind of incorporate these, these, uh, common, I would say common components- Mm right, into their workflows, right? Mm-hmm.
233
00:32:39,223.691411 -->
00:32:57,663.688411
Um, but we've also of course seen, you know, a lot of customers, and we, we think about it as well a lot, right? Is how do you verify the agents themselves- Yeah And make sure that they belong to the ent- to the business, this is the they, uh, you know, say they belong to, right? Yeah.
234
00:32:57,663.688411 -->
00:33:13,173.687411
Because especially if you're not sitting in the same company and you're not orches- you didn't orchestrate a multi-agent, uh, you know, workflow yourself, your interact- your agent is interacting with some third party agent that's sitting on, on with a counterparty on the other side.
235
00:33:14,263.688411 -->
00:33:35,833.689411
That needs to be verified as well, right? So that, that the verification and the, the, the trust between two, the two agents has to be verified as well, right? So there's something there as well that we're also looking at, and- Yeah um, making sure that we're accommodating for that trust and making sure that they trust each other before the, the, the, the job is done.
236
00:33:36,613.687411 -->
00:33:36,983.687411
Yeah.
237
00:33:37,203.690411 -->
00:33:43,823.690411
And also, like, uh, data staleness could be an issue as well, right? When an agent pulls data is an anonymous question as well here.
238
00:33:44,273.690411 -->
00:33:50,583.685411
When agent- Oh, yeah when agent pulls data from an outdated data source, how do you catch it before it affects, like, its output? Yeah.
239
00:33:50,813.685411 -->
00:33:53,53.690411
And then it passes on to another agent, you know, downstream.
240
00:33:53,113.687411 -->
00:33:53,293.687411
Yeah.
241
00:33:53,393.685411 -->
00:33:53,913.685411
Yeah.
242
00:33:53,933.694411 -->
00:33:57,233.694411
I mean, that's, uh, that's a, that's a difficult one.
243
00:33:57,493.691411 -->
00:34:04,355.692411
Um- You know, to, to, uh, kind of capture beforehand, right? Mm-hmm.
244
00:34:04,355.692411 -->
00:34:17,695.692411
I, I think, you know, it kind of goes a little bit back to the monitoring for your agentic for the agent behavior- Yeah uh, making sure they follow the right steps, right? Uh, and then periodically checking in if the outputs are, are the same.
245
00:34:18,125.692411 -->
00:34:33,425.689411
Uh, you know, we have also put a kind of a, a more robust program around, um, if somebody changes something upstream, uh, in our data, you know, supply chain, right? Mm-hmm.
246
00:34:33,435.690411 -->
00:34:43,535.691411
That has to be communicated and, and there's all kinds of, you know, operational processes in place to make sure that the, the, you know, that that information gets captured.
247
00:34:44,455.690411 -->
00:34:44,995.692411
So- Got it.
248
00:34:45,335.691411 -->
00:34:52,805.690411
And does, does that imply more into the data security and kind of, you know, what controls that you put in place when- Yeah, it's- all data sources? Yeah.
249
00:34:52,965.692411 -->
00:34:54,245.691411
Yeah, it's part of that.
250
00:34:54,325.689411 -->
00:34:54,655.689411
Mm-hmm.
251
00:34:54,815.691411 -->
00:34:55,585.689411
It's part of that.
252
00:34:55,695.692411 -->
00:35:03,345.689411
But, you know, it's, it's on stream because we have to monitor for, like you said, still, you know, quality- Right accuracy, you know, timeliness, things like that.
253
00:35:03,865.690411 -->
00:35:04,295.691411
Got it.
254
00:35:04,755.690411 -->
00:35:11,925.690411
And, and I think there's a related question as well around the variance of AI models, and you mentioned you're using a collection of small models and big models.
255
00:35:12,425.688411 -->
00:35:29,765.693411
Kenneth fra- uh, from our audience is asking, "How do you manage the variances of AI models and the versions as they, that are used as they progress? Do you upgrade or change models? And how do you handle backup or continuity of services if there are security breaches or natural disaster interruptions?" Yeah.
256
00:35:29,765.693411 -->
00:35:51,381.691411
So we manage So, you know, we have a bit of a controlled environment- Mm-hmm uh, where we don't just- Give everything to everybody- Mm Right? You know, it goes through a vetting process- Mm Um, you know, where they're tested and validated- Mm And then deployed, right? Mm.
257
00:35:51,381.692411 -->
00:35:59,451.692411
And then we go through those, you know, we re-review what we've tested and validated and deployed- Yeah right, uh, back.
258
00:36:00,61.692411 -->
00:36:08,671.691411
Uh, and, uh, we are also able to swap out models quite- Mm quickly, uh, between one another.
259
00:36:09,151.692411 -->
00:36:09,261.692411
Mm.
260
00:36:09,501.690411 -->
00:36:28,181.692411
And, uh, you know, in some cases we have, depending on the criticality of the case- Mm Uh, use case, we, uh, almost have backup models- Mm right, uh, that we have already tested and know that they can perform relatively similar to the model that we just deployed for this use case.
261
00:36:28,531.689411 -->
00:36:28,741.689411
Mm.
262
00:36:28,901.692411 -->
00:36:47,881.690411
Um, so and then, you know, of course, if you think about, uh, you know, uh, open source models, where they're coming from, right? There's the whole kind of, uh, conversation about and the whole idea around on-prem and making sure- Yeah we're me- you know, we have a...
263
00:36:48,731.694411 -->
00:36:49,851.690411
It's containerized.
264
00:36:50,211.690411 -->
00:36:50,551.690411
Mm.
265
00:36:50,601.689411 -->
00:36:55,71.690411
Um, and, uh, you know, to make sure where there's no sort of leakages.
266
00:36:55,491.688411 -->
00:36:55,861.688411
Right.
267
00:36:56,171.689411 -->
00:36:56,491.689411
Yeah.
268
00:36:57,431.691411 -->
00:37:05,461.690411
So this is an interesting question, right? You know, foundation model labs out there, you know, buying data sets, you know, kind of outgrowing their sort of, uh, frontier.
269
00:37:05,461.690411 -->
00:37:11,591.690411
And D&B seems to have made a call that, you know, your data is not allowed to be used to, for model training.
270
00:37:11,601.689411 -->
00:37:19,831.691411
How do you think- Yeah about where the line should be? You know, what should enterprises be asking themselves before they make a decision around, like, giving the data for model training? Yeah.
271
00:37:19,831.692411 -->
00:37:24,51.693411
I mean, yeah, so, you know, obviously, you know, training...
272
00:37:24,461.690411 -->
00:37:33,965.692411
First of all, training is, is a, uh, often misunderstood, uh, you know, word, right? Yeah.
273
00:37:34,245.692411 -->
00:37:40,395.692411
Uh, if you talk to our customer lawyers and, uh, they will say, "We want training rights."
274
00:37:40,825.691411 -->
00:37:41,165.691411
Mm-hmm.
275
00:37:42,45.692411 -->
00:37:49,775.692411
And then when you talk to the business people, and then you talk to the technical people, actually they don't want training, right? They actually just want RAG, and it's just another variation of this.
276
00:37:50,285.691411 -->
00:37:50,305.691411
Yeah.
277
00:37:50,335.691411 -->
00:37:57,225.692411
So majority of our customers don't actually ask us for training even- Yeah you know, even after we say no, they still don't ask us, right? Yeah.
278
00:37:57,225.692411 -->
00:37:59,645.690411
It's the lawyers who typically do and then fight with us.
279
00:37:59,915.691411 -->
00:38:00,205.691411
Yeah.
280
00:38:00,355.691411 -->
00:38:05,255.689411
Um, but that said, right, it's still a, it's still a difficult, uh...
281
00:38:05,255.691411 -->
00:38:09,885.691411
obviously, you know, for a data provider, that is, this is our bread and butter, right? Mm-hmm.
282
00:38:10,315.691411 -->
00:38:19,735.690411
And so, you know, we, we, we've, um, never had a case where we let anybody train, right? Mm-hmm.
283
00:38:19,735.690411 -->
00:38:29,835.689411
Uh, but if we, if we would, and if anybody really, really, really wants it and they really, really mean it- Mm-hmm Right? There, there are ways to containerize this, right? Mm-hmm.
284
00:38:29,835.689411 -->
00:38:42,495.694411
And, and still allow us to retain some control- Mm-hmm Uh, of the data of the model that's trained- Yeah on that data, right? And so there is ways to do that without having to completely...
285
00:38:42,495.694411 -->
00:38:56,615.687411
You know, so if we have enough control like that- Mm-hmm right? Hey, we put it in a, in a, in a, in a, um, in a container, right? It's, it's, uh, you know, like a data room type of, uh, right, uh, uh, environment.
286
00:38:56,675.689411 -->
00:38:57,725.688411
Uh, we have control.
287
00:38:57,725.692411 -->
00:38:58,545.691411
You have control.
288
00:38:58,545.692411 -->
00:38:59,725.687411
Nobody else has control.
289
00:38:59,725.692411 -->
00:39:00,975.692411
We have control of the model.
290
00:39:01,365.691411 -->
00:39:01,505.691411
Yeah.
291
00:39:01,615.691411 -->
00:39:03,539.0227444
You know, we can pull the plug, right? Mm-hmm.
292
00:39:03,539.0227444 -->
00:39:08,685.694411
If we feel like there's something is happening that is impacting our, our business.
293
00:39:09,315.688411 -->
00:39:09,555.690411
Got it.
294
00:39:09,565.688411 -->
00:39:14,395.686411
So you're investing a lot on the small lang- language model training and development on, on your proprietary models.
295
00:39:14,505.691411 -->
00:39:15,555.689411
Yes, yes.
296
00:39:15,555.689411 -->
00:39:16,205.691411
Yes, we're doing that.
297
00:39:16,205.691411 -->
00:39:16,515.688411
Yeah.
298
00:39:16,665.687411 -->
00:39:17,155.689411
Makes sense.
299
00:39:17,545.692411 -->
00:39:35,215.687411
So models themselves are, im- improving incredibly quickly, right? So what, you know, when you kind of step back a little bit, what part of the enterprise AI stack do you think that will actually become more important as models get commoditized, and everyone has roughly the same kind of models across the board, open weights, closed weights? Yeah, yeah.
300
00:39:35,235.687411 -->
00:39:44,635.693411
Well, I'm gonna give you the biased and the non-biased answer, right? Uh, you know, we think about obviously, you know, the data, right? Mm-hmm.
301
00:39:44,645.690411 -->
00:39:52,255.691411
That context layer- Mm-hmm Right, we, we think will become the, the, the, the...
302
00:39:52,255.691411 -->
00:39:55,945.689411
will become that val- most valuable, uh, component, right? Yeah.
303
00:39:56,665.694411 -->
00:39:59,485.692411
I, I also think there's the, the workflows, right? Mm-hmm.
304
00:39:59,595.687411 -->
00:40:17,445.692411
I think there's, there is that aspect of, um- Uh, uh, you know, there, you have to be a subject matter expert, and you have to know the data that you're dealing with to be able to write the best agent to interact with that data, right? Yeah.
305
00:40:17,445.692411 -->
00:40:19,165.692411
And, and give you the best results.
306
00:40:19,665.692411 -->
00:40:19,885.692411
Mm-hmm.
307
00:40:19,915.692411 -->
00:40:43,855.690411
Uh, but I also think that, so that whole sort of idea around, you know, verification and, and validation and, and, uh, grounding and- Mm-hmm Of the workflow, right? Or verification of the workflow, validation of the workflow, making sure that the agent is performing the right tasks, right? I think those types of things, you know, will become more and more, uh, critical, right? Mm-hmm.
308
00:40:43,855.690411 -->
00:40:50,15.690411
And, and there may be f- you know, not a lot of those kind of providers that can help understand that.
309
00:40:50,45.692411 -->
00:40:50,925.690411
And, you know, it...
310
00:40:50,955.692411 -->
00:40:58,345.690411
And it's some combination of the data provider- Mm-hmm Making sure that, yes, it pulled the right information, it executed on the right information.
311
00:40:58,705.692411 -->
00:40:58,955.690411
Mm-hmm.
312
00:40:59,45.692411 -->
00:41:11,195.687411
Right? And then the technology provider who can give you that additional layer of insight that, that says, "Yes, you know, that task was done correctly," b- based on the subject matter expert that designed it.
313
00:41:11,965.694411 -->
00:41:12,315.692411
Yeah, yeah.
314
00:41:12,315.693411 -->
00:41:17,905.689411
We call it, like, the three Cs, the context, the control, and then once you have context and control, you can compound on the AI.
315
00:41:17,905.689411 -->
00:41:18,375.689411
Yeah.
316
00:41:18,415.691411 -->
00:41:19,35.687411
There you go.
317
00:41:19,35.687411 -->
00:41:19,155.689411
Exactly.
318
00:41:19,155.689411 -->
00:41:19,785.689411
Exactly.
319
00:41:19,815.693411 -->
00:41:21,145.693411
You said it better than I did.
320
00:41:21,165.691411 -->
00:41:22,445.687411
No, it's, it's great.
321
00:41:22,445.690411 -->
00:41:22,635.690411
Yeah.
322
00:41:22,945.690411 -->
00:41:27,395.694411
So I guess there is a, you know, most of, uh, companies have a lot of their enterprise data.
323
00:41:27,455.692411 -->
00:41:29,205.694411
They may have legacy data infrastructure.
324
00:41:29,205.694411 -->
00:41:30,835.691411
There's an audience question as well.
325
00:41:31,325.687411 -->
00:41:34,115.687411
You know, that's probably not yet built for AI agents.
326
00:41:34,515.690411 -->
00:41:44,485.687411
You know, even, you know, given your experience of, you know, where you have come from and where you've done so far, what you've done so far on agentic AI, what do you advise them to start? You know, where, you know, where do you...
327
00:41:44,485.687411 -->
00:41:50,295.690411
Where, where can they start, you know, on this agentic journey, uh, to layer on agents on their existing data? Yeah.
328
00:41:50,575.687411 -->
00:41:59,205.691411
And I, I, I think, you know, w- we, we, uh, I think you first have to get that sort of, uh, the, the, the, the governance framework.
329
00:41:59,315.693411 -->
00:41:59,595.689411
Mm-hmm.
330
00:41:59,685.688411 -->
00:42:06,385.686411
Right? Uh, that's one, right? The second, I think, is you need the, you need to get the data straight.
331
00:42:06,585.689411 -->
00:42:06,985.688411
Mm-hmm.
332
00:42:06,995.693411 -->
00:42:17,885.692411
Uh, and that's probably the second most important thing that people, um, you know, they run towards let me buy the latest tools, uh, to build stuff, right? Mm-hmm.
333
00:42:17,885.692411 -->
00:42:22,385.692411
And then they realize they don't have anything to, to put in there, right? Mm-hmm.
334
00:42:22,385.692411 -->
00:42:26,735.691411
Um, and then of course, you know, the third is you need the tools, right? Mm-hmm.
335
00:42:26,735.691411 -->
00:42:31,295.691411
The, the s- the stack that allows you to scale quickly, right? Mm-hmm.
336
00:42:31,295.692411 -->
00:42:32,345.692411
And it's not, uh...
337
00:42:32,435.692411 -->
00:43:03,535.690411
You know, the tools are available, but it's about, depending on the enterprise that you, you know, the company you're, you're part of, uh, you have to layer in your own proprietary sort of controls, uh, right? Uh, risk controls and, and, uh, uh, data access controls, uh, data quality check controls, uh, agent monitoring capabilities, right? So there's a bunch of layers of s- and of course, your infrastructure, right, to make sure it's, it's located in the right place.
338
00:43:03,975.691411 -->
00:43:06,565.691411
Uh, you're monitoring for costs and compute.
339
00:43:06,925.692411 -->
00:43:17,955.691411
I mean, these are all pieces, right, that you have to put together, uh, uh, more holistically, right, um, before you rush to, to build, uh, stuff.
340
00:43:18,45.690411 -->
00:43:33,755.692411
Uh, because then it be- you know, if you, if you rush to build without doing those steps first- Yeah then you can get yourself, I think, into potentially a lot of trouble, um, you know, by, you know, because you're using unauthorized model or you're- Yeah.
341
00:43:34,65.687411 -->
00:43:38,225.693411
Mm-hmm putting data that wasn't authorized, right, to be put into a particular model.
342
00:43:38,565.687411 -->
00:43:45,55.693411
Or you, uh, you know, created a tool, or you, you bought a platform that doesn't allow you to scale, right? Mm-hmm.
343
00:43:45,55.693411 -->
00:43:53,665.690411
And every single agent that you build requires, you know, 10 people to approve and, and five technology teams to deploy.
344
00:43:54,565.687411 -->
00:43:56,75.686411
Um, you know, so those types of things.
345
00:43:56,95.686411 -->
00:43:59,5.688411
And then the, you know, the data that you're, you're...
346
00:43:59,245.687411 -->
00:44:15,65.690411
Most importantly, right, if you're starting to deploy these agents as assistants, let's say, in your, in your, in your organization, you're realizing that, uh, um, those, uh, the answers those agents produce are not reliable.
347
00:44:15,205.687411 -->
00:44:24,975.695411
And then now you have all these problems with people making the wrong decisions and taking the wrong steps because you gave them something that was producing the wrong answer.
348
00:44:26,331.692411 -->
00:44:28,661.692411
you could use the wrong data, so yeah Awesome.
349
00:44:28,801.692411 -->
00:44:29,71.692411
Yeah.
350
00:44:29,121.692411 -->
00:44:32,891.692411
Yeah, start with like agent governance, data governance, and put, put all these things in place.
351
00:44:32,891.692411 -->
00:44:33,251.692411
Yeah.
352
00:44:33,891.692411 -->
00:44:34,321.692411
Awesome.
353
00:44:34,501.692411 -->
00:44:37,461.692411
So I guess, you know, three years is a long timeframe.
354
00:44:37,461.692411 -->
00:44:40,711.692411
You know, like three years have, you know, ago a lot, lot of things were different.
355
00:44:40,981.692411 -->
00:44:43,931.691411
So now let's look at three years forward, right? Yeah.
356
00:44:44,161.692411 -->
00:45:04,901.691411
Where does the world go? You know, what, where, what do you see? You know, or do you see a meaningful percentage of business decisions being initiated and executed by agents rather than humans? You know, what, where, w- what does, what does it- Yeah World look like? Yeah, I Mean, yes, I think you're gonna have a lot of more autonomous- Mm Uh, decisions, uh, made, right, and, and workflows created.
357
00:45:04,901.692411 -->
00:45:24,651.691411
Uh, I think you're gonna have a lot of, um, uh, cross, uh, you know, B2B interactions, right, uh, with my counterparties performed by agents instead of people, right? Everything from, you know, e-commerce, right, you know, purchasing decisions, marketing decisions.
358
00:45:24,871.689411 -->
00:45:25,111.689411
Mm.
359
00:45:25,251.690411 -->
00:45:42,991.692411
Um, uh, you know, um, decisions around, uh, uh, you know, whether, you know, communication, right, uh, uh, to, to and from and who and, and how, and all those things.
360
00:45:42,991.692411 -->
00:45:48,21.690411
I think there's gonna be a lot of that, uh, just, just a lot of agents running around basically.
361
00:45:48,41.691411 -->
00:45:48,51.691411
Mm.
362
00:45:48,381.687411 -->
00:46:14,201.692411
Right? Um, and so, you know, I, I think there may be, uh, there's gonna be a, a need to make sure those agents are controlled, um, you know, uh, and, and, uh, monitored, right? Just like you would imagine people would be, right? So you have to build the right controls to make sure that, uh, they don't get out of control, uh, right, a- and run amok and make the wrong decisions.
363
00:46:14,761.692411 -->
00:46:25,541.687411
So I, I think that will be, you know, a, a much more robust, uh, environment where you wouldn't even know sometimes who you're speaking to as a person, whether- Mm it's a person or an agent on the other side.
364
00:46:25,941.693411 -->
00:46:26,351.693411
Yeah.
365
00:46:26,431.690411 -->
00:46:26,781.690411
Yeah.
366
00:46:26,951.687411 -->
00:46:27,741.692411
Yeah, absolutely.
367
00:46:27,741.692411 -->
00:46:32,351.692411
It was, it was just like to share like, uh, an in- in- interesting, interesting incident.
368
00:46:32,351.692411 -->
00:46:41,51.695411
I was actually filing, like, a bug on my Slack channel on my team last night, and then- Mm-hmm And then my engineer was just, like, talking to a platform bot.
369
00:46:41,121.685411 -->
00:46:57,961.691411
And then, and then, and then basically by the morning it created a PR and submitted it, right? And so we're living in a very different world Now- Did he know? Did He know he was talking about it? Did he know he was talking to a bot or no? I'm actually so I was, I was actually posting as I was talking to a human, but then he was getting the work done by a bot, right? So- Got It.
370
00:46:57,961.691411 -->
00:46:59,141.681411
Yeah that was very interesting.
371
00:46:59,231.686411 -->
00:46:59,481.686411
Yeah.
372
00:46:59,571.682411 -->
00:46:59,971.682411
Yeah.
373
00:47:00,871.685411 -->
00:47:03,461.692411
That's a It's, uh, that's, I think that's what's gonna happen.
374
00:47:03,761.685411 -->
00:47:05,91.687411
So- Yeah, absolutely.
375
00:47:05,91.687411 -->
00:47:08,871.690411
So let's end this session with, like, a, you know, 10 question rapid fire.
376
00:47:08,911.694411 -->
00:47:11,991.691411
You know, your hot takes, you know, one word answers, right? Yeah.
377
00:47:12,311.688411 -->
00:47:14,61.684411
Or, or what- whatever you wanna, uh, share.
378
00:47:14,371.685411 -->
00:47:19,89.692411
Uh, data or models, which matters more? Ah, data.
379
00:47:19,649.692411 -->
00:47:19,829.692411
Okay.
380
00:47:20,509.692411 -->
00:47:20,839.692411
Data.
381
00:47:21,619.692411 -->
00:47:24,399.692411
G- well said from a chief data officer.
382
00:47:24,619.692411 -->
00:47:34,773.692411
Okay, so, uh, open source or closed source? Uh, depends on the time of the day.
383
00:47:35,803.692411 -->
00:47:38,623.692411
You can pick both sometimes, you know, it Is- Yeah, it's probably both, yeah.
384
00:47:39,33.692411 -->
00:47:39,53.692411
Yeah.
385
00:47:39,413.692411 -->
00:47:42,453.692411
Big models or small models? Hmm.
386
00:47:44,193.692411 -->
00:47:44,673.692411
Both.
387
00:47:45,243.692411 -->
00:47:45,383.692411
Yeah.
388
00:47:45,933.691411 -->
00:47:51,59.692411
RAG or fine-tuning? Fine-tuning Fine-tuning.
389
00:47:51,949.692411 -->
00:47:59,837.692411
Human in the loop or fully auton- fully autonomous? Uh, both Mm Depending on your risk tolerance.
390
00:48:00,47.692411 -->
00:48:00,827.692411
Yeah, yeah.
391
00:48:01,317.692411 -->
00:48:10,607.692411
What's more dangerous, hallucination or bad data? Uh, well, one cause m- one may cause the other.
392
00:48:10,637.692411 -->
00:48:12,987.692411
So, uh, it's both.
393
00:48:13,387.692411 -->
00:48:13,627.692411
Yeah.
394
00:48:14,417.692411 -->
00:48:22,639.692411
Most overhyped idea in enterprise AI right now? Most overhyped idea.
395
00:48:24,259.692411 -->
00:48:27,129.692411
Um, hmm.
396
00:48:27,899.691411 -->
00:48:28,689.692411
There's a lot of them.
397
00:48:28,949.692411 -->
00:48:29,389.692411
Yeah.
398
00:48:29,459.692411 -->
00:48:32,719.692411
But, uh, I, I, yeah, I don't know.
399
00:48:32,739.692411 -->
00:48:36,189.692411
I would say, uh, interestingly, I think SLMs are.
400
00:48:38,119.691411 -->
00:48:39,379.691411
Interesting, yeah, exactly.
401
00:48:39,699.691411 -->
00:48:41,729.691411
Most underrated capability in enterprise AI.
402
00:48:41,729.691411 -->
00:48:41,749.691411
Underrated capability.
403
00:48:41,749.691411 -->
00:48:41,769.691411
Um, yeah.
404
00:48:41,769.691411 -->
00:48:41,789.691411
I, I...
405
00:48:41,789.691411 -->
00:48:41,829.691411
Uh, there's a lot.
406
00:48:41,829.691411 -->
00:49:02,469.689411
Uh, I, I think, uh, uh, uh, risk, uh, I would say, Underated capability umm yeah I I there's a lot uh I think uh risk uhh I would say uh cyber.
407
00:49:02,879.691411 -->
00:49:13,219.691411
Hmm Uh, is probably- Yeah Or- talked a lot about, right? But, uh, right now I think underrated- Yeah, Absolutely in terms of what it can do.
408
00:49:13,219.691411 -->
00:49:13,609.691411
Yeah.
409
00:49:13,929.690411 -->
00:49:15,39.691411
Given everything that's happening.
410
00:49:15,539.691411 -->
00:49:19,369.689411
Um, so one AI company you think is doing something especially interesting.
411
00:49:19,369.689411 -->
00:49:25,619.690411
You don't have to say Fiddler, but one AI company- doing something especially interesting that you, that you ran into.
412
00:49:25,699.687411 -->
00:49:28,539.694411
You know, I, I, honestly, I don't have one.
413
00:49:28,589.694411 -->
00:49:36,19.689411
There, there is a lot of companies out there that, uh, do a lot of- Hmm Uh, yeah, there's a lot of interesting things, I would say.
414
00:49:36,19.689411 -->
00:49:53,579.687411
I, I think that, uh, it's, it's like the dotcom- Yeah uh, boom, right? And, and, uh, it's, it's gonna be interesting to see how it all evolves, right? Uh, you know, we, we find a lot of value- Hmm In more customization, I would say, uh, you know, at least for our needs.
415
00:49:53,689.692411 -->
00:49:53,979.692411
Hmm.
416
00:49:54,49.693411 -->
00:49:58,709.689411
Um, as opposed to just kind of generic out of the box, uh, you know, stuff.
417
00:49:59,69.690411 -->
00:49:59,309.690411
Hmm.
418
00:49:59,389.689411 -->
00:50:08,49.689411
Um, but you know, that may be is also because, you know, I come from, you know, when I was at Bloomberg, everything was built from scratch, right? Yeah.
419
00:50:08,49.693411 -->
00:50:09,219.691411
So capabilities.
420
00:50:09,619.693411 -->
00:50:13,19.690411
So I like, I like something that fits me best, right? Yeah.
421
00:50:13,29.689411 -->
00:50:17,969.689411
To build or customize something for me as opposed to buying something- Yeah Out of the box, so.
422
00:50:18,429.690411 -->
00:50:18,939.687411
Makes sense.
423
00:50:19,309.687411 -->
00:50:20,209.689411
So final question.
424
00:50:20,369.693411 -->
00:50:22,239.690411
I guess you probably partially answered it before.
425
00:50:22,249.690411 -->
00:50:26,369.688411
In three years, will most enterprise decisions involve an AI agent, yes or no? Yeah.
426
00:50:26,889.689411 -->
00:50:27,379.689411
Yes.
427
00:50:27,519.687411 -->
00:50:27,699.687411
Yeah.
428
00:50:27,909.686411 -->
00:50:28,279.686411
Yes.
429
00:50:28,379.687411 -->
00:50:28,579.687411
Absolutely.
430
00:50:28,689.685411 -->
00:50:29,219.690411
For sure.
431
00:50:29,929.690411 -->
00:50:30,489.690411
Awesome.
432
00:50:30,579.684411 -->
00:50:31,769.687411
Thank you so much, Gary.
433
00:50:31,769.687411 -->
00:50:33,699.682411
This was a very interesting, engaging session.
434
00:50:33,719.683411 -->
00:50:33,959.683411
Yeah.
435
00:50:34,49.685411 -->
00:50:34,939.683411
Uh- Thank you.
436
00:50:35,169.696411 -->
00:50:36,69.696411
Yeah, absolutely.
437
00:50:36,119.696411 -->
00:50:36,319.690411
Thank you so much.
438
00:50:36,319.690411 -->
00:50:36,339.690411
Yeah.
439
00:50:36,359.683411 -->
00:50:38,679.682411
Thanks, everybody, for joining, and, uh, I appreciate it.
440
00:50:38,679.690411 -->
00:50:39,269.694411
Thank- thanks.
441
00:50:39,279.696411 -->
00:50:39,759.696411
Absolutely.
442
00:50:39,769.687411 -->
00:50:40,419.685411
Thank you so much.
443
00:50:40,419.685411 -->
00:50:40,706.3540777
Take care.
444
00:50:40,706.3540777 -->
00:50:40,849.688411
Bye.
445
00:50:40,849.688411 -->
00:50:42,869.687411
Thanks, everyone, for joining our discussion.
446
00:50:42,869.687411 -->
00:50:48,129.694411
So we'll come back with another special guest, uh, you know, another few weeks, and until then, you know, goodbye.
447
00:50:48,349.682411 -->
00:50:48,819.683411
Thank you.
448
00:50:48,819.683411 -->
00:50:49,69.683411
Thanks.
449
00:50:49,69.683411 -->
00:50:49,179.683411
Bye.
450
00:50:49,179.683411 -->
00:50:49,649.684411
Thank you, Gary.
451
00:50:49,839.684411 -->
00:50:50,69.684411
Bye bye.
