Asem Zabin: Welcome back to Transform. name is Asim Zubin, and I'm here with Jessica Tausman to talk ⁓ all things ⁓ data, information, intelligence, specifically in the world. Welcome, Jessica.
Tash Wilson: You've been listening to Transform by GHD. Big ideas, real impact. Welcome to Transform, tomorrow's thinking today. Real conversations on the innovation shaping businesses and our society.
Jessica Talisman: Thank you so much for having me. I'm really excited to be here.
Asem Zabin: There is a comfortable assumption that you can point at AI and AI at your data and it will work the rest out. Today, I want to test the opposite, without structure, AI doesn't reason, ⁓ guesses ⁓ And in our world, confident guessing is rather dangerous. So from all your research, what is the common way organisations get this wrong? when they try to turn data into a decision.
Jessica Talisman: Yeah, so I see this quite a bit where right now we're on the precipice where semantic layers is all the rage an effort to try to inject into that. So in most cases where organisations get this wrong is they think they can take their tables and rows within databases and just immediately manifest meaning from that data without machine and human readable labels, without definitions. So, you know, just taking things as they are within the database and engineering complex SQL queries in order to... try to create meaning not exactly work for AI, for LLMs. So see that collapse happen, but I also see the semantic layers ⁓ as that's coming forward that can help to get closer towards meaning. and manifesting meaning within data. ⁓ still, even with semantic layers, ⁓ don't have relationships between things. And still at a deficit ⁓ of meaning because meaning lives in the relationship between things.
Asem Zabin: Right. think what we're seeing in the infrastructure industry ⁓ we're information. In fact, data information is not necessarily the problem nowadays. I think it used to be probably a problem ⁓ a decades ago, but nowadays what we used to focus on building the asset itself. So everybody gets excited about the new physical asset. So whether it's a bridge or a water treatment plant, everybody gets excited about looking at it. And nowadays, obviously, we get lots of information with it. So when you finish a new building or a bridge or a new structure, you don't only hand over the actual physical asset, you actually hand over a lot of information. You get drawings and just lots of PDFs and lots of of lots of information, which is, great news for everyone. I think the insight we need is always just out of reach when it matters the most. Why that? ⁓
Jessica Talisman: You know, I see also the socio-technical component to this, ⁓ there's a huge gap between how we translate or understand or derive meaning from data. ⁓ So with an infrastructure and when you into physical spaces, that... that can be especially challenging because you can have a huge gap between how marketing and branding and ⁓ those forces perceive of the data. And then you have the engineers and the subject matter experts, the scientists and ⁓ ⁓ of the same things mean can be wildly different. ⁓ And so this gap between levels of but also the purpose, you know, so purpose ⁓ you know, data in the context of someone working in marketing, for example, ⁓ going to be wildly different ⁓ those that are engineers and ⁓ and subject matter experts.
Asem Zabin: Yeah, that's right. That's right. The infrastructure industry over the last, what is it, maybe more than a decade now, we have been doing really much better when it comes to information management. There has been some really, really good stuff, lots of emerging ways of doing things, lots of good ISO standards that have been emerging over the last decade, being updated, and industry starts to get it now. managing our information, the way we procure, collect, store and manage information. We're getting better at it, we're not really there yet. However, in the most simplest, honest way understand between the difference of managing information and managing knowledge in your own way. Jessica, how would you describe what's the difference between both of them, information management and managing knowledge.
Jessica Talisman: Yeah, so information is a maturation of data. So it's like we have labels on things we can identify. They have identifiers. There might be definitions and there might be documents or content attached. to that information, but to make knowledge again, it requires meaning, but it also requires humans and humans in the loop because ultimately humans own knowledge. So knowledge management is not new. It's a very old profession and discipline. I personally come from the library science domain, library and information science and knowledge management more than 3000 years old and is iteratively built through many technological changes. and innovations from like the printing press to our digital system. So knowledge management in and of itself really more about the discipline of organising. And what mean that is ⁓ that there are basic principles for organising, which includes the collection and elicitation of knowledge. As ⁓ you know, because coalescing several different possible artifacts, data points, information, so things that are on further along on that maturity scale, synthesising those things to give meaning.
Asem Zabin: Yeah, beautiful. That makes perfect sense. Again, we see here ⁓ the gap very expensive. get the is much is in the brain still and in lot of places that we go to. there is 40 years of records that are still like You've got 40 years of records for this specific asset and still no idea how anything connects really. There are people who know how these things connect. You will find in the organisation that this person has been working with us for the last 30, 40 years and go and ask them. But they're about to retire. So please catch them before they retire because they've got all of the knowledge about this water treatment plant or this building or this bridge or this road. yeah, in our industry, that's still really the case. And that's where I believe knowledge management is going to be really key for us, ⁓ because I said, it's getting really expensive. the focus, shift now ⁓ is ⁓ to as much as we can with the current assets versus building new. So we're getting a lot of pressure. Asset owners are getting a lot of pressure of doing more with what they have at the moment, but how can they do more if they don't know exactly what they have or they can't access everything? So that's, I find this very, very interesting. And like I said, we don't deal with that, it's gonna get really more expensive.
Jessica Talisman: And of that is and education of leadership. ⁓ I can't cite what the study was, but I've written about how 89, I think it was a McKinsey, it was a McKinsey study that 89 % of leaders don't even understand, fundamentally understand AI, but then also, is knowledge management has been the stepchild within organisations. Most organisations are lucky to have knowledge management at all. And so investment and and ⁓ helping to that muscle within organisations is not easy.
Asem Zabin: Hmm Yeah, agreed. a good segue. What see also a lot of the time, and I really would like your perspective this is... people think, so if we take all of these disconnected, poorly structured data and points a powerful AI model at it, we it's going to solve everything because, hey, AI is here now, so just give it all of these documents, all these systems, and it will solve it. What actually happens, Jessica, and why isn't what comes out the other end reasoning?
Jessica Talisman: Yes, there's several components to that. And one being that with LLMs, there are two layers. So there's the parametric layer, which none of us really have access to. That's the training data. like commercial AI models, they arrive and they already have been trained. there's bias introduced and and that's already sort of preformed. we point these LLMs at our own data and those worlds don't necessarily match. There's also confusion and this is the most basic layer, which is language. We seem to forget that these are large language models and part of what I do is I build knowledge graphs and ontologies and taxonomies, which really deals with defining these things, but making them consumable, machine understandable. So ⁓ it's also known as symbolic AI or good old fashioned AI and often positioned as the antidote to generative AI, which is just probabilistic. it's a pretty complex topic in terms of why it doesn't work. But ⁓ the most part, ⁓ dealing with what a model already knows. And you're injecting your own data that may not be organized or well-defined and crossing your fingers and hoping for the best that automation will happen. But really there's no grounding in the system. There's no grounding in the system that says this is ⁓ compass and ⁓ is ⁓ map, so to speak, like the roadmap ⁓ our data, for our information. And so ⁓ It requires structuring and work and defining and refining, which again goes hand in hand with information and knowledge management. for lack of those ⁓ within organisations, there's this misnomer somehow this automation going to magically happen, that an LLM is just going to magically understand our data. Because that's what's sold and promised. ⁓ I mean, that's of the marketing engine of LLMs is that it'll solve all your problems. ⁓
Asem Zabin: It's here, it's gonna solve the problem. Absolutely. Trust in AI is such a big conversation as well. And it's one of the main reasons why some people are still hesitant. Because it's like a ⁓ box that gives an answer but cannot show it's working, how do knowledge graphs or knowledge-driven systems change that?
Jessica Talisman: Yeah, so I mean, you can't change the black box, but knowledge graphs do give a vector of control in constraining the world and the subject matter upon which an agent or LLMs can act. So it increases reliability and accuracy of ⁓ queries responses from LLMs, especially with workflows. Knowledge graphs can help to decrease token spend can serve as That's the other thing. Knowledge graphs are commonly used as memory ⁓ so that you ⁓ maintain some consistency with your LLMs over but the reduction in cost for tokens I think is one of the most attractive things to most companies. While at the same time, you hear people in the industry complain about how expensive knowledge graphs and ontologies are to build. Like any other data system, yes, building ontologies and knowledge graphs does cost money, but understanding that that cost is an investment that's long-term that actually has the return on investment and knowledge graphs manifest or will present itself as a reduction in cost and tokens, increased then it de-risks the business quite a bit because you're not dealing with hallucinations as much. ⁓ So ⁓ it's sort of weighing the cost benefit.
Asem Zabin: Yes.
Jessica Talisman: rather than just looking at the upfront cost of ⁓ something like a knowledge graph.
Asem Zabin: Yeah, that's right. That's right. That's such a good point. speaking of risks, probably if we're a bit blunt about the downside, really, what are the real risks ⁓ of building ⁓ on ⁓ structured even disconnected data, especially in ⁓ our, again, our world, really, where we, you're a construction company or you're a ⁓ even a designer, a consultant, or owner like energy company or a municipality, what are the consequences that land in the physical world if we build AI ⁓ on ⁓ poorly or disconnected data and ⁓ actually it?
Jessica Talisman: Yeah, mean, the mistakes and the liability can be huge, especially like there's certain things that ontologies can do. They can maintain provenance.
Asem Zabin: Yeah.
Jessica Talisman: When you build ontologies, each thing gets a unique identifier or a URI. So there's stability with this truly unique identifier that helps to reconcile clashing information or data. And so I think ⁓ like a huge ⁓ proposition in that can define and model data in such a way that you can maintain some stability over time. One of the biggest risks not your data ⁓ properly describing your data properly is that the can very wrong. And delivering incorrect information comes at a huge liability with huge risks. One of the other things, and I'm just going to key off of something that you just said, is when you have like an energy company or these different entities that are interoperating or interacting or even part of a knowledge graph, that ontology, RDF ontologies are interoperable. As long as it's an RDF, there's you know, depending on who you talk to, anywhere from six to eight different formats that can be served in be represented in. But at base, it's interoperable. And so you can share that reality across stakeholders. Even outside of a company, can share like a fragment or part of a graph. But ⁓ that... type of knowledge, it's very easy for LLMs to hallucinate, to give incorrect answers, go down or dated answers. I see that a lot where it's old information that somehow, you know, populates and ⁓ ⁓ or within LLM representations. so having that control now within infrastructure spaces, digital are very commonly used ⁓ to to ⁓ manage sort of at scale that propensity towards misinformation so that you have something that's a little lower risk because it's not in production, it's adjacent to, and you can use the digital twin as sort of like a checksum. You're able to check the information from an LLM and you're able to verify information in real time.
Asem Zabin: That's such a good point. I was literally about to ask you about digital twin. It's such a big topic again in our industry. However, lot of the time what you see, the result is an expensive 3D model. And I was literally going to ask you about what does a true digital twin look like. But think you've already answered me, the one where reasoning, casualty, context genuinely kind of baked in. ⁓ That's such a such a way. mean, I don't know if you want to follow up on that.
Jessica Talisman: Well, and you're able to make sure that the information is correct and accurate at that point in time relative to your own system.
Asem Zabin: Hmm.
Jessica Talisman: So it gives you that luxury of sort of touching grass, so to speak, in your data to make sure are things in sync or things represented as they should be. Obviously, there's other ⁓ things play, ⁓ but also allows you to model things into the future, which a lot of people don't really think about. ⁓ It allows you to test things with low risk.
Asem Zabin: Yeah, very nice. I feel there's a lot of good side, obviously there's a lot of reasons why organisations should be doing this. your point of view, what do you think ⁓ actually stopping organisation from doing this well? When say this, mean building knowledge really well.
Jessica Talisman: You know, organisations were presented with LLMs. at that time, ⁓ was seen as a data problem. And a lot of that had to do with because it... scaling laws, is at first when, know, LLMs, was the idea was just more data, more data, the better, the models will improve. And then basically LLMs consumed the internet. And it was learned throughout this time that when LLMs start to consume AI generated content, that LLMs degrade. So that changed the data song, which, you know, paths towards discovery and figuring out, is the antidote to hallucinations and inaccuracies and ⁓ of the problems that are ⁓ rife throughout LLM type applications? knowledge not inherently part of most organisations. And so organisations continued as business as usual, their own organisational structures and didn't change their operating models ⁓ match the ⁓ landscape and demands of LLMs. And I would argue that's still the case. And so I do giants are ⁓ going to down, are going to die, because they're not changing with the industry and with the new research and larger organisations are very bureaucratic organisations. The part again goes back to leaders needing to be educated on this because ultimately it's about funding. So if we continue to fund these old ways, lot of that vendor lock-in. the one thing about Knowledge Graphs is RDF Knowledge Graphs, ontology-based Knowledge Graphs are open, they're free, there's no vendor lock-in. ⁓ you can spin up a Knowledge Graph for low cost, meaning that you don't have to buy a tool necessarily, although I prefer to buy tool when I'm working in a large organisation.
Asem Zabin: Yes.
Jessica Talisman: still this really hard dependency on old ways of doing things and this that you buy the solution. Not that you build and develop the muscle, but you the solution. And so we're just in this old habit of buying the solution. And that's just not panning out for most organisations.
Asem Zabin: So true. Yeah. I think if we want to bring it closer to way that someone can really relate to this. there any example where relationship between things, not the data itself, change the outcome of a decision that you've seen that you can just point to and give us an example of so that if someone is listening can actually be like, ⁓ wow, okay, yeah.
Jessica Talisman: Yeah, I'll give you an example. And it's actually from pre-LLM days, I mean, like the current commercial LLMs, although I've been working with transformer type technology since 2018. So we did use technically LLMs, I guess, or whatever the current version. But at Pluralsight, which is an ed tech company,
Asem Zabin: Okay.
Jessica Talisman: We launched, and this is just an RDF taxonomy. So a lot of people don't realize that you can get started building a knowledge infrastructure starting with SCOT RDF taxonomies. Those are both standards. So you get a lot of the same benefits, not all of the benefits on the ontology. It's the same machine readable form. So we built a really strong taxonomy with some special. classes and we built an auto-tagger and you know this our library at Pluralsight we taught technology and product management and like very technological things. We had more than 14,000 courses and each course had multiple components so it was a lot to track the library was large and obviously of these courses and instructors got paid. We were also partners with AWS as an example for EdTech. it was discovered machine learning was able to leverage that taxonomy to discover that we were overpaying instructors or authors to the tune of $5 billion over five years. And the symptom was you have a course that had been tagged, for example, or identified as being about Microsoft Azure. And another course was identified and tagged as being just about Azure. There was no relationship between those labels. were, you know, it turns out that there were 500 and some courses about Azure, 500 and some courses about Microsoft Azure, but they were talking about the same thing. So in many cases, were getting double paid Azure and Microsoft Azure coexisted on same piece of content within the database. And it happening throughout. we were able, know, just this other team taking what I had built was able to leverage that to save the business $5 billion.
Asem Zabin: Wow.
Jessica Talisman: So, I mean, like it doesn't only have to do with LLMs, it's also payables, like account payables and certificates and licenses and all of those things play into the larger ecosystem. And it has everything to do with the core understanding vocabulary.
Asem Zabin: Yeah, it's as simple as that. Again, if I bring it back to the world of information management at least, there has been a lot of research and continues to be about the amount of time and how much it's expensive just looking for information, just trying to find information. It sounds very simple, but the amount of
Jessica Talisman: Yeah.
Asem Zabin: amount of time and how expensive it is. 20 % or 30 % I can't remember the research of your time is lost on just trying to find the right information rather than doing something that actually adds value.
Jessica Talisman: And that's like an interesting thing that comes from library science that I've been thinking about quite a bit is whole orientation in library science ⁓ for the benefit of the user. Build the benefit of the user, be it human or machine. So you're thinking of that first and foremost. ⁓ And that informs your design and build because you're always thinking of how is a machine going to find this and understand how is a person going to find this and understand what this is. And that means that you're in service to others. when you build and service to others, it, you know, obviously becomes less about you and your team. It becomes more about the architecture and how you can inject meaning into that environment in order to help.
Asem Zabin: Yes.
Jessica Talisman: users find things.
Asem Zabin: That's right. What gives you the most optimism about where this shift is heading? the listeners can take one idea into their work tomorrow morning, what should it be?
Jessica Talisman: I think taking the time, I think most of us are pretty stressed. And like you said earlier, ⁓ you know, we're being asked to do more with less, ⁓ fewer resources. ⁓ so it starts as simply as an audit and figuring out what you have. So it's like can have a role or a stake in this. Coming up with simple, controlled vocabulary. And that's not
Asem Zabin: Yes.
Jessica Talisman: difficult, it's just something that takes time and support from leadership, but just that having an idea of what you have and how you describe things. Disambiguation, think, is one of the biggest opportunities for people right now in organisations, and it is the biggest problem for LLMs. So for example, I like to the and we have this very commonly in the English language, where you can have the same word mean different things. So you can have board, which is like a board, know, like a message board or whatever. You can have board like I'm bored. You can have board like a piece of wood. You know, there's all these different meanings for the same words. so, then organisations just because of the speed of business tend to be, and business culture, tend to accumulate quite a bit of acronyms that we just expect others to understand outside of our own environment.
Asem Zabin: Yes. Yes.
Jessica Talisman: Starting from that, make meaning of those things, of the acronyms. Reconcile your synonyms with preferred labels. Start to create a glossary, anything in any shape or form that can help to build meaning. And that will help you to understand what your estate looks like. you can't model what you haven't defined. So before even starting on a journey of ontology, starting labels and with taxonomy or controlled vocabularies or glossary to start to make sense. And so when I say audit, mean audit your language, audit your vocabulary. That is the surface. That is the interoperability surface. So ⁓ of what I just recently wrote about, ⁓ know, in taxonomies all the way down ⁓ how you find taxonomies throughout LLM architectures, ⁓ risk frameworks, throughout governance frameworks. Like they live taxonomies are throughout our entire digital ecosystem. And I think can get ⁓ ton of improvement with LLM operations just by implementing it.
Asem Zabin: Beautiful. ⁓ think ⁓ a lot of takeaways from this and we can talk about this forever. I think one of things is we owe our system some before we ask them to be clever. Jessica, thank you very much for joining us today. This is ⁓ Transform ⁓ Tomorrow's Thinking today. You can find on YouTube, Apple Podcast, ⁓ wherever you listen.
Jessica Talisman: Thank you so much for having me.
Tash Wilson: You've been listening to Transform by GHD. Big ideas, real impact. Welcome to Transform, tomorrow's thinking today. Real conversations on the innovation shaping businesses and our society.
Jessica Talisman: Thank you so much for having me. I'm really excited to be here.
Asem Zabin: There is a comfortable assumption that you can point at AI and AI at your data and it will work the rest out. Today, I want to test the opposite, without structure, AI doesn't reason, ⁓ guesses ⁓ And in our world, confident guessing is rather dangerous. So from all your research, what is the common way organisations get this wrong? when they try to turn data into a decision.
Jessica Talisman: Yeah, so I see this quite a bit where right now we're on the precipice where semantic layers is all the rage an effort to try to inject into that. So in most cases where organisations get this wrong is they think they can take their tables and rows within databases and just immediately manifest meaning from that data without machine and human readable labels, without definitions. So, you know, just taking things as they are within the database and engineering complex SQL queries in order to... try to create meaning not exactly work for AI, for LLMs. So see that collapse happen, but I also see the semantic layers ⁓ as that's coming forward that can help to get closer towards meaning. and manifesting meaning within data. ⁓ still, even with semantic layers, ⁓ don't have relationships between things. And still at a deficit ⁓ of meaning because meaning lives in the relationship between things.
Asem Zabin: Right. think what we're seeing in the infrastructure industry ⁓ we're information. In fact, data information is not necessarily the problem nowadays. I think it used to be probably a problem ⁓ a decades ago, but nowadays what we used to focus on building the asset itself. So everybody gets excited about the new physical asset. So whether it's a bridge or a water treatment plant, everybody gets excited about looking at it. And nowadays, obviously, we get lots of information with it. So when you finish a new building or a bridge or a new structure, you don't only hand over the actual physical asset, you actually hand over a lot of information. You get drawings and just lots of PDFs and lots of of lots of information, which is, great news for everyone. I think the insight we need is always just out of reach when it matters the most. Why that? ⁓
Jessica Talisman: You know, I see also the socio-technical component to this, ⁓ there's a huge gap between how we translate or understand or derive meaning from data. ⁓ So with an infrastructure and when you into physical spaces, that... that can be especially challenging because you can have a huge gap between how marketing and branding and ⁓ those forces perceive of the data. And then you have the engineers and the subject matter experts, the scientists and ⁓ ⁓ of the same things mean can be wildly different. ⁓ And so this gap between levels of but also the purpose, you know, so purpose ⁓ you know, data in the context of someone working in marketing, for example, ⁓ going to be wildly different ⁓ those that are engineers and ⁓ and subject matter experts.
Asem Zabin: Yeah, that's right. That's right. The infrastructure industry over the last, what is it, maybe more than a decade now, we have been doing really much better when it comes to information management. There has been some really, really good stuff, lots of emerging ways of doing things, lots of good ISO standards that have been emerging over the last decade, being updated, and industry starts to get it now. managing our information, the way we procure, collect, store and manage information. We're getting better at it, we're not really there yet. However, in the most simplest, honest way understand between the difference of managing information and managing knowledge in your own way. Jessica, how would you describe what's the difference between both of them, information management and managing knowledge.
Jessica Talisman: Yeah, so information is a maturation of data. So it's like we have labels on things we can identify. They have identifiers. There might be definitions and there might be documents or content attached. to that information, but to make knowledge again, it requires meaning, but it also requires humans and humans in the loop because ultimately humans own knowledge. So knowledge management is not new. It's a very old profession and discipline. I personally come from the library science domain, library and information science and knowledge management more than 3000 years old and is iteratively built through many technological changes. and innovations from like the printing press to our digital system. So knowledge management in and of itself really more about the discipline of organising. And what mean that is ⁓ that there are basic principles for organising, which includes the collection and elicitation of knowledge. As ⁓ you know, because coalescing several different possible artifacts, data points, information, so things that are on further along on that maturity scale, synthesising those things to give meaning.
Asem Zabin: Yeah, beautiful. That makes perfect sense. Again, we see here ⁓ the gap very expensive. get the is much is in the brain still and in lot of places that we go to. there is 40 years of records that are still like You've got 40 years of records for this specific asset and still no idea how anything connects really. There are people who know how these things connect. You will find in the organisation that this person has been working with us for the last 30, 40 years and go and ask them. But they're about to retire. So please catch them before they retire because they've got all of the knowledge about this water treatment plant or this building or this bridge or this road. yeah, in our industry, that's still really the case. And that's where I believe knowledge management is going to be really key for us, ⁓ because I said, it's getting really expensive. the focus, shift now ⁓ is ⁓ to as much as we can with the current assets versus building new. So we're getting a lot of pressure. Asset owners are getting a lot of pressure of doing more with what they have at the moment, but how can they do more if they don't know exactly what they have or they can't access everything? So that's, I find this very, very interesting. And like I said, we don't deal with that, it's gonna get really more expensive.
Jessica Talisman: And of that is and education of leadership. ⁓ I can't cite what the study was, but I've written about how 89, I think it was a McKinsey, it was a McKinsey study that 89 % of leaders don't even understand, fundamentally understand AI, but then also, is knowledge management has been the stepchild within organisations. Most organisations are lucky to have knowledge management at all. And so investment and and ⁓ helping to that muscle within organisations is not easy.
Asem Zabin: Hmm Yeah, agreed. a good segue. What see also a lot of the time, and I really would like your perspective this is... people think, so if we take all of these disconnected, poorly structured data and points a powerful AI model at it, we it's going to solve everything because, hey, AI is here now, so just give it all of these documents, all these systems, and it will solve it. What actually happens, Jessica, and why isn't what comes out the other end reasoning?
Jessica Talisman: Yes, there's several components to that. And one being that with LLMs, there are two layers. So there's the parametric layer, which none of us really have access to. That's the training data. like commercial AI models, they arrive and they already have been trained. there's bias introduced and and that's already sort of preformed. we point these LLMs at our own data and those worlds don't necessarily match. There's also confusion and this is the most basic layer, which is language. We seem to forget that these are large language models and part of what I do is I build knowledge graphs and ontologies and taxonomies, which really deals with defining these things, but making them consumable, machine understandable. So ⁓ it's also known as symbolic AI or good old fashioned AI and often positioned as the antidote to generative AI, which is just probabilistic. it's a pretty complex topic in terms of why it doesn't work. But ⁓ the most part, ⁓ dealing with what a model already knows. And you're injecting your own data that may not be organized or well-defined and crossing your fingers and hoping for the best that automation will happen. But really there's no grounding in the system. There's no grounding in the system that says this is ⁓ compass and ⁓ is ⁓ map, so to speak, like the roadmap ⁓ our data, for our information. And so ⁓ It requires structuring and work and defining and refining, which again goes hand in hand with information and knowledge management. for lack of those ⁓ within organisations, there's this misnomer somehow this automation going to magically happen, that an LLM is just going to magically understand our data. Because that's what's sold and promised. ⁓ I mean, that's of the marketing engine of LLMs is that it'll solve all your problems. ⁓
Asem Zabin: It's here, it's gonna solve the problem. Absolutely. Trust in AI is such a big conversation as well. And it's one of the main reasons why some people are still hesitant. Because it's like a ⁓ box that gives an answer but cannot show it's working, how do knowledge graphs or knowledge-driven systems change that?
Jessica Talisman: Yeah, so I mean, you can't change the black box, but knowledge graphs do give a vector of control in constraining the world and the subject matter upon which an agent or LLMs can act. So it increases reliability and accuracy of ⁓ queries responses from LLMs, especially with workflows. Knowledge graphs can help to decrease token spend can serve as That's the other thing. Knowledge graphs are commonly used as memory ⁓ so that you ⁓ maintain some consistency with your LLMs over but the reduction in cost for tokens I think is one of the most attractive things to most companies. While at the same time, you hear people in the industry complain about how expensive knowledge graphs and ontologies are to build. Like any other data system, yes, building ontologies and knowledge graphs does cost money, but understanding that that cost is an investment that's long-term that actually has the return on investment and knowledge graphs manifest or will present itself as a reduction in cost and tokens, increased then it de-risks the business quite a bit because you're not dealing with hallucinations as much. ⁓ So ⁓ it's sort of weighing the cost benefit.
Asem Zabin: Yes.
Jessica Talisman: rather than just looking at the upfront cost of ⁓ something like a knowledge graph.
Asem Zabin: Yeah, that's right. That's right. That's such a good point. speaking of risks, probably if we're a bit blunt about the downside, really, what are the real risks ⁓ of building ⁓ on ⁓ structured even disconnected data, especially in ⁓ our, again, our world, really, where we, you're a construction company or you're a ⁓ even a designer, a consultant, or owner like energy company or a municipality, what are the consequences that land in the physical world if we build AI ⁓ on ⁓ poorly or disconnected data and ⁓ actually it?
Jessica Talisman: Yeah, mean, the mistakes and the liability can be huge, especially like there's certain things that ontologies can do. They can maintain provenance.
Asem Zabin: Yeah.
Jessica Talisman: When you build ontologies, each thing gets a unique identifier or a URI. So there's stability with this truly unique identifier that helps to reconcile clashing information or data. And so I think ⁓ like a huge ⁓ proposition in that can define and model data in such a way that you can maintain some stability over time. One of the biggest risks not your data ⁓ properly describing your data properly is that the can very wrong. And delivering incorrect information comes at a huge liability with huge risks. One of the other things, and I'm just going to key off of something that you just said, is when you have like an energy company or these different entities that are interoperating or interacting or even part of a knowledge graph, that ontology, RDF ontologies are interoperable. As long as it's an RDF, there's you know, depending on who you talk to, anywhere from six to eight different formats that can be served in be represented in. But at base, it's interoperable. And so you can share that reality across stakeholders. Even outside of a company, can share like a fragment or part of a graph. But ⁓ that... type of knowledge, it's very easy for LLMs to hallucinate, to give incorrect answers, go down or dated answers. I see that a lot where it's old information that somehow, you know, populates and ⁓ ⁓ or within LLM representations. so having that control now within infrastructure spaces, digital are very commonly used ⁓ to to ⁓ manage sort of at scale that propensity towards misinformation so that you have something that's a little lower risk because it's not in production, it's adjacent to, and you can use the digital twin as sort of like a checksum. You're able to check the information from an LLM and you're able to verify information in real time.
Asem Zabin: That's such a good point. I was literally about to ask you about digital twin. It's such a big topic again in our industry. However, lot of the time what you see, the result is an expensive 3D model. And I was literally going to ask you about what does a true digital twin look like. But think you've already answered me, the one where reasoning, casualty, context genuinely kind of baked in. ⁓ That's such a such a way. mean, I don't know if you want to follow up on that.
Jessica Talisman: Well, and you're able to make sure that the information is correct and accurate at that point in time relative to your own system.
Asem Zabin: Hmm.
Jessica Talisman: So it gives you that luxury of sort of touching grass, so to speak, in your data to make sure are things in sync or things represented as they should be. Obviously, there's other ⁓ things play, ⁓ but also allows you to model things into the future, which a lot of people don't really think about. ⁓ It allows you to test things with low risk.
Asem Zabin: Yeah, very nice. I feel there's a lot of good side, obviously there's a lot of reasons why organisations should be doing this. your point of view, what do you think ⁓ actually stopping organisation from doing this well? When say this, mean building knowledge really well.
Jessica Talisman: You know, organisations were presented with LLMs. at that time, ⁓ was seen as a data problem. And a lot of that had to do with because it... scaling laws, is at first when, know, LLMs, was the idea was just more data, more data, the better, the models will improve. And then basically LLMs consumed the internet. And it was learned throughout this time that when LLMs start to consume AI generated content, that LLMs degrade. So that changed the data song, which, you know, paths towards discovery and figuring out, is the antidote to hallucinations and inaccuracies and ⁓ of the problems that are ⁓ rife throughout LLM type applications? knowledge not inherently part of most organisations. And so organisations continued as business as usual, their own organisational structures and didn't change their operating models ⁓ match the ⁓ landscape and demands of LLMs. And I would argue that's still the case. And so I do giants are ⁓ going to down, are going to die, because they're not changing with the industry and with the new research and larger organisations are very bureaucratic organisations. The part again goes back to leaders needing to be educated on this because ultimately it's about funding. So if we continue to fund these old ways, lot of that vendor lock-in. the one thing about Knowledge Graphs is RDF Knowledge Graphs, ontology-based Knowledge Graphs are open, they're free, there's no vendor lock-in. ⁓ you can spin up a Knowledge Graph for low cost, meaning that you don't have to buy a tool necessarily, although I prefer to buy tool when I'm working in a large organisation.
Asem Zabin: Yes.
Jessica Talisman: still this really hard dependency on old ways of doing things and this that you buy the solution. Not that you build and develop the muscle, but you the solution. And so we're just in this old habit of buying the solution. And that's just not panning out for most organisations.
Asem Zabin: So true. Yeah. I think if we want to bring it closer to way that someone can really relate to this. there any example where relationship between things, not the data itself, change the outcome of a decision that you've seen that you can just point to and give us an example of so that if someone is listening can actually be like, ⁓ wow, okay, yeah.
Jessica Talisman: Yeah, I'll give you an example. And it's actually from pre-LLM days, I mean, like the current commercial LLMs, although I've been working with transformer type technology since 2018. So we did use technically LLMs, I guess, or whatever the current version. But at Pluralsight, which is an ed tech company,
Asem Zabin: Okay.
Jessica Talisman: We launched, and this is just an RDF taxonomy. So a lot of people don't realize that you can get started building a knowledge infrastructure starting with SCOT RDF taxonomies. Those are both standards. So you get a lot of the same benefits, not all of the benefits on the ontology. It's the same machine readable form. So we built a really strong taxonomy with some special. classes and we built an auto-tagger and you know this our library at Pluralsight we taught technology and product management and like very technological things. We had more than 14,000 courses and each course had multiple components so it was a lot to track the library was large and obviously of these courses and instructors got paid. We were also partners with AWS as an example for EdTech. it was discovered machine learning was able to leverage that taxonomy to discover that we were overpaying instructors or authors to the tune of $5 billion over five years. And the symptom was you have a course that had been tagged, for example, or identified as being about Microsoft Azure. And another course was identified and tagged as being just about Azure. There was no relationship between those labels. were, you know, it turns out that there were 500 and some courses about Azure, 500 and some courses about Microsoft Azure, but they were talking about the same thing. So in many cases, were getting double paid Azure and Microsoft Azure coexisted on same piece of content within the database. And it happening throughout. we were able, know, just this other team taking what I had built was able to leverage that to save the business $5 billion.
Asem Zabin: Wow.
Jessica Talisman: So, I mean, like it doesn't only have to do with LLMs, it's also payables, like account payables and certificates and licenses and all of those things play into the larger ecosystem. And it has everything to do with the core understanding vocabulary.
Asem Zabin: Yeah, it's as simple as that. Again, if I bring it back to the world of information management at least, there has been a lot of research and continues to be about the amount of time and how much it's expensive just looking for information, just trying to find information. It sounds very simple, but the amount of
Jessica Talisman: Yeah.
Asem Zabin: amount of time and how expensive it is. 20 % or 30 % I can't remember the research of your time is lost on just trying to find the right information rather than doing something that actually adds value.
Jessica Talisman: And that's like an interesting thing that comes from library science that I've been thinking about quite a bit is whole orientation in library science ⁓ for the benefit of the user. Build the benefit of the user, be it human or machine. So you're thinking of that first and foremost. ⁓ And that informs your design and build because you're always thinking of how is a machine going to find this and understand how is a person going to find this and understand what this is. And that means that you're in service to others. when you build and service to others, it, you know, obviously becomes less about you and your team. It becomes more about the architecture and how you can inject meaning into that environment in order to help.
Asem Zabin: Yes.
Jessica Talisman: users find things.
Asem Zabin: That's right. What gives you the most optimism about where this shift is heading? the listeners can take one idea into their work tomorrow morning, what should it be?
Jessica Talisman: I think taking the time, I think most of us are pretty stressed. And like you said earlier, ⁓ you know, we're being asked to do more with less, ⁓ fewer resources. ⁓ so it starts as simply as an audit and figuring out what you have. So it's like can have a role or a stake in this. Coming up with simple, controlled vocabulary. And that's not
Asem Zabin: Yes.
Jessica Talisman: difficult, it's just something that takes time and support from leadership, but just that having an idea of what you have and how you describe things. Disambiguation, think, is one of the biggest opportunities for people right now in organisations, and it is the biggest problem for LLMs. So for example, I like to the and we have this very commonly in the English language, where you can have the same word mean different things. So you can have board, which is like a board, know, like a message board or whatever. You can have board like I'm bored. You can have board like a piece of wood. You know, there's all these different meanings for the same words. so, then organisations just because of the speed of business tend to be, and business culture, tend to accumulate quite a bit of acronyms that we just expect others to understand outside of our own environment.
Asem Zabin: Yes. Yes.
Jessica Talisman: Starting from that, make meaning of those things, of the acronyms. Reconcile your synonyms with preferred labels. Start to create a glossary, anything in any shape or form that can help to build meaning. And that will help you to understand what your estate looks like. you can't model what you haven't defined. So before even starting on a journey of ontology, starting labels and with taxonomy or controlled vocabularies or glossary to start to make sense. And so when I say audit, mean audit your language, audit your vocabulary. That is the surface. That is the interoperability surface. So ⁓ of what I just recently wrote about, ⁓ know, in taxonomies all the way down ⁓ how you find taxonomies throughout LLM architectures, ⁓ risk frameworks, throughout governance frameworks. Like they live taxonomies are throughout our entire digital ecosystem. And I think can get ⁓ ton of improvement with LLM operations just by implementing it.
Asem Zabin: Beautiful. ⁓ think ⁓ a lot of takeaways from this and we can talk about this forever. I think one of things is we owe our system some before we ask them to be clever. Jessica, thank you very much for joining us today. This is ⁓ Transform ⁓ Tomorrow's Thinking today. You can find on YouTube, Apple Podcast, ⁓ wherever you listen.
Jessica Talisman: Thank you so much for having me.
