AI safety needs more than good intentions의 전체 녹취록

discourse. Having said that, I think there are certain realities on the ground. The first reality on the ground is unless you've built frontier models and unless you've built these highly capable systems, I think it's very difficult for a bystander to know the capabilities of these advanced systems. And so this is also a moment in time to listen to what the leading researchers are seeing at the frontier. And I think that brings me to my second point. The moment in the past four days has been really focused on the essay that Dario Amodei has drafted around pacing the frontier. And it's really important to understand there's a very intentional choice behind the use of word pacing and not slowing down. So pacing really, and as I think about it, I wish he had written the paper with the title Governing the Frontier because that's what the entire conversation is about. And the reason this becomes a very pivotal moment is because there's a couple of things that have happened in the past three to four months, which I think the industry and the folks who are not in the AI community understand is one that we've reached a point where AI has become so capable in these frontier labs that it is increasingly helping researchers build AI. And that's basically called recursive self-improvement. And it is starting to appear in very practical forms so much so that around 25% of the code at Anthropic is right now being written by AI. And so I think when you start to think about the dynamic when the capability has accelerated to such a point, the question that begs is, should we be using compute resources for recursive self-improvement or here do we have an opportunity to redirect those compute resources to other useful functional things because AI is already very, very capable, right? So that's one key aspect of what's underlying this essay. The second key aspect of what's underlying this essay is the safety incident, which basically is an eye-opener for many, is the incident where OpenAI agents, not single agent, but a swarm of agents basically exploited Hugging Face. And the reason that is very, very monumental is because of not the exploit that ended up happening, but it was a sequencing of that exploits. First, the agents weren't given internet access and they discovered a way to obtain it. Then they weren't given any sort of interagent communication. And guess what? They figured out how to use an internal package manager to create a message board. And then they figured out a way to jointly figure out a communication strategy to exploit other agents. They were able to then hack into Hugging Face credentials. They were able to exploit other unknown vulnerabilities. And I think what was really fascinating for me is as I've been tracking, especially there was a retro done at Black Hat, which was very meaningful, is that as these swarms of agents started to coordinate and communicate, they always found a way to get to their objective. They ended up not only hacking

into the monitoring infrastructure, the research infrastructure, but also the evaluation infrastructure. And the reason this becomes pivotal is we really need to separate the capability from intent. And this brings in governance into it. Right? It's okay. It's these systems now are demonstrating methodologies where they are able to literally like humans. Humans, we have this beautiful, amazing power that you and I can communicate. We can align on ideas. We can rally the troops. We can use our stories to really transform the world. And I think that fundamental incident is a bigger, bigger story. And the Hugging Face incident matters because it's really turning the AI control from just a theoretical alignment debate into truly a governance problem. So just to summarize, I think I truly do wish the essay was focused on governing the frontier because that is what's happening in this moment. And then again, there's different ways to govern the frontier and govern the enterprise. And I really hope that we can align on ideas, listen and figure out the pathways to really utilize this technology in our favor and in our benefit. Jason Hiner: Yeah, thank you. You know, it's such a great point that the power of language in these things, especially in such a charged, kind of polarized moment, we find ways to carve out territory and then sort of defend that position to the death. And in dialogue, what we need is we need sort of an opening for everyone to find a way to participate. Everybody has their ideas, the things that they want to see out of something. In this case, the US, the government wants to see the ecosystem, the AI ecosystem in the US, to continue to advance and be at the cutting edge and be at the frontier. And when they hear words like pacing or slowdown to your point, I think they then interpret that as something that is going to inhibit their goals of keeping the US ecosystem in a very strong place. And there is the fear, and it has a reasonable fear that if you do regulate, in some cases, too early, the argument for that is that if you regulate too early, you can stymie innovation. But it isn't a zero-sum game. There is also a case to be made, and sometimes that's not being put out there as clearly right now, is that when governance and regulation works well, it creates sort of a set of boundaries that make it possible to go really fast, just in the same way as if you had a drag racer that had a fence around it. Well, it doesn't have to worry because it knows that it has this fence keeping it in a safe place so it can go really, really fast. That's one of our biggest challenges. I guess I would love to hear from you how the quality of dialogue has advanced in the past week as we've seen some digging in of the positions of do we need to pace the frontier or regulate versus no, we need to be careful not to put guard rails around this because then the US, the fear is that China is moving very quickly and has taken a very different position, especially around open source. They have an ecosystem where there's some sharing in between those labs and they're moving very rapidly. There's this fear from that standpoint that if the US slows down or stops to get its house in order, that it will get lapped in the race as it were. I would love to hear if you are hearing because I know you're also quite connected with the parties who are involved in these conversations. If you're hearing that there is still a conversation that's continuing about how do we set safeguards whilst continuing to advance. I'm hoping that you can give us some hope that the dialogue hasn't completely stopped. Navrina Singh: Absolutely. I think this is the moment that there's a lot of healthy conversation. I want to unpack something for the audience because I think there's a lot of again misinformation. Governance is not regulation. First and foremost, I want to start with separating that out. When you think about the spectrum of what does the spectrum of trust look like, on that spectrum of trust there's different ways to inject trust by verifying whether these systems are working, whether these institutions are working, whether the people behind these institutions are being accountable. That spectrum basically includes a couple of mechanisms and a couple of instruments. For example, one part of the spectrum is your own self-testing and attestation. As a product company, every organization that is building products has the responsibility to do their own testing and attestation and making sure that the products that are being put out in the public actually meet the requirements of the public. They are safe for public. They are following consumer protection laws. They have liability associated with it. So whoever frontier lab you are, you have to make sure that you cannot be putting products out in the market that are not safe because that's your first component off on that spectrum of trust, your responsibility around self-testing and attestation. As you progress, there's a second component. It's very similar to what we've seen in financial services, if you will. There's the second line of defense where you actually do independent, still internal testing where you have a completely separate group that is going to be doing red teaming. That is testing whether you're meeting the requirements not only from the use case perspective. You've sort of created the right evidence methodology. You have documented the findings. You're transparently really figuring out, as an independent, within the organization, are you doing the things and are you having the right testing procedures? Within AI, there's a lot of focus right now on red teaming, purple teaming, etc. mechanisms. Then you progress a little bit further, which is now you don't have to be part of the company, but can you be an outsider? That's the notion of what's called being called to right now in the industry, the embedded evaluators. That you can be a very separate

testing organization and you're brought in into a frontier lab to actually test these systems. But the difference here is it's not just testing the technical systems. It is also testing against the commitments that the organization has made. If I, as a frontier lab, am going to say that I'm going to be reporting on system cards, this outside entity is now testing my systems, but also saying, because did I publish that system card or not? It's a combination of, I would say, audit and testing, where you're looking organizationally what you've committed to and then you're looking from the technical perspective how are your systems performing. That's the third component of the spectrum of trust. Then you can extend that a little bit more further, where you bring in independent auditors to test the methodologies that are being published by these embedded evaluators. That's why the entire audit ecosystem was created for financial services. And then you stretch the spectrum a little bit further and then you might have regulations. The good news there, Jason, is we already have a lot of regulations that still apply to AI. I think many folks forget that things like, do you have discriminatory practices? Are you consumer protection laws? Are you abiding by those? Those exist and every company that is building AI systems has to adhere to those existing laws. And then, yes, there is frontier regulation that we all need to think about, especially on international coordination, because with AI comes new kinds of risks, emergent threats, there's new kinds of compliance requirements that show up, which might result into new form of regulation. The reason I wanted to unpack the spectrum of trust for you, Jason, which goes from self attestation, that is your company, your frontier labs responsibility, all the way to regulation is because governance many a times is like, oh, that means you're going towards regulation. And when you think about this spectrum, Jason, what becomes really interesting is frontier labs, I truly do believe that as these companies think about going public, they are going to be held accountable to public institutions. It is really critical that they do their own testing and evaluation and robustify their governance. And I'm going to be holding anthropic and open AI accountable to that. But at the same time, they're doing that. So that's why we need these other mechanisms of embedded evaluators. We need a very vibrant independent verification organization. We need maybe new regulations, but there are also existing regulations. So I hope, Jason, that gives you a sense of the spectrum. It's not one thing. And that's why, as I mentioned at the onset of this conversation, that right now, ideas matter because we have to tackle testing and evaluation, verification of evidence against all these different mechanisms to inject as much trust as we can into these systems and where these systems are actually going to be making consequential decisions. Jason Hiner: So one of the things I heard you say is that to start, we don't necessarily need new laws or regulations to start on the governance on the path to governing these things well. Is that fair to say? Navrina Singh: I would say that there are existing laws and regulations that already apply. And yes, we should be looking at new regulations as well. But I think there's a somehow there's this among certain folks, they think that their AI products are not right now governed by certain regulation. And I think that distinction is important to make. Jason Hiner: Okay. What do you think about this idea of regulatory capture? That's one of the things that's come up over this past week a lot, which is that for the audience, regulatory capture is this concern that the big labs, Google, OpenAI, Anthropic, they've offered to help the government understand they've been proposed frameworks. So the most famous of which was the one by Google DeepMind and Demis Hassabis offering some ideas, some frameworks. And you've seen it from you've seen some ideas offered by OpenAI Anthropic as well. But with regulatory capture, the concern is that these companies are going to shape the form of regulation in a way that will protect their market share, will protect their position, and will make it harder for competitors, including some of the, you know, the dialogue has has raised, including some open source competitors who may not have the resources to also have people on staff to, to work as many people, right, to, to work with regulators to work alongside maybe independent observers being part of the organization. And so we'd love to get your perspective on, on regulatory capture, how much of a concern it should be, and, and how much of that is coming up as part of the dialogue. Navrina Singh: So again, very nuanced conversation. And if you see me not taking sides is because it is nuanced conversation. Having said that, I worry about regulatory capture a lot, and it's a real thing. So I do worry if there is, you know, the power is consolidated in the hands of a few labs, that is a true worry. And I think this is one of the reasons I'm a big proponent of open source. And I truly believe that there's, you know, as, as United States, and also our allies investing in open source is really critical fundamental step. So regulatory capture can prevent even young companies, you know, in startups trying to show up in this ecosystem, prevent them to be competitive with some of the labs that then end up owning a big part of what the, what that ecosystem looks like. I think the flip side that again, I might offer to the audiences, when you are so far ahead in terms of technological development. So imagine you're climbing Mount Everest, and you're the first one to climb Everest. Now you've reached the summit, you've seen things on the journey to summit that others haven't, because they still haven't climbed the summit. So you can actually start to communicate information

back to them, giving them pathways to how they can get to the summit. But as you have to remember, Jason, that again, humans are very interesting, I would say creatures, because you have to weigh not only their expertise, but also their incentives. So if I'm now at the top of this Mount Everest, and I have created a new kind of shoes that I want to sell more. Now the information that I can relay to the people who are climbing the summit is buy these shoes, because that's the only way you can reach the summit, because I have already reached it. And because I'm the expert, but again, I have the incentive to sell more shoes, right? And so as you can imagine, there is a little bit balancing that you have to do from expertise perspective, but also from the perspective of incentive alignment. And I think right now, that's where the friction is happening. I truly do believe that the AI labs and the frontier labs and the researchers there have seen things and have built things. We should listen to those experts because they've reached the summit. At the same time, yes, do they all have incentives? We all have incentives. Everyone is arriving at it from different places. So this truly is an alignment challenge and an alignment problem that is being discussed in this discourse is regulatory captured real thing. Because if it's the power is consolidated in the hands of a few frontier labs, what ends up happening, there's this other posture that is happening, especially in the enterprises that we serve, we generally serve Fortune 500s in regulated and unregulated sectors like healthcare, insurance, pharma, CPG retail. And so all the enterprises right now are trying to utilize foundation models, they're trying to utilize frontier capabilities. But at the same time, they're asking themselves this question, should I be renting intelligence, or should I be owning my own intelligence? Because I don't want to give away all my power, right? And I think so as you can imagine, there's a lot of incentives that are at clash, along with a lot of great expertise. So this is the moment in time that listening becomes really critical. And then so the three things I would like leave you with is yes, regulatory capture is a massive concern. I think it's really important that we invest in open source, but at the same time, the frontier labs have the expertise that we should be listening to, and then at least paying attention to and then making informed decision coming together to really figure out how do we progress our enterprises, our communities, and our country forward collectively on this transformational technology. Jason Hiner: Alright, I want to talk a little bit about open source, and then I want to ask a little bit about Credo AI, because there's some stuff that you're doing specifically that I think is interesting, especially, you know, sort of forward deployed engineers or consultants, you know, it's something you've been doing for a while before it became one of

the hot terms of 2026. Let's talk about open source for a second, though, because you have some background there, you are an advocate for us for it. One of the things that we're seeing is that there's been obviously this explosion of AI agents, there's been a lot of concern about token costs. That's driven some interest in on prem AI in open source, on device. At the same time, more regulated industries are getting interested in using AI, and they have some really strict data sovereignty and data governance, governance concerns. And so for them, taking, you know, an open source model, customizing it, running it locally, they have a lot more trust in that than they do sending it to the cloud. And it also saves them a lot of money. It has a lot of value. And then along those same lines, you can take it and make your own domain specific model based on some of those open source models. And that's what we've seen from folks like Thompson Reuters, CrowdStrike, Salesforce, they have all taken these, these large models, and then they have made them domain specific, you know, to cybersecurity, to, you know, business and business processes, and to legal all of those kinds of things in the three cases that I mentioned. So open source has an enormous amount of momentum behind it. And yet there is also the concern that still a lot of it is the Chinese labs are very far ahead, you know, in open source. They are rapidly catching up to US models and faster and faster. But we are seeing some more broad based support for open source models. So from NVIDIA, from Mistral, a lot of people don't know that Google and OpenAI also have open source models that are out there and are being used. So what are you seeing in terms of the open source ecosystem diversifying and, you know, growing in some energy? And then what are you seeing in terms of enterprises adopting them doing just as you're saying, like deciding that they want to own their own intelligence? Navrina Singh: Yeah, what a great question, Jason. And I think truly the industry, especially the enterprises we work with, are at a very interesting point of trade offs. And they're trying to figure out how do they trade off between capability, cost and control. And by the way, you can't pick all three, you have to pick two, right? So in some cases, it's like, I want to move as fast as possible, I deeply care about highly capable systems, and I deeply care about cost, I'm willing to give up on control, you make a certain choice. Whereas in some cases, you're like, nope, I want highly controlled systems, because I'm in a regulated sector. And I care about cost, maybe capability, I don't need the latest and greatest Astra model, I'm happy with the older versions of GPT. So as you can imagine, Jason, we are at this really pivotal point of trade offs. And we are seeing live in front of us, many of our customers making these choices. Having said that, I think what I find fascinating, and this is where, you know, Credo AI's work becomes really consequential, is irrespective of open source, open weight, proprietary frontier models, governance has to be able to solve for all those choices. Because at the end of the day, when you as an enterprise have decided that I'm going to lean in on the frontier models, I'm going to bring those in. And then I'm going to deploy that for a use case I care about, you still have to be able to unpack your entire bill of materials and figure out the risk, you have to also figure out the opportunity and the compliance across that entire bill of materials. And similarly, if you choose to use maybe let's say an open weight model from thinking machines, and you're like, okay, I want to own my own intelligence. But at the same time, what that affords you is the ability to sort of not only own your own intelligence, but you have to now embed a lot of control, which you otherwise would have got from the frontier labs. So I think, again, Jason, we are in this very interesting land of trade off. My bigger concern right now, and things that we are focused on is similar to in product when you have like a tech debt or design debt. What's happening right now is the governance debt is increasing quite drastically. And the reason that governance debt is increasing is your frontier capabilities are accelerating. And we're still at that inflection point where we are going to see the rate of change super exponential, the governance investment in governance, and the investment in governance capabilities haven't really kept pace with the frontier capabilities. And as a result of which you're seeing this growing gap, this governance debt. And I think that's the debt that I want to start focusing on reducing. And that's why Credo AI exists. Jason Hiner: Hey, everybody, thanks for listening to this episode. Quick note, and then we'll get you back to the conversation. We love bringing you this content every week. We're always trying to figure out how we can deliver you the most value to help you understand how AI is transforming business and transforming the world. If you're enjoying the show, there's an easy way for you to give a little value back and help others learn about the show as well. If you're on Apple podcasts or Spotify, drop us a rating and leave us a review. And if you're on YouTube, hit like, subscribe, or leave us a comment. That's it. It only takes a minute, but it's a huge help. So thanks in advance for pitching in. And now back to the show. I'm glad you brought up Credo AI at the end because I'd like to talk about that next because when we talked last month, one of the things that I was really impressed with is that you've been working on this for a while. You founded the company on this idea. And so I'd love for you to unpack that with the audience. When did you start? Why? And what's been the journey to helping companies really get their arms around AI in some really clear ways so that they have a very clear path forward in their

journey in AI that's especially important in industries where there are very important regulations and guidelines already in place for the ways they manage data and that. So you have been helping these kinds of companies for several years now. So talk a little bit about that journey. Navrina Singh: Yeah. Very interesting journey, Jason. As I mentioned prior to Credo AI, I was at Microsoft building some really exciting AI products. And I think the interesting thing is when you're always at the front end of technological evolution, you start to see what's coming. And even I didn't anticipate how fast it would come at us. But back in 2019, a thesis that I had was if the advancements that I can see right now in AI, even if they just linearly increase, not even exponentially increase, there is going to come a point where the AI builders will need to make sure that they are building safe, compliant, trustworthy AI systems. And at the same time, the AI governors who are coming from risk and compliance will get access to AI technology that they can actually start becoming builders. And when that happens, the gap between AI builders and AI governors will collapse. And to make that really interesting way to build safe and trusted technology, I call it the last mile problem in the AI frontier, which was can we actually embed governance throughout the development lifecycle. And back in 2019, everyone, the minute you use the word governance, it's like rolling your eyes because it's like, oh, my God, you're going to slow my innovation down. Jason Hiner: Yes, right. Navrina Singh: Yeah. And it's very interesting. And I very methodically chose the word AI governance for the category we were creating, even though I knew that the PR and branding would not work for us, because the minute even before we've entered the room, you've made an assumption that governance is going to slow me down. I think fast forward to now, governance is the growth driver for AI. We actually are seeing that the enterprises we serve who are either buying a lot of third party foundation model, you know, models, or they're buying a lot of third party applications, or they're building their own AI, the companies that are actually have been working with us and have invested in governance, they're seeing this interesting growth in actually leveraging AI to get the ROI, the top line growth, as well as bottom line, they are able to develop products faster. They're seeing increased productivity within their enterprises. They're able to because of transparency, retain customers longer, as well as acquire customers faster. So you're seeing this very interesting, I would say, opportunity that governance is unlocking for organizations who are betting on having that foundation of risk management compliance and oversight, that they're able to actually win faster with AI, both internally with their employees, but also with their customers and other stakeholders. Jason Hiner: Very good. What do you feel like the path looks forward looks like, you know, going forward for you all? One of the things you mentioned to me is that this idea of forward deployed, you know, employees remind me what you call it from your organization. Navrina Singh: Yeah, absolutely. You know, I do want to give huge credit to Palantir and Alex, and I think they really were foundational in creating what's called forward deployed engineers. And the idea behind forward deployed engineers was, can I actually embed my engineers in these very complex enterprise environments and basically learn enough about them so that I can make better products. The reason Credo AI became the first ones to actually develop a similar model, but for governance, we call it forward deployed governance. So our forward deployed governance has two parts, forward deployed governance advisory and forward deployed governance engineering. And it's built on a similar idea like Palantir, but the forward deployed governance team is really focused on really understanding the AI maturity and the AI governance posture of an enterprise. And so my forward deployed team will go into an enterprise and very quickly, we'll be able to figure out are they at level one? That means they don't have policies, they don't even have people, they don't even know how to do governance all the way to are they at level five plus, where they have thousands and 10,000s of agents running autonomously and they need oversight and governance to be completely fully autonomous. So going from level one all the way to level five, the forward deployed governance team is really focused on understanding one where you are on that level and then secondly help you either with people, process or with our product to really bring that governance oversight to your organization. And as you can imagine, Jason, you know, this is something I deeply think about is as AI capabilities are advancing, it is actually the human, our ways of working and our processes, that's not keeping up. So many of the organizations that we work with are really trying to figure out how do I make my workforce more literate? How do I make sure that the token per worker is actually an indication of, you know, my employees are actually being AI forward? How do I make sure that I have the right guardrails in place? Because for public companies, as you can imagine, there is a lot at stake if you get AI wrong and you're putting products out there that are making consequential healthcare, financial services, employment decisions, and again going back to there are a lot of laws that you have to adhere to. So our forward deployed governance team is really focused on helping you sort of transform and grow your business with artificial intelligence by starting with governance as a growth driver. Jason Hiner: Very good. How many of those do you have that are deployed out there? Navrina Singh: We are a small team of about 90. More than I would say 12 percent of my team is forward

deployed governance. So again, we are a series A startup and with a mighty team of 90 people, we are changing the world. One forward deployed governance expert at a time. Jason Hiner: Very good. All right. How about in terms of the state of the dialogue? You know, you are involved in many of these things. As I remember, you know, you take trips to Washington. You are part of a lot of the conversations because of the role your small but mighty, you know, series A startup has already doing with many of these agencies, organizations of all sizes. What do you feel like the path forward looks like, you know, right now? Is there reason to be concerned? Is there reasons to be hopeful? Where do you feel like the state of, you know, the conversation is? Because obviously, it has now become this sort of national, international news and it is something that is unlikely to be going away anytime soon. Where do you feel like it's going to net out and what does it look like? What's the path? Navrina Singh: Yeah. So, you know, it's Jason, you bring up a really good point. I think since the founding of Credo AI, we've played a very interesting role in the ecosystem because the thesis that I had was for governance to keep pace with, you know, AI capabilities, which by the way, it's not because of the lack of investment in governance and we can talk about that in just a bit. I think our role was really not just around building the great products, but it was also being sort of an orchestrator in this ecosystem. So, to your point, we have obviously great inroads into enterprises. We work a lot with policymakers and regulators and standard-setting bodies all the way from NIST all the way to ISO. We also work with Frontier Labs and our constant conversation with them. So, our role as an orchestrator is really to enable how do we bring governance standards into products that can scale enterprise transformation within these enterprises. And the reason I started with that is where we are headed and you're asking a founder who always operates in a constant state of optimism and paranoia because that's the role of a founder, right? Jason Hiner: Of course. Of course. Navrina Singh: For me, yes, there is a paranoia around, wow, these AI capabilities are advancing so fast. As an engineer, I am seeing firsthand that when you have fully autonomous agents doing maybe a huge percentage of my work on a daily basis and we are looking at agents per worker within Credo AI as well, you are seeing that pace of change. It is getting to that exponential point. So, I think the paranoia really stems from, oh my God, is that governance debt, are we going to be able to close it? And I don't know right now how we can without the right investment, without the right focus across the spectrum of trust that I mentioned. But then, as a founder, I also think about optimistically, what a great opportunity for us to, one, actually with the right investment, build better governance products. And I

think this is my day in, day out focus, not only Credo AI winning, but I think this is the moment the entire governance assurance security ecosystem is all going to rise. And what a great opportunity for us from not just like a prosperous future, but economically benefiting everybody. I think the, this wave is going to make all the boats rise together. The second optimism point is the rallying we are seeing at the international level on having this discourse as to how do we collaborate on international standards for what good looks like for measuring these frontier labs, but also when they deploy in enterprises, especially at the use case level, what does that look like? And then the third point of optimism that I get very excited about is I think this is a really nice coming together, even with different perspective of the research AI research community, AI experts like us, and the policymakers and regulators. And I think the more we can listen and exchange ideas and align on ideas, the better it is going to be for us to guardrail and govern this foundational technology that is here to stay and not going to slow down any time. And I think is going to be a very important, not only competitive, but beneficial endeavor for, for our country and for humanity. Jason Hiner: You know, when we met last time, remember you put it in perspective, you had some numbers that put in perspective the challenge of governance. Can you remind me what they were? There's some resource challenges that show, you know, the uphill battle that you were talking about before. Navrina Singh: Oh, yes, where to start? How many more hours do we have, Jason? But I think just like concretely, I think there's three things that I would love to point out to is one, if you look at the amount of investment that's going into frontier AI capabilities, as well as the AI application layer, there was a very interesting research which was put out recently by Gartner that we are spending about $2 trillion just this year on that frontier AI capabilities and application. And by comparison, the amount that is going collectively into governance, security and observability is around, you know, $500 million. So that's like a 5000 to one ratio, just to give you things in context of how much we are investing in capability versus how little we are investing in governance. And that, by the way, in the startup ecosystem, the VC dollars, I've not done that analysis, but I do want to, that is also a bit small. So that's one thing. The second thing which actually is even more eye-opening and it is shifting is within an enterprise, when you look at, I speak to a lot of public company boards. Most of the public company boards, given historical context, took a little bit of time, but now they have cybersecurity, you know, board members, they have a lot of focus on security, they have independent audit, but they haven't really grappled with how do I create AI governance focus role, which is paramount right now. So there are now, right now, committees of individuals involved, because AI is impacting all parts of your business. Great first step, but I think that needs to translate into how much is an enterprise spending on AI adoption versus AI governance. And, you know, we can share some numbers, but I think it's again, a very abysmal, less than 1%. And I think that is changing, which is great for our business. And we are seeing that, but it needs to happen faster. And then the third thing that I'll leave you with is just, I think in terms of AI literacy, I don't think you can govern something that you don't know how to even use. So we are actually seeing the number of AI native and AI forward organizations, they are like less than I would say 1%. And the public companies, as well as large enterprises, need to invest more in AI literacy so that your employees can benefit from this technology, but also do so in a safe, compliant, trusted manner. Jason Hiner: Very good. All right. Navrina, very sobering stuff, but also hopeful as you touched on your, put your entrepreneur hat on and saw the opportunity there. One thing I want to make sure to clarify as well for the audience is, you know, governance is an umbrella in this case that is including cybersecurity, privacy, all of those things as well. And just to help frame it so that we can help people so that the frame isn't just governance equals regulation. In this case, governance means safety, security, privacy, data sovereignty, you know, all of these kinds of things as well. Right? Navrina Singh: Absolutely, Jason. And again, I'm a big foodie, so I'll leave you with the food analogy. Governance and oversight is really around accountable mechanisms across your organization, across your AI deployments, your use cases, and across runtime. And the way that it layers in across these three layers of cake is it holistically looks at what are, what are the decisions and choices you're making in your data sovereignty, in your privacy posture, in your security posture, in your reliability, robustness, and in your internal testing, in your monitoring. And it gathers all that information and then looks across this three layer cake of organization, use case level and runtime to make a sort of like a decision on what you should as a business be doing versus not doing. And what becomes really important is something we share a lot is governance is the brain is deciding the what and the why, what you should be building, why you should be building it, how should you be deploying it for your customers for your different stakeholders, whereas the security, the observability, privacy, these are the muscles, they are actually at runtime figuring out what things are going wrong and feeding that signal back into the brain to say, let's not do it again, or I'm allowing it and these are the changes you have to make. So governance really is that holistic posture, which informs very technically what you as a business have

committed to, and do you have the evidence and verification supporting that your systems, your agents are actually holding those commitments responsible at runtime at use case level. Jason Hiner: Very good. All right, I'm going to ask you the same two questions I always ask at the end of every episode of all of our guests. The first one is what's the AI tool that you're using right now? You obviously have access to a lot of founders and companies and you see a lot, but what's the tool that you're using right now that you wish more people knew about? Navrina Singh: That is a tough one because I am very avid and a super user of Claude. So everything from our coding endeavors to building design templates and videos, I would say that Claude is my go-to day in, day out. I am also actually very impressed with GPT capabilities, especially Astra models. So those are my two goals. I have not used any other tools beyond that because they actually meet the purposes and if they don't, I actually end up building agents using that. Jason Hiner: Very good. How about in the age of AI, one of the promises was like, AI is going to be able to automate away a lot of the grunt work, a lot of the things that we don't want to do. And so the idea is maybe you would touch grass a little bit more, but the likelihood of that seems pretty low at the moment. The more people I know that are doing more and more with AI tend to be working more and more at the moment. And so one of the things that I love to ask leaders are, because leaders now are talking about how do I optimize my time for maximum leverage? How do I usually, how do I use my opportunities with these tools to prioritize my highest opportunities? What's your recommendation for leaders, your best tip for leaders who are trying to maximize, get maximum leverage for their time in the age of AI? Navrina Singh: Great question, Jason. I always tell my team is focus on three things because that's our super powers right now. One is develop deep expertise in one domain because at the end of the day, what's going to become important is these, we cannot compete and we should not be trying to compete with AI in terms of creating outputs. But are those outputs valid? Are they accurate? Will really require our own expertise? So think about LLM as a judge where LLM is judging the outputs of LLMs. I truly believe that we, if we develop our expertise in certain areas, we can actually work better with AI and the human AI combination is going to be so much greater. So always tell my leaders focus on building expertise deep and wide in certain areas. The second thing, big investment in taste because I truly do believe creating outputs is so easy, but creating beautiful products, useful products, it's a matter of taste. And I think humans have this fundamental, I would say ability to have that, you know, to love beauty. So I think this is the moment in time that we should actually be leaning into our tastes a little bit more so that we can build the right outputs. And then the third thing that I ask my leaders to spend a lot of time on is judgment. Because at the end of the day, yes, you know, I, for example, have created an agent, which literally is sort of replicating my board and is before I go into a board meeting, setting me up with all the different scenarios that are going to be posed to me about my company from the board. And then what I have found is in that moment, the judgment matters so much to be able to collaborate with AI in a way that is going to actually amplify the way I operate as a CEO and founder. So I think investing in expertise, taste and judgment is the way that we win human and AI together, not one or the other. Jason Hiner: Navrina Singh, thank you so much for being here. What a great conversation during a very sort of busy time for you and what a, you know, a very powerful moment, you know, that we're in an industry. Thanks for all the work that you're doing and thanks for conversation. Navrina Singh: Thank you so much for having me, Jason.