This New AI Study Is INSANE, and Claude Is GATEKEEPING

Nahshon Cook-Nelson: Coca-Cola has the option in their advertising contract to say, if the Coca-Cola ad is playing on television, I want my feed of people listening to the Coca-Cola ad to also see the Coca-Cola ad in the feed while they're looking down at the phone, not up at the television. Having someone who has general AI expertise build your solution is a lot different than someone who knows the domain really well. Domain expertise is important, but documenting your domain expertise is even more important. The tooling is becoming more important than the model itself. Welcome back to the Return on Intelligence podcast. I am your co-host, Nashawn Cook Nelson, growth marketing lead here at 923 AI Studio. And I'm joined as always by our wonderful CEO, let the church say, Andrew, amen. Nay, nay, and ay, nay, nay, and ay, here to represent. You have like the best last name, amen, a man, amen. Tell that to the FBI. They think I'm a terrorist. And I'm not even kidding. The last name is on a list. That's not surprising. I feel like you just have the best stories ever. That's insane. Yeah, until you get pulled over with your kid in the backseat and they're asking you all these questions about why your last name is what your last name is. Before we get into you helping people implement AI without wasting time and money, which is what we're here to do, I just want to say I'm sincerely sorry about what happened with Yukon. Michigan we had to do you guys like that, but your better team. It's your better team you cons had a dynasty Michigan was a powerhouse I We got 22 offensive rebounds and kept that game within six points. That was impressive to do fair I'm proud of the boys for as far as they took you con and they I mean I had the exact championship you con Michigan I scored 89 point 88 points on my bracket. So like no way that's pulling off that final game is Pretty impressive. I had the full Final Four, the full left half of the bracket. The Florida bracket was the one that killed me. I didn't predict any of that. But I got the Michigan bracket right, the Yukon Duke bracket almost all right. So yeah, I feel pretty good about picking good coaches. And that Michigan coach is a really, really good coach. Dusty, dusty. Nothing dusty about dusty. No, he's good. Hurley is obviously legendary, Dusty on my radar now is top five coach and good, like better than Izzo at this point. And that's a lot saying for people from Michigan. yeah. ⁓ yeah. Izzo is a legend. All right. Speaking of Dusty. Congrats on your win though. Congrats. Well earned. That's team in the bracket. Definitely better than Arizona. Best team in the bracket. I knew that from the beginning. Scary team and they deserved it. Thank you. Speaking of Dusty, someone came out with a new model. Again? Anthropic. Yesterday? another one just came out. Yeah, there's that there's that Twitter meme where it's like I go to sleep I wake up a new AI model is there's a some new breakthrough This is the mythos the mythos model the one that is apparently too dangerous to release to the public the one they keep trying to guardrail But it keeps getting internet access. Anyway, what do you what are your takes on this thing? First of all, why is it called mythos? They're doing like a ⁓ structure, right? Their naming convention was Hykou, then Sonnet, then Opus. It's symbolic, but what is it symbolic to? What's Mythos? I have no clue. That's a good question. Not the first one that came to mind when they said that this thing could like take over the world, but knowing its name could help. Yeah, I'm sure. The ancient Greek thing, my guess is like it's something along the lines of like story. The words Mythos are like, you know, fundamental. It's a story. I mean, story, but I have no idea what they're trying to reference in that. We'll have to look that up. The model itself, what is interesting to me at least is we should be talking past the frontier models because what Mythos is pulling off and what they're doing, especially with breaking every single security process that we've ever invented on computers and scoring what, 93 % on the SWE benchmark, we're all assuming that like the brain is the responsible party for solving all of these problems. And it's not anymore. The reason why in December, agentic coding and vibe coding became so prevalent was the tooling and the ability to control your browser, the ability to extract information from your computer, co-working using the terminal. The tooling is becoming more important than the model itself. The brain, the frontier model, of course, it's just predicting the next word. But how we're using that is why mythos I think is going to become legendary, right? It's going to become this model that we kind of look back on and say, this was the turning point, but it really has nothing to do with the brain that much anymore. How much more information can you chug into these things to predict the next word? We're there. Like it's just predicting the next word, regardless of what more smarts we put into it. The reasoning. the brainstorming, the ability for it to loop on itself. The observation aspect of these models, I think has greatly improved in the last five months. The ability for it to act and write code and then check itself and recursively understand if that code was according to the plan. And then going all the way back to the plan and creating a task plan and then sticking to that plan to develop something. Those are all tooling. And I would imagine that if we got our hands on this thing, we're going to realize that that is what the breakthrough is. the access to, we've always called them the hands in, like when we started talking about agents themselves, we always said, you know, there's the memory, the model and the hands, the tools. I think the tools are finally starting to take shape. That's what I'm saying. When you say tools, what are some good examples? Are you talking like out of the box tools? you custom tools that it doesn't matter? No. Yeah. I mean, it could be scraping tools. It could be a virtual machine. It could be the ability for it to. I don't know, understand security methodologies using some sort of tool. Cause remember the context window still kind of sucks. Like it's not going to remember stuff. So we have to find hacks and solutions as engineers to provide it memory, to give it a plan and then make sure it checks back on its memory because it's not going to remember some as it executes its next file. Running debuggers is definitely a tool understanding like how agents can operate and which agents are necessary, which ones are useful for the thing it's trying to accomplish. Those are all the tools. If you've vibe coded or goose cloud code. all of sudden it comes back and it's like, go install this thing. And you're like, okay, enter, yes, sure. And you install it and it's like, then it keeps chugging along. Those are the tools that I think it's using. Nice. So you get less excited by a new Frontier model and you get more excited by the tools that the models can begin to use. Cause even if you go use Sonnet with the right, giving it access to the right hands, you say that'll be even more powerful or useful to you than having some amazing model with. with no hands. Yes, but there's also a large part of my day, which is talking to companies that are saying, Hey, we tried that chat, chippy tea thing today. It's cool, but it doesn't do what we thought. Can we, you know, make it better? And so we're constantly teaching enterprise companies. were just on the phone today, Nishan with what a $10 million to a hunt could have been a hundred million dollar trucking company that's leasing land. And these types of companies are getting into the idea of what AI means, how it can work for their business. And when you just can predict the next token and you can put them through an agentic process, I don't need a new Frontier model anymore. It doesn't bother me that Mythos came out or Claude or whatever Chachabi-T is producing nowadays. The models that we have in our hands are producing substantial returns for the businesses that they're installed into, especially if they're installed the right way with guardrails, with knowledge bases, with team OS, all of that. If you're installing it correctly with the current models, you're still revolutionizing the process that a company currently has. Yeah, I like what you said. Something you said on that sales call and you just said here is the process. We kept asking them, what's your existing process for doing this thing? Because the better you know that, the more you can understand if that's still relevant for an agent to do or how you would set up what tools it needs access to. And so I think one question I have for you is, do you see domain expertise being really important as people begin to build these agentic solutions? because having someone who has general AI expertise build your solution is a lot different than someone who knows the domain really well. Yeah. Domain expertise is important, but ⁓ documenting your domain expertise is even more important because the agents can't read from your brain. They have to read from a structured knowledge base that you've maintained over the years or the process that your company has maintained over the years. And I think people forget the fact that like, you're not just going to install ChachiBT chat into it. for a few conversations and have it understand your 20 years of knowledge that you've ingested from the company that you're in. What we have to do is learn from the domain experts and they have to document to us their processes. Every system we've replaced, whether it's been a $10 million revenue increase with $200,000 of cost, or we took a $500,000 business and made them a million dollar business and, know, what was it? $150,000 of technology. Any of those solutions requires documented processes across their organization. for us to understand where the agents can succeed, where we have to guardrail them, where they can fail, and then install that properly so that they don't have to monitor it, it doesn't break, it's not like a N8N solution that has like duct tape tied to it. It's truly a robust system that they log in, they trust, they know they get to the answer, and it deterministically, for the most part, gives the same answer every time they ask the same question. If someone's scared of documenting processes, my favorite thing is, I don't know if you've seen that loom. allows you to generate SOWs if you create a video. ⁓ So you can record yourself. I just record myself. I'm like, ⁓ I need to send Alina how to do this whole thing. I record myself, hit generate SOW, and it takes screenshots from the video. So it'll show me, it'll show what I was showing at that time and create it into a doc. And then you can, there's your procedure, right? Yeah, it's a really cool process. And it's the person reading the procedure, puts it into Gemini and says, me the three things I need to learn from this. Yeah. Yeah, it's true. just, there were times back where I was like doing physical binders of procedures and you know, like putting things together and then two weeks later the platform changes and now that procedure is irrelevant. ⁓ That should be our marketing material. Do you have a hundred binders on your shelf? Like that's where the entry point of AI is. Cause you can just scan them now without a scanner. You just take a video of them all or whatever you need to do to get them into digital. And then you can document all those binders. That would be an awesome project. ⁓ man. Yeah. Yeah. You were talking to me ⁓ earlier. You should say. Are you allowed to talk about nuclear codes that you've seen or is that off limits? I worked in the nuclear industry. I worked on nuclear submarines, but the information about nuclear, how the nuclear submarines operate, I wasn't even privy to. So I don't know things about how nuclear technology works inside of submarines or on the United States, but yeah, the nuclear submarines and how they're built. Okay. All right. Onto the next thing. Meta's making moves. Meta... is, you know, they don't want to be left behind. So they acquired MoteBook. And for those of you who don't remember what MoteBook is, that's the social platform where agents are talking to other agents about their humans and doing everything from complaining all the way to actually revealing personal information about their humans. I was curious what you thought about that acquisition. It's completely pointless, except for one thing. If you wanted someone else to build the new Facebook and you wanted to pay them without giving them a single dollar to build it, test it, deploy it, launch it, get a hundred million people on it, and then purchase them. The playbook you do is you buy MoteBook and then the second one, and then the third one, because eventually somebody's going to hit a home run. Social media and Zuckerberg, I've read a few books on him. His bet is that social media is kind of like rolling a dice. Eventually the users start coming and then you get this thing he calls the R factor where there's more people recommending friends on it. And it becomes the place where all the eyeballs and the agents and whoever is trying to be attracted to shows up. You don't know when that happens, but there is a tipping point and Malcolm Gladwell is great at explaining it. When that tipping point happens, it's about like 23%. But when that 23 % starts rolling over to like the maximum amount of people that are now starting to use the same social network, they're no longer going to be using Facebook. And Zuckerberg knows that that day is coming. So how do you get the world to build the next Facebook for you? is you tell them you will acquire anything that gets traction. And so now people are looking at MoteBook being like, I could build that. I could build that better. And eventually someone's going to win. And Zuckerberg's already told them through purchasing this product that he will buy the next successful social media for agents. Wow. So you don't think they're going to start running ads to agents and then the agents will put an in-feed ad on the MoteBook platform and then the agent goes buy something for their... They're human. don't, but I do think, you know where the next ad platform is, is ⁓ you and I both use Readwise, which is a phenomenal tool. Like, and there's this tool called Sniped, Sniped, S-N-I-P-P-D. Of course it's a startup, so they don't use vowels. You know what talking about, right? Yeah, yeah. It's the podcast, like clipping and listening and stuff. Yeah. So I was, I, so basically how it works for those of you who don't know is like, you can be listening to anything. So I download all my books instead of on Audible. I extract them to an EPUB, run them through Snipt. And the reason why I do that is as I'm listening on my headphones or my phone, I double tap and it will take what I just heard the previous 30 seconds and the next 30 seconds and it creates a flash card for it. And I have all these flash cards saved for the last, I don't know, four years that I've been using Readwise and Reader. And every night before bed, I go through the latest 10 of the day. So it's kind of like refreshing your memory of what you learned that day. And it's through a series of flashcards that come from listening or reading, because when I'm reading something, I highlight it and I send it to Reader. When I'm listening to something, I double click and it sends it to reader. The other day though, I was doing it and I was reading my flashcards and I listened to Lenny's podcast and he talks a lot about, um, there was an advertisement for it's one of those software products. That's like monday.com linear. think it is linear. And that was the snip that I was reading of my content that the agent had picked up for the 10 things for me to learn that night. And it was just. the straight up advertisement from Lenny's podcast saying linear, how great it would be for my company to use linear. So not only had it taken the advertisement, it compressed it down for me, re-managed it or reconfigured the text to make it sound relevant more for me, cause that's what Reader does. And then showed me that at night as like a thing to review. And I'm like, that's the future of advertising. ⁓ But people would have to be like capturing, like I think you're, you're, ⁓ power user, but those are probably also the people who have a little bit more money, at least in the tech space. Don't forget, we built Raycon, right? Raycon's power memo note taker. That little note taker, it's kind of like the plot, but it's way better because it actually transcribes your day. It actually has knowledge about who you are, like, keeps track of more regular people stuff instead of plot that's like a business card and taking notes for meetings and like... But the Raycon one, like it's a phenomenal piece of technology that you just record and I use it all the time. And there's still like, hey, remember these few things. Remember the doctor's office. Remember that to pick up the kids. Like it's way more relevant for that. So I think all day recording is within the next six months for most people. And then you feel like people should get ads based off of their recordings. Like, hey, I saw that you had a lot of meetings related to this. You might actually benefit from this. Is that cheating though? Cause that means they took my data, analyzed it and... So that's one way. The way that I'm speaking of is if I was to hear an ad in the background while that was recording, then they would have permission to repeat that ad. now that's interesting. Ad recall. Because then you would say, my gosh. So then people can know like, hey, when you advertise with us, you're going to reach this many humans, but you're also going to reach this many note takers and agents, which we have a higher recall rate with these agents. Yes. Yes. That would be... Yes. And think of where that goes, right? Cause the next cool thing is if you're advertising in places in which audio happens or video happens, I can have two screens open and you know, Facebook does this by the way. So you can advertise on Facebook. Not many people know this. This is why I'm, I'm stating this out loud for like the surprising people. You can advertise on Facebook. If you're Coca-Cola and you're watching the Superbowl, Coca-Cola has the option in their advertising contract to say, if the Coca-Cola ad is playing on television, I want my feed of people listening to the Coca-Cola ad to also see the Coca-Cola ad in the feed while they're looking down at the phone, not up at the television. That is an option Coca-Cola has when they advertise on Facebook. That like double hit, I think is what agents are gonna start understanding because you know the context the person is in. You just got permission from Coca-Cola to advertise, so why not put it in the place that their eyeballs are? It's crazy that at one point we were afraid that our phones were listening to us and now we're like, all right, I need my phone to listen to me. Constantly. We're like giving it permission for our own value and of course somebody's gonna exploit that. Yep, it's inevitable. This has to happen on iPhone. I have a Google more frequently than I carry my iPhone, but Google has this thing called remember a song. Is that on iPhone? Remember a song. I don't think I know of the feature. So like if I'm out in a bar or just a restaurant and a song's playing, I can look at my phone and it'll tell me which song's playing in the background. Yeah, you can say like, hey, what's the name of the song? Like that you're saying? Yeah, but Google takes it one step further and it's passive. It's on my lock screen. So if there's a song playing in the background and my Google phone is just sitting on the table, the name of that song will be on my phone. That's a problem because it's all about the wake word, right? That's Alexa's big thing, the wake word. I'm never listening unless you say the wake word. So you have your Google to be always listening. You don't feel weird about that? That's my point. That feature is specifically... supposed to be listening for songs. But they get- no. You know that Google's not doing that, right? They definitely are just like, how do we get them to keep their phone on all the time? Well, maybe they care enough about music that I can get their permission. Well, in that big, long acknowledgement thing, I'm sure I clicked yes. Yeah, no, that's crazy. Because when it says like, ⁓ Alexa, I only listen when you say Alexa, it's like, that's fair. OK, maybe Alexa, maybe, maybe, probably not, but maybe. But if you say I'm only listening for a song. Like, I could start singing right now. Is that? Yeah, you'd show up on my phone as a song. And, you know, that product manager should be like a billion dollar product manager at this point. They came up with that idea, right? Because Google, they probably started from the back. They're probably like, how do we get to listen to everyone legally while getting their permission? And they're like, ooh, what do they care about most? We can listen for people singing songs and tell them what song's playing. And then all of a sudden, boom, they have the data. It's crazy to think about. I got two other things before we get out of here. You still got a minute? Yeah, yeah. Okay. So one is your age old favorite. Managers and executives are not in agreement. There's this new study that came out from Harvard Business Review showing that managers are pushing back. Executives are saying, hey, go implement AI, go be effective, go here's money, here's budget, go make it happen. Managers are coming back saying it's not working, it's not happening. And then executives are saying, no, I see what I see online. I see what I see in stocks, block, everyone. just make it happen. So there's tension. I'm curious. This is a real study, BCG, McKenzie, et cetera. What do you think is going on here? I think it's the obvious case of like shiny object syndrome. You could look at a construction site and then go tell the guy standing next to the construction site, here's a hammer, here's some nails, here's ⁓ some plumbing glue, or whatever they call it, plumber's tape, ⁓ and some pipe. And you should be able to do everything that those contractors are doing, because they got the same two hands and two feet that you do. Managers are realizing that it's an art. It's, takes time and these little Claude code things that other companies are putting together, we're probably built, especially like block. They're built by really good engineers that have a lot of talent that early access to some of these models that are building it with security in mind first with making sure the architectural is properly set up before they start by coding it. I know when our engineers at least code, like they don't just hit the blue button every time Claude comes back or their orange button. They, ⁓ they actually read what Claude's doing while it's happening, which Sounds crazy to me, because I don't, I have no clue what it's saying half the time, but they read it. They sit there and read what's happening and they're like, ⁓ no, stop, don't go that direction. And they veer it off into the direction they wanna go. That's a completely different approach that a company is doing and then claiming that they have success without looking under the hood at the process. These managers are now being told to copy that and the manager doesn't have time to sit and read and understand what code is and be an engineer. And on top of that, be a product manager on top of that, talking to the board. And so they're being pressured to use the system to same way. These successful companies are that have early access with the right frameworks, with the right structure. And that's why agencies exist. That's why consultants exists. That's why hiring people that are younger, that are already using these tools that have already experienced fairly or failure that can solve the problem, you know, without. remaking the same problems that your company has. That's what you do. You hire the young, energetic, interesting people that have already tested this or came from block and now want to try something new. That's how the world works. And that's how accounting businesses worked back when Excel came onto the scene. You didn't keep like fighting the managers that were trying to push you on pen and paper and saying accounting can only be done with T charts. You started hiring the best accountants that were on Excel and realized the productivity was doubling, tripling, quadrupling. And you continued to hire those people until the T chart. pen and paper people were no longer around. I think the managers are getting into that mode of how do I achieve the needs that the board or my bosses while using these tools, but also how do I get help? the getting help part is not being mentioned. Claude code, you hinted at it and I want to acknowledge like there's real impact on the market, even as an agency, as a development partner, people saying, Hey, I'm going to use Claude code instead. So it's not just happening on the individual level. It's actually companies who have no software engineering experience at all. No engineers on their staff actually saying, Hey, ⁓ I know we're mosquito company. We're going to use cloud code. So I'm, curious, how are are you processing that as an owner of an ⁓ development agency and how do you see this landscape? we just had our quarterly update and you were on there and I said to the team, I don't know what the future brings. I don't know how Claude code is going to be integrated into all these businesses, but I do know like two things. Like we actually lost our third project today to Claude code to up, we're 88 % since October 1st of 2025. So we're like four months of a great run and we're starting to lose projects again, which is new to us with this Claude code. Cause we've never had it in our sales or in our marketing that like, we're to build you this thing, but also you could do it with Claude code. The thing that Claude code doesn't help with. is you become QA, you become engineering, you become product design engineers, you become the PRD creator, you become the bottleneck of the next version of Cloud Code. We were joking, like we should launch a $50,000 like recovery service. We're basically like, if all your engineers use Cloud Code, who's going to audit this? Cause no engineer is going to want them to look at a bunch of agentic code and how the crazy things are all meshed together with the dependencies and crazy variables that it created out of nowhere. Who's going to audit that? What human is going to read that and understand that it's a spiderweb of nonsense. The joke is my vibe coded tool that we actually need to use across the company for a few different departments has to go onto the internet. And when it does that, my CTO or my co-founder had to look at it. He was like, you're an idiot. Like I took, I don't know, 400 lines of code and made it 40. I then took 4,000 lines of code and made it a hundred. And he was like, it's just, there's so much bloat. There's so much code being pushed through and it's not necessary and it's hard. And so if you keep building on cloud code and you keep using it as a primary like tool for your company, you're going to like have way too much bloat that is impossible to fix. And you're stuck with the Excel problem that every manufacturing company on earth that hates their ERP system. is using today, which is some random Excel sheet that has 18 tabs and 45 variables and some VBA code on one page and none on the other highlights formats, excels tables, some sort of formatting that like print a perfect page for their manufacturing shop and one of the tabs and VBA code that taps into Microsoft access on the other one. All so they didn't have to use ERP. And we're thinking that like that world is over, but the Claude code world is new and fun when it's just going to happen again. with Cloud Code. It's going to be the Excel of 2028. Yeah. I think you said two things I found extremely valuable. One is bloat and unnecessary. Yes. That means unnecessary cost as well. We have this graphic going out next week that we'll put here. And it's basically a really small guy and then a really big guy. And then the small guy is like, hey, man, you got to lay off the prompts. I just love that. The bloat is so real. And it's not just engineering. It's marketing. It's copy. Websites, it's it's everything. It's just so much more than it needs to be I think we'll swing back to like essentialism minimalism Etc and then the other thing was cloud code in the hands of someone It's like the Thor hammer give cloud code to me and I'm gonna be like dropping it like I can't even lift it up But give it to Dennis who's an engineer for nine plus years just here. Yeah, never mind 20 years experience. Yes, and he it's a total different He's flying through the sky doing amazing things and pulling his hair out reading the horrible things that it's fits out Yeah, but it's like the spec driven development these understandings and the thing you hinted at earlier is time if you are a Construction CEO you don't have the time to waste going and setting up open claw on a Mac mini and think Why do that when you could just pay someone to help you set up something that scales your company? That's the thing too is like I've noticed a lot of individuals like, oh, Johnny knows how to do this thing. But does Johnny know how to deploy that across a 500 person company? Does Johnny know how to do that for 50 people? Is Johnny now going to become the IT department at the company? That's right. Without schooling, without training, without experience. Yeah. There's more to it than generating the code. And I think you put out recently, like, the code actually isn't the bottleneck anymore. You quickly realize that. Yeah, it's the same thing with like, there's a lot of companies that, and especially agencies that I hear of that are building the zero to one and they get these entrepreneurs that are non-tech entrepreneurs. And so they figure out they're going to help them, you know, solve the zero to one as a, as a technical support for their non-tech founder. part that the company, and I promise you, the agency will not tell them on the intro call is that that's only 10 % of the cost and the time, because what about marketing sales? You probably need to get a CEO on top of the the product that you're doing, because if you don't want to run this thing all day long and you just want this product to succeed, you're to have to get someone to run it. Maybe an operating officer. Then you're going to get some sort of phone call, customer service. Like you're going to have to explode this entire product that we created for you 10X and your cost for us is 10 % of the total cost necessary to get your return on investment back. It's the same thing with cloud code. Your first product that looks really cool and fun and kind of works on a computer for your actual information is 10 % of the operations that it's actually saving. The rest is training the people, getting them on it, getting them to use it repeatedly, getting the right answers out of it every single time, then QA-ing it, finding a bug, circling back and fixing the bug. Who has time for all of that and getting it into production? It's just not a thing that companies need to be doing right now. So if you're going to do it, do it right. Get the help. Or even if you're going internal, have a... hire experts, doesn't matter how internal, external, partners, whatever. Johnny who's in sales and is using some great tool, good for him. But when it becomes that company initiative, it's time to treat it just like any other company initiative. Guys, if you're interested in conversations like this and you find it valuable to not waste a lot of time looking through all the muck of information. Click the first link in the description to get on our newsletter. This is the AI newsletter worth reading. We send it every two weeks because we take the time to sift through all the noise and give you guys what's most impact. And if you are a company that says, okay, we want to do a company AI initiative, another link in the description is to book a discovery call with Andrew, Mr. Amen himself, and he will take a look. and see if we can find some AI opportunity for you. good. It was good talking to all of you. Hopefully you stay subscribed and we'll keep putting these episodes out. Sounds good. See you guys all next week.