Why companies are canceling Claude (Custom AI Harness Explained)

Nahshon Cook-Nelson: How can you monitor what is going on across all of these individual Claude accounts? If they go custom, maintenance is always a question. One thing I love that we do on this podcast is we talk real examples. If everyone's using Claude and all your competitors are using Claude, what is your differentiator? Your differentiator has always been your IP. The reason why you may want to go custom is also a question of both scale and governance. Harnesses usually have specializations, right? Generic harness can only be just that generically helpful. Claude Code is a good example of a harness that I think is just like productizes its own thing. First starts with figuring out. Welcome back to the Return on Intelligence podcast where we're all about helping you implement AI by bringing you experts who are actually in the field implementing these solutions, not just talking about it. Today we have one of our very own product managers, Michelle. Michelle, hello. Hey, Nishan. I'm so excited for this episode because we've been wanting to do this for a long time. You're a very entrepreneurial person. You're also a very product oriented person who's had the pleasure of working with very large companies and mid size and small. So why don't you kind of tell someone who's listening to this what they're going to walk away with from listening to this episode? Sure, I hope the person who is listening to this episode walks away with first of all understanding what a harness is, but then also figuring out if building a harness, a custom harness, is right for their business. I think there's a lot of solutions out there that answer a lot of really good problems, but there are points where sometimes building custom is the only right answer for some of your internal business problems. So let's get into it. I'm curious why a harness? Like there's so many things we could talk about models, agents. Why is a ⁓ Harness an important conversation to have. Yeah. Well, first of all, I think it's a buzzword that's being thrown around quite a lot right now. And people are like, ⁓ another thing, right? But I think before we answer that question, I think we need to step back and talk about how does a harness or what is a harness in relation to like the world of AI. So the first one is we all know what a model is. And to explain a harness, I think we need to orient ourselves that a model on its own is just like a tool and mechanism that predicts the next best word. That's really it. The harness is wraps the model and I would say makes it really useful and topical for a ⁓ certain business problem. A harness like shapes the context the model gets. It shapes the tools it reaches for. So in Claude, you may have seen where it automatically reaches for something in your productivity tools or in your native recording tools. A harness is what's helping it figure out that that's where it needs to go to try and retrieve an answer. And it also helps figure out what is the judgment on how do we define the work to be done and when is it done enough to be able to return an answer. So again, it's all using whatever model you want. But the harness is what I think makes the model just a little bit more powerful and able to answer some of these more complex problems that you may want to use AI to solve. So examples would include Claude, you mentioned Chat GPT. So are those harnesses or are they like using harnesses? Like how does that work? Yeah, that's a good question. So Claude and ChatGPT are both model providers and they also have these harnesses that are probably already embedded into your workflow and you may not even realize it. So Claude Code is a good example of a harness that I think is just like productizes its own thing. But really, what it is is a mechanism in which helps you develop better code by understanding context of different things. Like what's the type of thing that you're trying to solve? What is the goal and what is the outcome of this code I should develop? First, people who aren't developers. Who are maybe using Claude. If you have ever asked Claude to look at a previous call that you had with a client and you have these connectors on to whatever AI note taker you may use, and you say using that call, do the following, and you have maybe project files or you've linked it to your Google Drive. The harness is what's making all of that possible. It's understanding that you have this need and that there are different context or layers that we need to incorporate to figure out if and I'm personifying the. The the model here a little bit to figure out what is the next best word, what is the next best answer for this particular query. One thing that stood out to me when you just brought up Claude Code is that there was a new harness created for the work of coding. So it almost seems like if you're a lawyer, it'd be really nice if Claude had Claude Legal and it was all about like legal work. Is that kind of what you're hinting at when we talk about wanting to create a custom? Harness? That's a really good question and actually a great way to think about it. So harnesses usually have specializations, right? Generic harness, it can only be just that generically helpful. But if we develop a custom harness that is armored with, if you will, the right context for the job to be done, the right skills, the right instruction, it's going to be a lot more useful for you for whatever you want to use it for. So thinking about and leveraging Claude Code as the example, you're absolutely right. Look at Claude. design. It was very, very similar approach where we take this job to be done. Which is coding or designing or law. Maybe that's another product area that they could be developing. But we custom design harnesses to be able to allow the user who's approaching this tool to better achieve results. You can use Claude to probably help you with design, but those of us who've maybe have used Claude and Claude Design know just how more helpful Claude Design is because it has special specialized harnesses to help it do that particular job. So we're starting to get now into this idea of a custom harness. What is a custom harness? And then how do I decide if I should just let Claude do its thing? Like if I started six months ago, I'm gonna do a design Claude and then now I'm screwed because now Claude design exists. How do you ⁓ delineate what's worth doing and what's worth just using the existing tool? That's a good question. And it's actually the exact value I think that we help our clients figure out and navigate. Now, a lot of our clients, I'll be honest, I don't think care what we use, right? As long as we solve the problem. And they just want whatever problems they're facing to be solved. I think it first starts with figuring out what are we trying to accomplish, right? What are the long-term business goals and what are the specialized either workflows or problems that that business is facing that needs to get done. If the outcome is deterministic, which is another word that I'm gonna try and define here, or something that is repetitive. Think of like, for example, pulling in yesterday's invoices and summarizing the total, the inputs and the outputs are always going to be the same. You don't need a harness for something like that. You can generally just use a script or something that is more solid and concrete because it's always gonna yield the same answer. A harness is better equipped when The problem and the job to be done is a little bit more convoluted. It's a little bit more squishy. Think about internally on any processes that may not be standardized. Think about deliverables where you may have 10 people in your organization all work on something similar, but at the end of the day, despite them still being able to get the work done, the deliverable looks different across all of your employees. That's a potential problem to solve with the harness because we recognize that it's likely. There are all these inputs maybe of unstructured data. Think of like call transcripts or structured data that may exist in like CSV files or Excel spreadsheets that need to come together and help get a user from point A to point B. And to be honest, figuring out when to go custom, I think is more. is better answered when you really assess where do we want to go long term. Because there's a lot of great custom, I'm sorry, out of the box rather, harnesses that exist in cloud right now that can help you get work done. But going custom I think is most beneficial when you have a widespread organizational issue and standardization as well as privacy and other things are important when you're trying to figure out how to approach including AI in it. So what I got away from that in terms of like squishy is things that are high stakes workflows that are not as predictable and maybe where your team is struggling with adherence to procedures even with existing AI solutions. So they're all using Claude, they're always using that specific skill, but then all the outcome looks completely different. Totally. One thing I love that we do on this podcast is we talk real examples. I would love because I know you just created a custom harness for a few of our clients and you've been guiding them through that process. Can you talk me through a real example of how you went about doing that? Yeah. So we have a client. Who ⁓ came to us because they know that they need AI to level up their organization. And there were actually a lot of different areas where we could probably get started. But some of the things that we try to evaluate is where is the best place to get started? Because if you think of just operational fit efficiency, let's solve the right bottleneck first. Anyway, what we identified was that they really needed help figuring out how to best standardize essentially a change request document that they have to create in order to help their own customers maybe. Solve custom problems that they may see. And despite any organization's best efforts, this document lacked standardization, despite it being actually really important. And everyone at the company who is responsible for maybe developing this type of document honestly developed it slightly differently. And even though the work is still being done, one thing that we noticed when we were trying to evaluate is this the best place to start and how do we solve this was figuring out what were the downstream consequences, both internally. As like if this document needs to be handed off to someone else, what are the feedback loops that are happening because the document is lacking information that the next person needs to do their job? And then even further downstream, if there are still questions that aren't answered, and then it gets back to the customer or client who their change request is actually for, there's even a larger feedback cycle. So one of the things that we try to look at is figuring out: okay, given everything that we Know that this particular client needs to use as data to be able to develop this document. How do we develop a harness that maybe leverages transcripts or data that they have on this client or unstructured data that may exist in the means of notes, CSV files that may have information about this client and combine it to not necessarily write the document, but to actually develop a thinking partner that helps the person who's responsible for writing the document actually write the document? One thing that we were able to leverage. Is that because this client has historically written many of these documents before, we can actually also use that as part of the input, if you will, or the or the context and help generate just a more a much more powerful thinking partner. And even though we actually do add some text generation, if you will, the text generation is only made possible because the harness helps the user and all of the data that possibly exists out there that could be helpful have a conversation and figure out How do I, the person who's responsible for this document, best develop this document that will answer all of these necessary outputs, if you will? Those necessary outputs, by the way, are what we actually equip the harness with. So it the harness knows how where am I supposed to get to. Essentially. And the impact here is just fewer bounce backs from now we no longer have half baked documents. They're a little bit more robust. They're able to stand the test of time. They're more comprehensive. And we're seeing less feedback cycles overall because we're able to just augment a document that much more with relevant details and information. That gets me very excited. Very excited. That's Because th like it just clicks some things. And I hope responsible Ralph or responsible Rachel, that's who we call Our audience of people who are responsible for rolling out AI to their team. Someone was asking me the other day, like, why when I could just do that with Claude? It's amazing because it's like for a single person, solopreneur, one-person team, go bananas, like go have a field day figuring that out. But when you have a team of 30 people, we've worked with the sales team recently that had over a thousand people. Now we talk about adherence to procedures and consistency in QA. Now you're able to get that level of. scrutiny and support. Maybe your engineering team wants to figure out how to create a custom agent and run it in their cloud code and sit and cloud deploy it on aid. Maybe they want to figure that out. But does the person who's at the nuclear site who's responsible for catching those discrepancies, those degradations on a nuclear machine, want to figure out how to create a custom agent or do they just want to take a photo of the degradation and know what it is? Yeah, exactly. Exactly. That person wants to take a photo and leverage every other resource that existed in the company. To try and help them figure out the next step. That's what a custom harness is, is surfacing up the right information, if you will, in the right time to get you from point A to point B. Now, abilities. So I've seen like there's MCP, right? Connectors. For those who don't know, model context protocol is what powers connectors in your favorite LLM. Do we have the ability to do things like that? When they start creating custom harnesses, can they bring in data that Claude can bring in? And can they bring in data that maybe Claude doesn't even support yet? ⁓ definitely. Think of like APIs, think of webhooks, think of like MCPs as well. Like we'll use a combination of whatever is going to get us the data that we need, right? And sometimes you can use a combo of those. Sometimes it's more limiting on the vendor itself. That third party tool may only have one or the other or whatever. And what's really nice about harnesses is that we kind of are picking and choosing what we both have access to. And then also, given that that data is presented, then applying maybe like a skills file or instructions on top of that to figure out how do we interpret this information and where do we go from here. Talk to me about ownership and sovereignty because we know one of them is model agnostic is a really nice thing. We know Claude likes to go down every now and then. ⁓ so things like that. Where it's like, ⁓ cool, I I'm not claw dependent. What are some of those other benefits around that sphere? Yeah. So the way to answer this question, I think, is just to put it quite maybe Bluntly is like if everyone's using Claude and all your competitors are using Claude, what is your differentiator? Your differentiator has always been your IP. It's always been the way that you all do business, the way all you all approach problems. Claude, despite your maybe best attempt at adding skills or et cetera, which bu by the way is still achievable, is can bring that IP to life. But the reason why you may want to go custom is also a question of both scale and governance. But I think competitive edge is something that should be critically evaluated when you're thinking about do I want to use Claude or do I want to build something custom either internally or go with someone else? Because if anyone can subscribe to Claude tomorrow, right now, in a minute, and still have access to the same type of tools as you, and you haven't had the opportunity to apply your organization's special. And what get what makes you all great, then everyone's doing work the same way. You're yielding the same outputs. It's no longer a question of are you putting out something that is different than your competitor? You both are essentially potentially rather putting out the same thing. So now, walking away from this, should someone consider using or owning? Well, maybe give us a like a checklist to walk walk through. I just listened to listen to this episode. I'm like, this sounds pretty cool. How can I kinda go down a list of knowing custom or Claude or whatever chat? cetera. ⁓ okay. If you are an IC and the individual contributor or you're a small team and you're either new to AI or you're well versed in AI, as long as you all have a good internal communication, it's probably that Claude or other types of models that are providing harnesses can address a lot of your day-to-day responsibilities, a lot of your day-to-day tasks. If that is not you and you're thinking more organizationally, or that you have the marketing department and the Finance department and you are all working towards a common goal, then maybe Claude or these out-of-the-box solutions, if you will, may not be the best for you. And I think the reason why is because how can you monitor what is going on across all of these individual Claude accounts? And I don't mean in a big brother kind of way. I mean more of just is everyone using the same skills document? Which a skills document has instructions on how to do something. Is everyone using the same definition? Is everyone using the same ⁓ Input values that will instruct the context of the harness and help them figure out again point A to point B. If that sounds like, yeah, you know, my organization needs access to all those documents, and I really can't guarantee that everyone is going to be doing it. And maybe you also don't have the means to have a champion in each organization because that really is what it takes. It takes a champion to make sure that everyone's on the same page, then maybe you should consider going custom because if you go. Custom, what we can also do is add an observability layer. We can give you ideas into how are people using the system, but you also can get like an admin panel to say for my entire organization or for maybe only this organization, give them access to these files, give them access to these connectors, give them access to these set of tools. That way you know anyone who's logging into your system, given the role, given the permission, whatever else you configure, they are all using the same inputs. And so generally, even though using a harness means non determined. Deterministic outputs, you're generally all moving and marching towards the same beat and hopefully yielding very similar results in the end. I love it. I think the one thing I would add that I noticed you hinted at earlier and you made me think of this is the future proofing. And I think it's the one thing we didn't address that I do think want you to call out a little bit to Ralph and Rachel. Okay, you found this one workflow for legal. And now you've built your company GPT legal. Can you talk about how that now has created a foundation? And there's certain words that might come to mind around now they can build on top of that those other tools for other departments. Now, are you asking this if they're using a GPT using like cloud or open AI or if they go custom? Custom. So if they go custom, maintenance is always a question that we get. Like, what's the maintenance on this looking like? If we develop with observability in mind and maintenance in mind, which we do, obviously we need to, honestly, if you're developing a harness, you you must. But if we do that, then what that allows you to do is if you're we're connected to your SharePoint, if we're connected to your documents ⁓ repository, I imagine that you uploading those documents to your document repository is already part of your workflow. Guess what? If it's there, the harness has it too. If it's there, your custom solution already has it. In terms of skills, let's say That we have a skill that documents a procedure, exactly how you would write for the case of the previous case that I was talking about for the client. The skill that writes the procedure on how to write this change request document. Let's say that they're starting to use it, and in about 18 months or so, they realize that they want to refresh this change request document. There's a couple things they want to modify, and maybe there's a couple of additional parameters or constraints that you may want to add. Usually these skills are written in markdown. And so you can go in and a natural language just kind of instruct and change that procedure. So over time, as your organization matures, you're able to make ⁓ small tweaks along the way. So the most topical information is readily available. Or in the case of like the document being able to leverage your a document provider, as long as you are uploading files, we have the most recent thing. And I won't I won't shy away from the fact that maintenance is definitely a thing, right? Like we need to make sure that we are maybe not always updating to the latest model. There's that that's a whole different, you know, conversation. But There are going to be times where you may want to switch models, you may want to reflect on how work is being done, but if designed correctly, those small tweaks that may happen iteratively as we learn to work together in this new world using these new tools should not be a headache. Now, that's because you have an admin panel, right? Or something like that where everything is happening automatically with small tweaks. If everyone has those projects or skills locally on their cloud environment or on their Claude tool that changes completely if you don't have something like that. Because now what you need to ensure is that everyone has updated the skills, that everyone has now updating or pointing to the right file path, that everyone is incorporating these new changes. That is definitely achievable, but not without, I think it's paying and making sure that someone remains on top of it. We say pay now or pay later. So you can pay now by doing the effort necessary to set up a custom harness if it's For that specific use case that you highlighted, or you can pay later with lack of adherence, exposed data, shadow AI, you know, people using things that they aren't supposed to. It reminds me of the BYOD. And ⁓ like we we do BYOD and you install Dr. Spreno and you they know that hey, this is these are the things it's monitored. And I think Ralph and Rachel can get away from supporting several different vibe-coded things and more at that level up. working on the AI program change management from the top level rather than being in the weeds day to day. Totally. And that's such a critical I'm glad you brought up change management because I think that's another thing about, okay, so given that you said maybe I'm an organization that needs to go custom. Going custom in the world that I just painted, where maintenance when designed this way, is more minimal than if everything is locally on people's machines. That frees up the time of this champion to actually go do what I joke around is the real Work, it's change management because there's a lot of different new things that your employees or your organization is going to need to learn, adopt, modify. And that takes someone both distilling the information down to them, reporting up on what isn't working, right? Because this is all new. You're trying to figure it out together. And if Ralph and Rachel are spent making sure that everyone's got the same skills or got the same file paths or whatever, then Ralph and Rachel aren't able to really, I think, focus on the true value and task at hand, which is that change. Management conversation that is definitely a real, real challenge that needs to get staffed if you're going to do something like this. If you are resonating with being responsible for rolling out AI across multiple people, it's not just you alone that you are responsible for, then click the first link in our description and get on our newsletter. This is a newsletter written for people who are looking to onboard change management and get their team equipped with the tools necessary to be successful. Michelle, Thank you for an excellent episode. Thank you, Nishon. It was great.