ÜBER DIESE EPISODE
According to Gartner, at least 30% of generative AI projects will be abandoned after proof of concept due to escalating costs, poor data quality, or unclear business value.
There are a lot of factors working against them, and too often failure happens when teams are right in the middle of building. They’ve spent significant time, money, IT capacity on something that may very well have been doomed from the start. So how do you know if something is actually worth building with AI or if you should just focus on buying instead?
Riley Rogers: Hi, and welcome to the Win/Win Podcast. I’m your host, Riley Rogers. Join us as we dive into changing trends in the workplace and how to navigate them successfully.
Here to discuss this topic is Keith Weaver, Senior Director of Global IT and Enterprise Applications at Highspot. Thanks so much for joining us today, Keith. I’d love if you could just kick us off by telling us a little bit about yourself, your background, and your role.
Keith Weaver: I have been leading our global IT team at Highspot. I’ve been in IT for, I think, getting close to 20 years now, just overseeing all kinds of different systems and specialties within those systems. Our IT team at Highspot is IT operations, making sure everyone has devices, equipment, all of that globally. We’re now in seven countries.
As well as system engineering, making sure everyone has the right tools, that they’re optimized for the business. And then we have specific teams in IT that specialize in financial systems and HR systems. In the world of AI, it’s a whole new world and it’s a lot of fun.
RR: So 20 years in IT, you’ve kind of run the gamut. But it sounds like the last two and a half years have crammed almost 20 years’ worth of change into a very short window. When we were planning for this episode, you mentioned to me that you read four or five newsletters every morning just to keep up with the pace of innovation.
Can you walk us through why you’re spending so much time learning and how IT has had to evolve to keep up with this pace of development?
KW: What I’ve seen is mostly it’s the same things that we’ve done, except it’s done at a much faster pace. It is kind of an unheard of pace that AI is moving. Every day, the reason I read five different newsletters is because there are so many new things getting released, whether it’s a new model or a new tool, and then there are things within the tools that are getting released.
Trying to understand all those things, the new protocols that are getting released, the new ways people are building or executing on AI, or the new things people are learning, is just really critical. And all of that is at a speed that we’re just not accustomed to. So I think that the objective of what an IT team does to continue to keep the business unlocked, able to do their best work with the technology and tools while making it secure and scalable, those things are the same.
It’s just the speed now that it’s running, and the speed from the business. The business is moving so fast, and their expectation is that IT continues to deliver at that pace. I’ll give you some examples. I get probably, me personally, not my team, but me personally, I probably get 20 to 50 requests for new connections, new skills, new tools every week.
That’s in Slack, in Jira. It’s just nonstop, and that’s just unheard of before AI. It just didn’t happen. It would be like, “Oh, we have this big initiative, and we’re gonna do this thing.” And now it’s just like, “Can I have this? Can I have that?” And all good things, it’s just how do we do that, and how do we keep up?
RR: So when you have 20 to 50 requests hitting your desk every week, it can probably start to feel a little overwhelming. And I wonder if the folks who are sending these requests your way maybe don’t know the considerations behind every single one of these ideas.
I’d love if you could give us a view of what you’re seeing across the industry as both technical and non-technical teams are all noodling on the idea of how to build with AI, how to buy AI tools, how to bring them all together and create a combination that actually works. What are you seeing?
KW: Well, I think if you think back a year to a year and a half ago, mostly people were buying tools. Building was… It’s so funny ’cause we’re talking about a year ago or a year and a half ago, but building was much harder. And really with Claude, the Opus class models that came out, it really started unlocking the ability for everyone to start building.
And so since then, the conversation shifted from, “Yeah, we can go buy this tool,” to “We can build this internally.” And there’s a lot of excitement to build because building’s fun. The pendulum just swings often on these things, and so we’re kind of in that wave where everyone’s super excited about building something.
And I think we’ll end up in a place that is more normalized, where we’re building some, we’re buying some. We’re doing a hybrid approach. I think that is where we’ll end up. It’s just, it has swung really far to, “Look, we can just go build that thing.”
RR: So it sounds like we’re kind of in the early phase of a cycle that you’ve seen time and time again, where surrounded by enthusiasm, but maybe we don’t have much direction. And like you said, building is fun, so I imagine you probably hear a lot of, “Hey, look what I built,” or, “Why can’t we just build this?”
So when something like that lands in your inbox, what’s the first thing that goes through your mind?
KW: If it’s, “Look what I built,” there’s lots of questions, like how is it being deployed? What’s the next step with this? Who’s gonna own it? How is this actually out in the world, and how are we gonna make sure that it’s as scalable and secure?
If it’s a conversation, I would say this is more so where we hear, “We want to build this thing,” or, “We can just go build this thing. Why would we buy this?” I think the thing that I would ask is, what is the long-term path for this? Because deploying is just the first step. It’s the iteration. It’s the test. It’s continuing to deploy it. It’s maintaining it. It’s supporting it. It’s the full life cycle of any software project.
What resources are gonna do that? How are we going to maintain this and support this long term? Those are usually the questions that I start asking.
RR: And you’re quite equipped to speak to the build side of the equation because you’ve done it. You’ve tackled the planning, the deployment, the maintenance, the ongoing support, the full life cycle, like you said, of a software project. And you’ve done it with AI.
I’d love if you could walk us through your first test case. Tell us a little bit about what you were thinking going in, and what you learned along the way about what works, what doesn’t, and what was surprisingly difficult.
KW: Yeah. So we built an IT AI agent and an HR agent, and we did that with a cross-collaborative team. I think we started it last fall and then deployed it in February or March this year. And I would say the main objective there was not that these are the tools that are gonna change the way we function. The main objective was a learning. It was an R&D investment.
So we knew that we had to, as an IT team, be able to help the business deploy these agents, and we found some use cases that we could. The IT one we can own in-house. The HR one, we already collaborate with them very closely on a number of things. So it felt like a good first step for us to dip our toes, learn a lot about how to build these, what works, what doesn’t. And these agents are more than just chatbots. They really were taking actions in tools like Jira and Workday, Highspot itself.
And we built them using the Workato platform, and what we learned through it is it was actually really exciting and really easy to get to a place where this mostly works, and this is pretty, it feels like this could be really powerful. That was, like, the first three weeks. The rest of the time was, “Oh, gosh, how are we going to get this thing so that it doesn’t give somebody some really poor answer from a security standpoint or an HR answer that needs to be rooted in truth?”
And then how can we make sure that it repeatedly does these actions no matter what people prompt, that it’s really controlled? I often described it to my team like it was training a four-year-old. I have four kids and they’re all older now, but it was like I was back in those toddler stages. Like, “Why’d you decide to do that? That was a bad decision. Stop.”
And trying to train it and get it to that point, it just took a massive amount of time. And since we’ve launched it, it’s a lot of resources to read the conversation, understand what was asked, understand what it did and where it went wrong, then refining it, then retesting it, then redeploying it. It’s a cycle. It’s this continual cycle, and that takes a lot more resources and time than I originally thought it would.
RR: Ballpark, what did that investment look like? How long did it take you to build? How many queries did you have to manually validate before you trusted the thing? And how much are you still investing today to keep it alive?
KW: In the beginning, you just start giving it manual prompts and testing it to see what it does. Then you produce your test cases, and you manually go through those test cases, and every time you make some major changes, you go through those test cases, make sure that it’s still hitting. Because just changing one little thing, for example, if you take a tool and change the prompt or the description of that tool, you actually have to do regression testing on all the things that someone could prompt, because just changing that one little description, the agentic reasoning could choose to use that tool or not use that tool at the wrong places.
So it’s really important to do this regression testing. Then we started going through how do we automate some of this testing, and we were able to do some of that. But I think the biggest takeaway is with software, traditional software, you do regression testing, but if you change this bit of code, really that’s what you have to worry about, anything within that bit of code, not the whole stack, not the whole thing.
This was, I think, a really big takeaway. We would make a small change, and something unrelated, completely unrelated, would start acting weird. And then you have model changes. You’re like, “Oh, I want to move to the latest model.” Well, that latest model now, the reasoning’s different, how it picks up tools might be different, and so that also has been a challenge.
I think over time, especially as the tools evolve, the automated testing will get easier. We can have Claude, for example, go through and give it a bunch of questions, evaluate the responses. But you’ve got to build that too. That’s all something you’ve got to build in order to maintain this thing.
RR: Now, you said this project was conceived last fall, launched, not necessarily finalized, ’cause it’s never really finalized, but launched publicly in the February March timeframe. So now we’re almost a year out from initial conception, couple months out from launch.
Given your point about things changing so much in the last year and a half, now knowing what the landscape looks like, would you have built differently or would you have built at all?
KW: It’s a great question. I think if I go back to my objective that I needed to learn and I needed my team to learn how to deploy these things and the challenges of it, I wouldn’t change what we’ve done, and they’re still deployed in production right now, although we are not spending a massive amount of time making them better.
Would I change what I’ve done? No, but that’s because of where my objective was. My objective wasn’t to massively change the HR process or the IT process. It was to learn. And I think that given the right investment, we could move them further along.
However, I do not believe that building something that every business has to do, IT and HR stuff, that… all those questions that we get, both of those teams get, are the same across most organizations. Most of the things employees need to do in these functions are the same across all organizations. So I don’t know that I would double down and want to go build the agentic functionality behind IT and HR.
What I would want to do is look at our specific processes, things that are unique about our business, not only where we have the information, but what tools we use, and then maybe specific processes of how we do something, and figure out how do I build that piece and tie it into something that’s delivered.
And there are so many, just talking about the IT space, there are so many systems right now that have the agentic functionality and give you the ability to tie in all your different tools. So I would probably try to leverage those rather than just building everything from the ground up.
RR: As an aside, I will say I have used both the IT and the HR agents. They’re quite cool. The HR one talked me through my vision insurance at 8:00 PM on a Saturday, so thank you for that. Very helpful.
So you mentioned that this was something that you and the team invested in as a learning, so you could understand what the process of building an agentic workflow looked like, an agentic tool looked like. So having now seen what it looks like, what must be invested in order to do this successfully, how do you draw the line between what you can build, what you should build, and what you’re better off just buying?
KW: I think you should build the things that are unique to your business. You could think about this as your moat, but I think it goes beyond that. It’s where your processes or the way you handle something is very unique. In those cases, it may make sense to build something from scratch.
But if it’s not unique to your business, like go back to the HR and IT, most of it’s just not unique. And so in most cases, when you don’t have something unique, then it likely makes sense to buy it, because you get the power of the organization that has built that tool behind it, pushing that tool forward much faster than you’re ever going to push it forward, ’cause we’re just not gonna hire that many engineers to do it.
And so if there’s an IT tool out there that is bringing all that agentic functionality to the forefront, then why would I not leverage that, as long as I believe in the company, they have a track record to deliver, and they have the ability to build on top of it for some unique IT process that I have or some deterministic termination workflow that I want to execute. If I can do that, then I would say let me build what’s unique and tie it into something where the base or the product is being delivered.
RR: Outside of that uniqueness component, if you’re a GTM leader trying to make that call, what other questions would you be thinking about and asking yourself?
KW: I think I would just go back to that uniqueness test. Is this unique to my business? Is this a unique sales motion, a unique thing that we do here because of what we’re selling or how we’re selling that I can’t expect a vendor to build this or be able to deliver it? Or is it roughly the same thing everyone else is doing, the same problem everyone else is trying to solve?
There are unique components, ’cause there are always unique components to this, but those unique components can be built alongside or included within that tool. And here’s the wonderful thing, if you just take AI or Claude, for example, we’re using Claude internally, you have a tool that has all the MCP capabilities like Highspot, like Salesforce, and you have some unique process with some maybe even proprietary tool that you want to do.
Well, the great thing is you can bring that all into Claude. You can use Workato or build a skill within Claude and use that, whatever you’re building for that unique process, that unique system, and tie it in with those other tools that already are giving you a lot of functionality, and now you’ve expanded what you already had, and you’re spending your time in that uniqueness layer.
What’s unique to us? That, to me, there’s value in that, to build there, because you’re not gonna get another organization to spend the time, money, or energy building on that.
RR: So it sounds like it kind of comes down to a basic cost benefit call. Is the upfront investment and ongoing maintenance worth what that unique build will actually deliver?
KW: When you’re doing the cost analysis, you can’t look at just what it costs to build. You have to look at the full life cycle. An analogy for you is, if you go back, maybe 20 years when SaaS really became a big thing, you had Salesforce kind of leading the charge. In all of the sales cycles, they would say, “You no longer need servers, and you no longer need admins.”
The business manages this themselves, and everyone was super excited. Oh, this is gonna, the ROI’s gonna be so great on this. The business is gonna own this. We can move really quickly because we can change workflows and do this stuff and own it all internally. We don’t need an IT team to manage servers, et cetera.
And everyone got really excited about this, and I remember all those years ago being like, “Guys, I don’t know if that’s really gonna be true.” So now you fast-forward, and it hasn’t been 20 years. For the last 15 years, we’ve been actually hiring system admins like crazy. We have Salesforce admins, NetSuite admins, Workday admins, Jira admins, and we are spending millions of dollars on these admins to maintain these systems that were supposed to have a great ROI.
And I haven’t seen any studies on it, but I bet if we could compare on-prem with old software that we used to buy and put on-prem to SaaS, we would say that it was a lot cheaper to do it. And we’re not gonna go back. We’re not gonna do that any differently anymore, but I think we have to use that same type of lens. The ROI is not just the build. It is how are we gonna maintain this and keep it long term.
RR: Yeah, and that’s the pendulum swing you mentioned earlier. Excitement, then reality hits, then we’re back to the next shiny thing. For those in that stage of enthusiasm that maybe don’t know what they’re getting themselves into, what do you wish they understood before kicking off a build or approaching IT with an idea?
KW: I think the full cost. That you can’t do the math by just looking at what it costs to build, or that I can get a couple of my key people to go build this thing in Claude, but it’s how are we going to scale this in the organization? How are we going to deploy this thing? How are we going to maintain it? How are we going to support it?
Who’s gonna take the support questions at midnight when they come in? And then how are we gonna iterate to keep ahead and make sure that it can do everything that we need it to do and stays ahead of where the business is going?
RR: And when you’re having those conversations, at what point should IT actually be in the room? And what happens when you’re not?
KW: I think that IT should be brought in the room as soon as a team is saying, “We can build this instead of buying it.” And really, I think that IT is not there, or hopefully we’re not there to say no. We’re there to help shed a light on what it’s actually going to take long term so that we make the best decisions.
Because ultimately, I think we understand the technology and we understand the challenges of deploying stuff like this and then maintaining it long term. That’s what we do. And so if we’re in the room earlier, we can at least shine a light on it and give kind of that picture of reality so that we’re making the best decisions and there’s no surprises.
Because the worst thing is that decision to build is made, they start building it, they realize they need IT’s help, and IT has a roadmap. We don’t have the resources or the ability to step in and solve this problem. Now you’ve wasted money.
RR: Yeah. I saw a really interesting stat from some original research by Influ2. They surveyed marketing and sales leaders that are in the market for technology, and of those leaders, only 10% of them saw the IT perspective as valuable during the evaluation. 38% of them, however, then came back and said our biggest blocker, the number one blocker, in fact, was IT. So that disconnect is very clear, and it’s causing very clear consequences down the line.
KW: I think what I would say is the best IT teams don’t want to be blockers. We don’t get excited blocking what the business is doing. We get excited leveraging the technology and moving it forward as fast as possible and seeing the business successful. So that’s what we want to do.
It’s really hard when decisions have been made that back us into a corner and we’re like, “There’s no way to scale this, support this, continue this.” And then we’re seen as a blocker, but if we just had the conversation earlier, we could actually mitigate that much earlier.
RR: So it sounds like the theme here is that your IT team wants to be a partner, not a gatekeeper or a blocker. And so knowing that, bring them along for the ride. Give them a seat at the table. Your programs, your investments will be better for it.
If you had to boil this all down to one tip for marketing, sales, or enablement leaders on partnering with IT as they bring AI into their tech stack, what would it be?
KW: I think if you can bring the IT team in and have a forward-looking view over the next year, over the next two years. It’s really hard to look two years out with AI and how fast it’s moving, but over the next year maybe, this is where we want to get. This is what we want to automate. This is what we need to fix. This is what we need to deliver in order to continue to execute on our objectives.
If IT is on the same page there, we can be a proactive resource and help to get them there. We read all these things every day, there might be something that I’m seeing in the market or happening that I can bring to the table. Or at very least, when you come to me in six months and say, “Okay, I pulled the trigger on this thing,” we had that conversation. I knew it was coming.
I knew the direction we were going, and directionally I was aligned with you, and so now I’m supporting that initiative rather than it being a surprise with no resources. So let us help with the roadmap. I think if we could do that across the whole business with IT, we would win together.
RR: The one pretty clear takeaway, I think, whether you build, whether you buy, or whether you land somewhere in between, bring your IT team into the conversation, and you’re set up either way.
Keith, last question for you. Just curious, what would you say to someone who looks at this build, buy, blend conversation and asks, “Why don’t I just build Highspot myself?”
KW: When you build something as big and as complex as Highspot, there’s a lot of learnings that happen, a lot of iterations, a lot of changes to get to where you are. And trying to build that from scratch is very difficult. And I think the next thing that you may hear is, “Well, we can cobble this together.”
I would just say you’re going to maintain that thing that Highspot has a full engineering team to make that as wonderful and as best as it can be. And how many engineers can you staff to do that and keep the velocity of it so that whatever you’re building is gonna move faster and be better than what Highspot’s building? And then what’s that gonna cost you? Going back to that build versus buy, when you start looking at how many engineers it takes not only to build it, but then to maintain it, support it, deploy it, I think the ROI doesn’t pencil out.
RR: The moment you throw out the phrase cobble together, I think you’ve kind of already answered your own question. Keith, thanks for the window into a world most of us in go-to-market rarely see. This was a lot of fun, and I’m really excited to share the news. IT is there to help, and when you give them a seat at the table, your AI investments will be better for it.
KW: Yeah. Thank you so much. I appreciate the time.
RR: To our audience, thanks for listening to this episode of the Win/Win Podcast. Tune in next time for more insights on how to maximize go-to-market success with Highspot.
