Full Transcript for Hard-Earned Lessons from AI's Front Lines, with Melissa Tondi

don't think that went particularly well. So maybe you can share that story and then kind of tie that back to this idea of we really need to actually think a little bit more critically about how this technology is gonna, you know, where the rubber meets the road inside of our businesses and how do we measure and deliver, you know, improved results. Melissa: Yeah. it I I also kind of highlight that as one of the biggest mistakes that I wound up learning a lot from. so yeah, as as we talked a couple of weeks ago on this one, you know, when when we were first starting our journey, probably close to now three years ago, when we when we put investment into at that time what was called an AI enablement function, and then kind of evolved into kind of an AI operations model as we realized that the operating model really was what we wanted to. focus on but you know there were there were lots and lots of information flowing through from leak from leadership to individual organization leaders like myself around ai And there was some back and forth on whether we should have one or two metrics that applied to every single person at the company. And at the time, when we were talking through that as an option, it made sense to us. But of course, we had not done this before. and so the the the per the the actual metric that we had used was a percentage of employees using AI weekly, and then that turned into you know some sort of a time slash cost savings, and and at that time it was about eight hours a week, right? If you think about a typical forty hour work week, that is you know about a twenty percent ish savings or so, right? So it sounded really great in theory and with the leadership and at that time we were aligned But what wound up happening was that we were seeing that folks really weren't bought into that. And that measurement wound up really looking at activity versus value. Jason Griffing: Mm. Melissa: so people were able to game that, or they were thinking about gaming it. And and in in some cases we had active conversations around the the table around how folks could game it and whether or not that that in and of itself should be criteria when we're thinking about what kind of measurements or metrics that we should be putting in place. So you know, the the other when we applied that percentage of employees using AI metric across the board, what we saw was that it actually hides that one team or that those few teams that may have already been transformed while another team is really just generating noise with that. Jason Griffing: Mm. Melissa: you know, w we and and as that kind of Use the term lovingly, but blew up really in our in our faces, we realized that the better way to be looking at that was to work with individual organizations or departments and very much aligning with leaders. So, you know, going back to the original metric, that was really kind of defined by the senior leadership team, our C-level engineer or leadership, rather than individual organization, department, or team leader. So we kind of took that down to the next level and said, hey, Jason Griffing: Mm-hmm. Melissa: we we want to hear from all of the organizations here. And if we aligned with those levels of leaders, we'd have a much better success rate of defining valuable metrics and measurements based off of what their work actually was defined as, right? You think about support teams versus engineering teams versus a finance team versus an H. or people team, how they define better was very different than how even amongst those those organizations. So we realized pretty quickly that In instead of giving each of those departments a metric, it was much easier to onboard and get to that high adoption rate quicker when every leader within those departments and organizations, plus either their entire team, depending on how big or small their teams was, or representation amongst those teams. When they owned their outcome metric, they were it was much quicker to see success and we were able to Jason Griffing: Yeah. Melissa: get that. AI adoption rate up much higher in a short amount of time by simply coming down one or two levels from the original metric that was defined at our senior leadership level. Jason Griffing: Yeah. Yeah. I I I appreciate you like openly sharing that that failure mode and that trap that you fell into. Cause I, you know, I hear about this a lot from I work at a fairly small company. I hear about this more from friends of mine and so forth that work at larger companies where there's this idea that we're gonna push AI out there and we're gonna measure everybody on usage and trust that that usage is going to produce better results for the business. And like as a heavy AI user, I can empathize. with where that comes from, because I have personally experienced the value in terms of not just efficiency, but the quality of the work I can put out and so forth, all going up dramatically as a as a result of AI. So I like, I can completely understand where the seed of that idea comes from, that like we're just going to sort of measure on usage and then trust that the results for the business are going to come out of that. But I think even putting aside the real possibility that people can game the system, there's all these conversations about token maxing out there, these really large companies like Meta and stuff where people are just building agentic loops to burn tokens so that they can climb the leaderboard and look like the heaviest AI user. But even putting, you know, the gaming aside, the gaming of the system aside, there is just the reality that a well-intentioned person who is trying to do more and do better for the company. could be in there just burning tokens to no good end. Like it's what they're doing is not actually helping the business. So I like the idea of kind of coming at it from the other direction and saying, hey, let's first define with each department what is the business result

that we want to see improve. And then to your point, I would imagine that there was a lot more buy-in from each department with that sort of rollout. Can you talk about how how did the team, when you took that initial approach, You talked about the fact that it it just fundamentally didn't work, but how was it received by the team? Did you guys get pushback? Were people upset by that? I can imagine they probably felt like it was sort of being foisted upon them. Share a little bit about what you were able to discern about the experience of those on the receiving end of that policy. Melissa: Yeah. well first of all, you know, the the message coming from at that time it was me that was then kind of communicating what the agreement was that we were gonna measure a percentage of employees' time and then with a with a goal of eight hours per week per person. so you know the the people who had never been in a meeting with me or never probably met me or even knew who I was, now all of a sudden they're getting a a mandated message from somebody that they had no Working relationship with so forget the trust aspect of that. There was not even a relationship or or built there, right? So you know, then there were also people, you know, people of course heard that metric and said, well, okay, I have a department of of five people. so if our outcome or goal is to save eight hours a week per person and times five people, well, that sounds like one person's just job. That sounds like one person's 40 hour ish work week. So is this ultimately going to be replacing me as a person, as an employee or as a contractor at this company, right? So, you know, we what what What we knew would happen after we received lots of feedback. And by the way, the company where we first rolled this out had a lot of really great feedback loops. So we ate we were able to hear this feedback very quickly, almost within minutes of hearing the initial message coming from me, right? So that really caused a pause for me to say, hey, there's likely a better way to be doing this. You know, things that we were concerned about is that, sure, we would get compliance, but not really the adoption that we were seeking, let alone the value, right? I mean the value must be at the top of the the the the measurable outcome and people who are being asked or in some some cases required to comply if they are not able to directly correlate to the value, it will be a losing battle all the time. So you know we were concerned about the compliance without the adoption. you know, people may just open the tool to satisfy the dashboard and then return to the old workflow. So we weren't really seeing a lot of the value there. Jason Griffing: Mm-hmm. Melissa: you know, and in some cases it There were cases where it was driving shadow AI. People were kind of quietly using their own tools because they had a preference or they liked them better or they were told that it would do something for their job that we had no understanding of. And that would that quickly became a governance and data risk. But really Jason Griffing: Right. Melissa: the the deeper problem was fear. A mandate sounded like a replacement to a lot of folks. At that point, the trust dropped significantly. And of course, without trust, you never learn where AI actually works. So, Jason Griffing: Mm. Melissa: you know, again, we were able to quickly pivot, but not without a lot of kind of then reputation building, right? We had to fix the the trust erosion and fix some of the damage that that messaging had Jason Griffing: Yeah. Melissa: with some folks and quickly be able to deliver that trust, which again, which is why we've Pivoted than to working with individual leaders and either all or some representation of teams at that level. Jason Griffing: Yeah, got it. I think this this fear of job displacement is something I would love to zoom in on a little bit because it's very real and very understandable. And I think we all have to be honest in this dialogue that like, yes, there are some jobs, some roles out there that are going to be impacted by AI. And some people are going to lose their jobs. And that is not something I think that any of us, especially those of us who are trying to be thoughtful. about AI, like relish in that. But fundamentally putting that aside, I and I I would love to know if you share this perspective. As somebody who's been using AI very heavily since ChatGBT came out, I just don't see it in most roles. I do not see AI as a replacement for human judgment, human intuition. And there are still very pragmatic and practical reasons why humans need to be in the loop, being accountable for the output, looking for errors, you know, these sorts of things. I'm curious to hear your perspective on like how I'm I'm gonna make an assumption that you guys weren't out there trying to mass, like displace a bunch of workers, and that really this was about trying to put the tools in the workers' hands so that they could do better and do more. how did you dispel that fear? Did you find that like just going down to the individual department level and working with those leaders to establish the goals? Was itself enough to make people feel a little bit more comfortable? Or did you have to come more directly at that fear of like, hey, why should I be aggressively pushing AI within my department if the goal is just to replace me? Like what's my incentive? How did you guys how have you n found like navigating that dynamic within companies? Melissa: Yeah, I mean we definitely heard and it was surprising because we heard from folks within the the technology organization, which, you know, I think a s an assumption, of course, would be that technologists would be the early embracers and adopters of any technology, including and you know, AI or AI agnostic, right? Any kind of technology technological disruptor, you would assume that the folks who are closest to the technology would be the folks

that were the most curious. and and and kind of getting in front of it. That was not the case in with a few folks within the technology organization because they were gen they genuinely saw or heard or you know read all of the case studies around Some of these you know, models were capable of doing, right? And so if your main job was to be writing and deploying code, and then all of a sudden you could see a tool like Anthropic come in and and completely do something that would take you hours, if not days or weeks within minutes, right? That is a little scary. So What we wound up doing was, and again, going back to, you know, I I and I I keep saying this is one of the best mistakes that we made, about that first original metric, was that we we t we turned the conversation around focusing with the leaders of the organizations and departments first and and had them create their own backlog of work either with their team or or or that's just knowing if they had already had work that geez if we had a little bit of time what would we like to do better for our job and then really come to the table with that and then also have a conversation to say look what are some things that you just never have time to do? And let's work on figuring out a way to get through that backlog of work that we quote unquote never have time to do and see where AI or other tooling. There could be other solutions in the company that we could apply that were not AI-based, but let's focus it on topics and items that had already been identified by the team that they did not have enough time to do, or there was some other challenge to be able to do it. And let's see collectively where we can apply AI or other process improvements or other tooling to get through that list. And that way it was more like, hey, if you had more time in your day to be able to get through this list, wouldn't that feel better than where you are constantly? Are you needing to work long hours during seasonal? you know, seasonal activities that happen there and wouldn't it be nice if you had a little bit more schedule or repeatable, consistent time for you to do your work, Jason Griffing: Mm-hmm. Melissa: plus all the other I I areas that you've identified. So focusing it on tasks that were related to their job and not some just a very simple kind of eight hours per person per week, I think really helped dispel some of the fear that we were hearing from individuals. Now there were some individuals that were not only were they not going to be adopting, let alone And embracing, but they were actively against it and very commute highly communicative in that. And some of these folks were people who were highly respected in the company because they had been there for a long time or were subject matter experts in the industry. And if they said something that was negative or against adopting AI, a lot of people would listen to them. So we identified those folks and really had then focused our kind of one-on-one efforts on really. Understanding where someone who could be a potential ambassador, but now is a detractor, how could we engage with them so that now they became that ambassador and representative and now then advocate for what we were doing? that took Jason Griffing: Mm-hmm. Melissa: a lot longer than I think we thought, but it was so well worth the investment in having those one on one conversations and really getting them not, I hate to use the term getting. them on board or having them buy in because it really wasn't about that. It was more about just giving them the space to communicate their fear and their mistrust and hearing them tell me how my initial rollout affected them, giving them that space where they could Communicate and share their concerns, and then making sure we did not leave that forum or that meeting without having an outcome that we both agreed with, with them coming to the table with a solution or at least giving them options that they could really talk about and embrace afterwards. Jason Griffing: That's interesting. And were you able to successfully navigate it with all of these detractors? Were there any folks that just ultimately like couldn't get on board? I'm really curious because this is such an important topic, especially when you're talking about people that are senior within the organization. If they're just vocally and adamantly against the rollout of AI, that can be a really big hindrance for any company that's looking to adopt this technology. So I'm curious to hear just a little bit more on like How did those conversations ultimately get resolved? And were there like particularly sticky cases that you guys had to navigate? Melissa: Yeah, I would say we had probably about twenty percent of the company at that time who were active detractors of of AI. and once those folks were either identified by their leaders or they had identified themselves by how broadly and widely they were communicating their you know, their stance or opinion on it. We we got that twenty percent of folks down to probably less than five percent, to be Jason Griffing: Mm-hmm. Melissa: honest. And that was probably around a month of kind of us really focusing on and making sure we scheduled time and had right forums and all that. And then once we got got the folks down to less than five percent of folks, quite honestly, it really then became a manager to person kind of then relationship. We Jason Griffing: Yeah. Melissa: handed that off and said, Hey, you know, and then We we tied some of the performance evaluation. We we at that time did twice yearly performance evaluations, and then we tied scorecards or OKRs to their actual AI adoption there. And so then it became more like this is how your role in itself is changing. And then that really became a manager to employee kind of conversation there. But we we focused on getting that. number of detractors down significantly so that then we could really deal with them as more one-offs rather

than a mass Jason Griffing: Yeah. Melissa: or a larger percentage that that really did have the the ears and the eyes of the rest of the company. And then for the the the the you know 15 or so percent of of folks who were detractors and who were now on the bus so to speak, we really focused on figuring out whether it they had a desire then to become actual AI ambassadors. And that was a really interesting Turnaround because of those 15% or folks that you know became adopters of AI, a decent amount of those folks wound up becoming ambassadors for their team. So it was almost like a 180 from, hey, Jason Griffing: Hm. Melissa: I'm against it and now I am for it. And it really was back down to again that human inner interaction where we gave them the space and the time to give their opinions. And some of those opinions we were able to quickly Dispel just by using data. Some of them were opinions that nobody could dispel because that was a human feeling. But giving folks who were those highly respected folks and had that the eyes and the ears of the company more than some of the other folks, newer folks in the company, like myself, was again really kind of that investment that we weren't sure how it was going to turn out, but it wound up turning out successfully. Jason Griffing: Got it. And so, and so really fundamentally it was sounds like about just sitting down, actually hearing, hearing them out, what were their concerns. A lot of them, it sounds like were probably rooted in things that weren't actually the reality of what you were trying to do as a company, which I I can understand. Like as somebody who would sit on the receiving end of it with AI, there's it's such a big, broad and impactful topic that it's easy to look at it and if all you're getting is sort of a company memo about here's what we're doing, then it's it would be very easy for your brain to spin off into all sorts of fearful and negative narratives. So it sounds like at its core, this is about really just leaders and managers within organizations being willing to go out there and do the very real work of sitting down with your teams, talking to them about AI and making sure that you're on the same page about what the goal even is, because it sounds like from your experience, That if you can just make it clear what the actual goals are, that in a lot of cases you're gonna get rid of like 90% or more of that fear and uncertainty that's got, you know, people scared about moving forward with this technology. So that's that's a great lesson there. You've mentioned the term ambassador now a couple of times. I'd love to go deeper into that. What is the role of an ambassador when you're rolling out AI within an organization? What does that actually look like in real practical terms, what are some of the ways that you've seen ambassadors work successfully to help companies with this transformation? Melissa: Yeah, so an AI ambassador was really we were We were first looking to solve the problem of how do we quickly get information that is actively being talked about, collaborated on, or even solutions being provided out to the masses and have somebody who could speak the language of the different organizations, departments, teams from there, and literally in that in that kind of hierarchy, starting with organizational leadership, right? So at that time we had six individual organizations. Within the company. And we started with kind of working with the leaders of those organizations and asking them, hey, is there anybody right now that you have identified or has identified themselves as being curious, either AI curious or an AI user right now? And if so, we would love to talk with them. And so we rolled out the ambassador program by saying an ambassador anywhere from two to four hours per week. And the idea is that you would come to our committee meetings with, you know, we had twice-weekly committee meetings where the first half of those meetings was talking about what all of the folks were doing from organization to organization. And then with a very deliberate communication plan of okay, here's now what you're responsible, each ambassador to sharing out to your team in whichever forum comes next between our committee meetings. Meetings. And so we established that not only were they coming into these committee meetings as being able to work through and help collaborate and define our AI enablement and operations practice, but also that there was some responsibility for them to share the same messaging out to their teams. And so that was a really interesting one because we were probably a little bit prescriptive or what felt like prescriptive on what we were actually. Asking the ambassadors to share, but that was deliberate because we wanted to ensure that the messages were not going to be degraded from the game of telephone coming from one person to another, right? And so we were very prescriptive, and here is the information that each ambassador will share with their teamslash leadership between now and then. And so we made sure that that was a responsibility and that folks were doing that and had that. That quite honestly helped a little bit with the trust that had eroded with me in particular my original messaging coming out because now they're hearing from somebody that they worked with on the day-to-day. And we knew that that ambassador was now an an embracer and certainly an adopter of AI and understood its power and its value in the right hands and with the right governance that it could do, not only for their role, but having then them being that representative for other roles within their organization. So that's Jason Griffing: Mm-hmm. Melissa: essentially what the ambassador program looked like and how it was built within there. And Jason Griffing: Yeah. Melissa: we really weren't looking for folks who were technical. It was easy to find the technical folks within technology because again, of course, we were all

technologists, but we were really looking for the curious and sometimes those respected peers, right? The folks that just were natural leaders within their organization, whether they had the title or the role or not. And and somebody you know, w we wound up creating this ambassador program by people who in some cases were just kind of quietly experimenting on their own. And so by working with their managers or leaders there for them to identify at first, and if they didn't identify anybody, then kind of digging a little bit deeper. That's really the the the the best skill sets of ambassadors for that program were the the curious and the people who naturally experiment on their own without being told to do something and those became the the foundation of what an ambassador looked like for our AI program. Jason Griffing: I love that. And the biggest thing that it makes me think about is I think with this new technology, it's so powerful and so transformative that as leaders and managers within our organizations, it can be easy to get lost in all of that and forget that fundamentally, if we want to roll out AI within our organizations, we can rely on proven, simple, battle-tested change management frameworks. So create Melissa: Exactly. Jason Griffing: feedback loops, right? This ambassador program is about bringing in Identifying the right people, bringing them in. Obviously, in a larger organization, you're not gonna have time to go talk to everybody. So you have to f you know, find the right people, bring them into this feedback loop mechanism that you're creating so that information can get out to the right teams and then even more critically back into the leadership group that's f making these decisions and pushing this technology out. And And again, I think it's just easy, unfortunately with AI to get so wrapped up in the excitement about the technology that we forget to just go back to basics and, you know, focus on proven change management frameworks. Melissa: Yeah, and I wanna I wanna call out something that you mentioned earlier about kind of like this company-wide memo of, you know, again coming from me. but that peer credibility does so much more than any company memo or company communication coming down, right? Especially when there is no relationship. The larger the company, probably the more levels between the person who is sending out that communication. And they likely will have never worked with individual contributors in the company, right? So Wanted to make sure that that was something there. And the the other thing that we really focused on, because language is so important, whether it's natural language that of course that the the tooling itself are is working on, or just our language from person to person. A team's you know, we could turn central tooling, which of course we were that was our mandate, was to have a a centralized tool set that then folks in the company could use. Turning centralized or central tooling. Into their team's language was also a major benefit from what that ambassador was doing. I could never speak the language of somebody in finance because that is not my that's not my credibility or my experience. But we had somebody who was from our finance organization who knew exactly the right language in order to interpret the communication that was coming out from those committees back into their team and they were. able to Jason Griffing: Right. Melissa: really translate and represent that language much better than somebody who, you know, was not part of that role or that or that team to begin with. Jason Griffing: Yeah. Yeah, they they could really, I'm sure, boost like the relevance of that messaging and really dial it into the things that they know that their team is really gonna care about or need to understand. So that's a really good point. You mentioned the term governance there a couple of times and I'd love to zoom into that. Governance is an idea I think we can all sort of infer what governance might look like within an organization, but we're all figuring this out together, right? This is all new and and many of us are out there trying to figure out, okay, what does this actually need to look like in my company? Can you make the concept of AI governance a little more concrete for us with some specific examples of what what does a governance document or program look like in your experience within an organization? Melissa: Yeah, this was this was the flagship I think w of of what we were doing here and how important it was. you know, as as I as I I do a couple talks on this and have written a couple articles on this, but we tend to kind of staple on or or tack on governance after the fact. and what we learned very quickly is that really we needed to to lead with the governance. So governance in our organization and actually not in not just in our organization, but for for me personally now for the you know dozen or so companies that I've worked with over the last three and a half years around this governance looks like approved tools. What can you and should you be using and more importantly what is the company providing to you f via a cost there, right? And so you know early on we had we we were also using Chat GPT, OpenAI, and We realized that there were some folks that were s and we offered enterprise licenses across the entire company. And we realized that there were still some people that were using their own personal Chat GPT licenses, right? And so we were like, it's not just enough to say we use Chat GPT, but we actually use this instance of Chat GPT. So we had to put the the policies around. Here are the tools that we approve coming into our ecosphere. So we have approved tools, data classification, specifically what what What may go into which tool? How do we classify that? How do we what what can we sh can we be putting in there and what should we be putting into each of those tools? Then we talk about human review and

specifically human review by risk level. So we had to do a lot of work determining, hey, where are the risky business areas within our company and how much human review do we need or want to have? And what are some of those business processes that need little to no human review, but very clearly calling that out. customer disclosure was another one. And as you and I talked a couple weeks ago about this, we we knew that we we we needed to get the message out there to our customers with our external AI usage policy. And that quickly then defined our internal usage policy as well. So we disclosed, we were very clear with our customers, we will employ AI solutions here. But our promise to you is that you will never wonder or question or not know when you are interacting with any type of AI solution in there. And that was really important for us to make sure that we were very honest and widely communicating. when we were using AI with within our customer-facing solutions. And then, you know, then kind of going into IP and code ownership, vendor evaluation, and regulatory exposure. So all of those aspects really need to be thought about and defined when we talk about governance. And that's what the governance framework looks like at organizations where I've I've brought this in. Jason Griffing: Got it. That's super helpful to put put sort of a framework around that. And again, it makes me go back to that idea of like, you don't have to overthink this. Like fundamentally, if you're a a leader, owner of a company and you're thinking about AI and you have this sort of sort of nervous feeling that like we haven't put up the basic scaffolding and kind of guardrails around this, you know, start with the simple stuff. Like what are the tools that we use? How do we use them? Do our customers engage with AI? How do we close that? Like there are some very basic things that you can start with. And then I I I would imagine, like, correct me if I'm wrong, but that you would probably encourage leaders to start simple. If you're just getting started and you're feeling overwhelmed by this, like get something in place. Don't over-engineer it because you can always add to the governance framework. And that's easier maybe than going back and like saying, we changed this or we removed that. Like you want to give your team a pretty Clear document to work with. So I would imagine starting simple and layering on over time as you come up with more things that you need to protect against is probably the way to go. Would you agree with that or would you disagree? Melissa: Agree with that. And I think again, the ambassadors were really able to reinforce that those norms locally with their teams. Again, speaking the same language, knowing what the folks did on the day-to-day. and you know, we also ensured that we reviewed the policies on a cadence. we s really started out at a monthly cadence. and then as we kind of got a little bit more stable in in that, where not a lot of changes were coming in, we enforced a a quarterly review of both of those policies and our governance. And you know, I think it's it's it's worth saying that governance nobody can follow is really just a documentation of future violations. And that's what we wanted to avoid Jason Griffing: Yeah. Melissa: at all costs. Jason Griffing: That's good. I like that. well, I'd love to shift gears as we're coming down the home stretch here and talk about agentic engineering because I know you you're currently working in an environment where you guys are deploying some of these agentic engineering practices, you know, the whole dark factory concept of of software engineering. I'd love for you to share what that means with listeners if they're not familiar with that term. And before we jump into some of the details of it, give us some context about what you guys are doing. right now to really empower more people within organizations to to be builders and and what does that actually look like in practice? What are you working on right now? Melissa: Yeah, I would say a high, high percentage of our folks within our company are interacting with our framework. And so they are builders of solutions. and that is something quite honestly a beautiful sight to see. I was pleasantly shocked and surprised when I started at this company about four months ago and because it was really all of the things that I'd been talking, speaking, writing about, trying to build and or consult on with other companies and it was there it was in practice. And I and I go back to there was you know, a a very strong framework put in place, an agentic framework put in place. And really that framework ebbs and flows and and meets the needs of both the company and the organizations within those companies, but also the people there. So yeah, I you know, it it's it's almost shocking sometimes when I hear myself say it, but I would say again, high majority, and I'm talking about 90% of the company, if not more, are Actually interacting with the framework and they are delivering and building solutions now and they have that capability of doing so. And that goes with a a very well thought out, defined, but but also flexible framework that can be quickly evaluated and inventoried to ensure that it is meeting the needs of the business and the metrics that we want to see that we've you know noted as valuable. In there. So and then you know, as we the best way that I could I really define an agentic engineer is it's really it's somebody who now becomes a generalist of meaning, maybe they had specialty as a software engineer or as a product manager, but they know enough about the other roles that make software engineering or or delivering and building and delivering software solutions, they know enough about all the other roles where they can truly become the orchestrator. So they, that person, that agentic engineer is building these agents now that are That take care of the special specialty and

specialized skill sets there. And but with always having the opportunity to be able to consult with humans who are specialties in there, right? So we don't just say, okay, Jason Griffing: Mm-hmm. Melissa: you're a now and agentic engineer, go ahead and go there. You know, go and start delivering. And now you can do that in a dark corner and never talk to anybody. So we want to make Jason Griffing: Yeah. Deployed a prod. Yeah. Yeah. Melissa: sure that that human interaction is still there. You Jason Griffing: Yeah. Melissa: know, the humans set the intent, the constraints. Jason Griffing: Yeah. Yeah, and I I'd love to just underline here that you have a deep background in quality, quality assurance, right? And I I'm highlighting that because I think there's a lot of people out there, and I would account myself amongst them, who look at this idea and say, okay, well, within my company, we could now in principle have a whole bunch of people building stuff and and shipping code. that's scary, I think, from a quality assurance standpoint. So It's very it's noteworthy to me that you as somebody with a quality background are embracing this idea of the dark factory again, which is this idea where like the kind of intent and and guardrails and things go in and out the other side you get, you know, a feature and what's happening in the in inside of that that dark factory is all handled by the AI. Talk a little bit more specifically about this framework and even more specifically about how I would imagine components of this framework are around ensuring that quality is adhered to, that certain core architectural tenets within your software environment are maintained, the integrity of the overall architecture. Like how do you guys handle ensuring that the quality and maintainability of the code base is is is, you know, properly respected in this new agentic engineering world that we're entering. Melissa: Yeah, so we you know, w we put a lot of thought into what were some of the b pre-agentic engineering or pre-AI, what were some of the the challenges that we as more traditional software engineering folks within a a a typical software development lifecycle had faced day-to-day, right? And one of the biggest ones is testing, either not enough or too much, that would then kind of come in and compromise schedules and all of that, right? So, in addition to that, you know, there were just not a lot of coding standards or hygiene that were enforced from the top down, right? Some individuals would enforce, they would have their own kind of metrics around code or test coverage on that, but it was never really something that you could go back to a central leader and say, what is the standard for code coverage, right? And and it was really difficult to for one leader to really answer that question because it really wasn't enforced. And so we had an opportunity to build this framework where it was, you know, it runs on a shared infrastructure essentially. There's a common library of commands, there's a common skills and agents and hooks, and then the common standards that are enforced around coverage, testing, code, all of that, right? And and what was interesting is that, and you know, your your comment about me being in quality, it it doesn't escape me because you know For most of my career, I was, you know, the the the the role that was trying to enforce better quality, better testing. not that quality and testing necessarily are the same, but you get to quality by some sort of test. and so in some cases, you don't necessarily need that to be a separate testing team. I I believe very strongly that an individual's quality of work has nothing to do with whether or not they have a partner with a quality. title in their you know in their role or or what they did on the day to day so I think you know what we said was hey here are the standards that we want enforced in this framework and then those frame then those standards essentially became enforced and by a series of gating those that's where the enforcement came in right so we we relied heavily on the agents to say okay we want to make sure that we are not missing tests or that that you know somebody can't push something out to production without having a robust set of tests that meet this criteria. What was interesting is that when I came in a couple of months ago and did an evaluation of just the tests that were part of the pipeline, we were actually, this and again, this is gonna shock some people who do who know me and my career, we had too many tests running at at for individuals. So we were actually running huge, major suites of tests for kind of, you know, arbitrarily essentially, right? And we were like, great, well w we won't we won't be missing tests because we're going to run the entire gamut or the the whole suite of tests in there. And so we almost over indexed and said, we take testing so seriously that we're going to really lock it in and gate that. And so we wound up saying, hey, I think there's a w there's a better way to do this. So we we put together what I just kind of called a risk classifier, right? Of which is really like, you know, go back six To nine months of data, and how would we classify the types of deployments that we as a company had been doing over the last six to nine months? And let's see if we can logically put together risk classification. So, what we wound up doing was instead of running the entire suite of tests for every single deployment or merge or check-in, we wound up being able to tag types of deployments that were coming up and then apply a suite of tests that were appropriate to. That and then you know, obviously Jason Griffing: Mm-hmm. Melissa: the higher the risk, the more tests, and the more human in the loop, there, right? And that kind of gets into some of the dark factory stuff that we're also working on as well, right? A lot of people think about dark factory, well, and it is really, but it's getting

to the point where 100% autonomous, right? And so I think a lot of times that that turns into everything must be 100% autonomous, and that was, I think, a an opinion that was shared. By some folks, but as in everything, we have to be able to tier types of things and associate those with risk. So now that dark factory, there are some things that can and are 100% autonomous where there is zero human interaction in there. But the higher the risk, and again, based off of data and that common language, and based off of folks that are actually doing the business and have subject matter expertise. In the industry in which we are supporting, we now have a tiered way of being able to have 100% autonomy in places where it makes sense, the less risky, and some level of autonomy closer to the higher the risk, the less 100% autonomy Jason Griffing: Mm-hmm. Melissa: we get to. And so we're really figuring out what that means for those high risk areas that we've already classified as high risk, and what does dark factory mean? mean in those situations and those thing are things are right now that we're actively working through and defining. Jason Griffing: Yeah. The the point about, you know, automating all of this and particularly test coverage definitely resonates 'cause you hear a lot of people's, you know, make make this argument that like, well, A AI can't be just, you know, writing these features like the quality is gonna you know, go go in the toilet and so forth. And it's like, well, r realistically, like if you look at human engineers, like the test coverage and things like this is generally pretty poor because nobody really loves writing tests and so forth. And so you can actually like in some ways make the argument that at least when it comes to things like test coverage, that quality goes up when AI does the code. But separate from test coverage, you know, there are a lot of like subjective decisions that go into designing software. And I'm curious, do you guys have like like how do you guys think about architecture? Because that's like again a lot more subjective than writing a test that can tell you whether this passed or failed. There's a million ways to skin the cat when you develop a feature. And sometimes AI has the tendency to over engineer or introduce unnecessary abstractions and just make things complicated and hard to maintain over time. So how do you guys think about the more fuzzy, squishy side of software engineering where people have to make choices about how things get built? And those choices can really have an impact on the long term maintainability of the code. Melissa: Yeah, y you know, this framework has been in existence for about a year and a half or so, somewhere around there, right? And it's kinda gone through its own reiter iterations, reiterations, all of that. we're currently r evalu continuing to evaluate the the framework itself and and putting in there. But, you know, things around the architect I think there are there

are lessons learned of, you know, maybe taking the steps to think about what would an architect, an actual software enterprise architect, w what would they be con concerned about when when a framework is in place there and kind of going back there. So we're we're we're doing a little bit of Cleaning up from maybe some architecture mistakes or n n unknowns at that time when we built the framework out there. But the way that we're counterbalancing that right now is that, you know, we're we're co encoding standards into the factory rather than relying on reviewers. that's probably the big ones, right? And I mean quite honestly. And you know, we're we're also one one thing that's kind of reared itself in the last couple of weeks that we're looking at real time right now is that we we do allow for easy bypass, right? So a human right now could truly bypass a lot of these things because we wanted to weigh heavy gating and and gating that could not be bypassed, but and and see how that would work. And some people are looking at it's it's better for me in these cases to get something out quickly to market, so to speak, rather than to put heavy enforcement or standards and code in there. And we we still want the human to be able to make that decision within the the Jason Griffing: Mm-hmm. Melissa: right boundaries and guidelines. But I think for for the Those the standards encoding them into the factory, like I said, rather than relying on reviewers was is a major way that we're kind of looking back and saying what would we do differently from a maybe not heavily architected framework, but now that needs to be heavily architected. Hooks, linters, security scans, and then architecture rules themselves become gates now, or that we're we're building in that the agents must pass without really. Really looking at again going up to the higher risk classification, the more human in the loop or the less autonomous based off of risk there are, we want to make sure that the agents have those gates in there with very little exceptions to bypass those. Yeah, we're looking at you know scaling reviews by risk. Not everything needs the same scrutiny as we kind of went back to the you know the testing thing. and then really probably the biggest lesson from kind of coming from the quality. Standpoint is that you can't inspect quality at the end. You have to build that into the system that produces the work as far left as humanly possible or as AI possible. Jason Griffing: Mm-hmm. Yeah. I love that. It sounds like the approach that you're taking is really trying to balance a lot of different variables in terms of introducing f standards and processes, but also maintaining some flexibility and evaluation. I like the framework you've laid out of like evaluating, you know, if you're just moving a button on a page of the UI, like that clearly does not need the same scru scrutiny as building an entire new architecture for a brand new feature set that a bunch of stuff is going to get built on top of in the future. So you have to be thoughtful about how you're deploying this. And there's a lot of shades of gray, so to speak, in terms of of how you approach it. So I really appreciate you sharing all of those perspectives. This has been a really fun conversation. We're going to wrap up, but before we do, I'd love to close with a common question, which is, you know, how optimistic or pessimistic are you in terms of our ability as you know, business leaders and a sus as a a society to navigate this AI transformation in a way that turns out is a net positive for people. Melissa: I am cautiously optimistic. and the caution is the point there, I think. but the upside, quite honestly, is enormous. really, the outcome depends on whether we deploy AI with the same discipline we'd bring to any high-stakes system. governance testing and honesty about the limits is where it is. optimism isn't passive, it's a quality practice. Jason Griffing: Yeah, I love that. Well, Melissa, thank you again for for joining me. This has been a really fun conversation for anybody who's listening who might want to connect with you, ask a follow-up question, learn more about what you're doing, what would be the best way for them to do that? Melissa: pretty active on LinkedIn, so LinkedIn's the first way to go and then we can talk about what's next there. Jason Griffing: All right, awesome. Well I will be sure to include a link to your LinkedIn in the show notes. And once again, thanks so much for joining me. Melissa: Thank you.