Monitor Applications with Empathy with Bill James

What does it mean for an agency to monitor applications with empathy to achieve successful mission outcomes? Bill James is the President of Federal Business LLC and FedSmarts LLC. He is also a former Deputy Assistant Secretary of Development and Operations in the Office of Information and Technology at the Department of Veterans Affairs. He joins Carolyn and Mark to talk about the importance of application monitoring, culture, and empathy when executing a mission.

Episode Table of Contents

• [00:30] Introducing Our Guest, Bill James
• [09:29] The Onus of the User Experience
• [20:33] Applications Monitoring Is Integrated into the VA Process

Episode Links and Resources

• Bill James
• FedSmarts LLC
• Federal Business LLC
• Department of Veterans Affairs

Introducing Our Guest, Bill James

Carolyn: So today, our guest is Bill James. He is president of Federal Business LLC. In his previous role as Deputy Assistant Secretary of Development and Operations in the office of Information and Technology at the Department of Veterans Affairs, Bill led the VA's largest information technology organization to deliver enterprise-wide technology products and services to veterans.

He has been able to carry those skills into his current role as president of Federal Business LLC. And today, we're going to get Bill's perspective on why Application Performance Monitoring or APM is no longer a luxury, but a necessity. And he just recently put out a blog that, I'm going to nerd out here, I really like the blog. It's easy to understand.

One of the things he says in it, or some of the perspectives we're going to get from him, is how APM for VA software applications is necessary now and critical for the future. And how it helps the VA, and I'm going to throw in there, like any organization, any agency, avoid or recover from outages, increase VA OIT productivity and observability, offer insights into investments needed for innovation and understand and improve the customer experience of veterans. I love that last bit. The customer experience.

Bill: Thank you very much, Carolyn and Mark. I'm really happy to be here today, and you've touched one of my hot buttons. I'm really interested in all of it, how the technology ultimately relates and improves the end-user experience. Specifically and particularly, our veterans. And that's why I loved working at the VA so much.

Focus On Veterans’ Experience

Carolyn: Well, and that topic I feel like is especially timely Mark. Especially with the presidential executive order around user experience. I mean, you're kind on the cutting edge, Bill. I mean, you've been doing this before it was cool. You've been worried about the customer experience.

Bill: That's right. I grew up as a programmer, a coder, and as a mathematician. It was always interesting to me how we could build a code and write it. And we thought our job was done when we hit the end card, back in the day when we had punch cards.

But that wall, was frankly was a false wall, and what we never thought through, I think clearly enough into what that code actually did for the end-user. So I think with the new executive order and clearly the focus on the veterans' experience in the VA, that wall came crumbling down for me particularly. It was really a great place to work and a great place to exercise this whole idea of customer experience from the IT perspective specifically.

Carolyn: We're definitely going to dive more into that. Before we go there, for our listeners that may not be as familiar with application performance monitoring or APM, will you give us a quick definition of what that is?

Bill: Yes. It's the heartbeat of your systems, and specifically of software. So, many folks have gone to the doctor or seen these electrocardiograms, where they put these things on your chest and you have the little needle that draws how your heartbeat beat is beating.

What Is Application Performance Monitoring

Bill: Software needs that very same type of telemetry, where it can show everybody it's alive, it's working, it's working well. And then you can also take that very same monitoring capability and say, "Well, okay, fine. You think your software's working well in the black little box, how is the user experiencing it?"

So your software may be working, but the user network may be down so the user can't see the software, even if the software's running. This end to end understanding of what is working and how does it relate through all of the layers of technology to the end-user, and having the tools that really give you a dashboard to allow you to see that from a management perspective, see that monitoring data from a management perspective, that's APM. So in my view, it's the software EKG.

Mark: I think this is a very fascinating topic, not just because we work at Dynatrace, but because of what you've said and what I've read in your articles. I have seen resistance across other agencies, to adopt and to look at the world the way that you have described it. Why is that happening? Can you help us understand why we see this across other agencies?

Bill: I think that a lot of it is cultural. I used to say, and I still believe this, that DevOps equals empathy. And it's not just empathy for the end-user, but Dev and Ops, and the other phrase, DevSecOps, the development, the security, and the operations, grew up as stovepipes in the IT world.

Empathy: Monitor Applications and Care About Your Neighbor

Bill: And they still operate that way in a lot of ways. So empathy means you care about your neighbor. You care about the other parts of your organization and you obviously very much so, care about the end-user.

I think every organization, every federal agency, every company, needs to first of all, think about who your customers are. But secondly, internally, think about, how can our culture be inclusive and empathetic in the sense that, I don't live in a stovepipe, everything that I do affects somebody else.

It's more than just soft words because you can measure that and you should be able to monitor that. Which takes us right back down to, from a software perspective, the application performance monitoring. I want the software folks to care at three o'clock in the morning, if their software application goes down, I want them to be the first one to get the phone call, "Hey, your code broke."

And then the blog, I quoted my YBI YOI "You built it, you own it, right?" You don't drop and run. Once you build that code, you own it forever. And not only do you own it, but you own it with your partners, the security partners, and the operations partners. So I think APM gives that foundation for everyone to share a common view of what's going on. Therefore it begins to break down those stovepipes.

Carolyn: That goes back to what I mentioned before. One of the most important benefits to me of the DevSecOps is that everybody becomes responsible for that user experience. We take the onus off of the user. You gave an analogy of the airline pilot. I'm going to let you share that analogy.

The Aha Moment to Monitor Applications

Bill: Yes. So the idea is, imagine if you were a passenger on an airline and the pilot announced "I have no instruments in the cockpit, so please let me know if one of our engines stopped running." You're asking the passengers to let you know, because "Hey, I'm flying blind and I sure hope everything works out, but if it doesn't, please let me know." I mean, that's silly.

But that's how IT used to work, and to a large degree, in VA, it's very different now, we've changed, come a long way. But we have plenty of instruments that monitored networks and plenty of instruments that monitored computers. And of course, we had thermometers and data centers. We have all sorts of instrumentation around the computing part, but we didn't have much instrumentation or monitoring on the software part.

So the ops had a lot of instrumentation, not so much the Dev, not so much the applications. And so, when we came up to the delivery of the software supporting the mission act in June of 2019, I was nervous. Because we didn't have what we needed from a software monitoring perspective. We scrambled and cobbled together, a lot of things, but it's come a long way since then. And that was that aha moment for me really, it's the, "Oh my gosh. We don't have the instrumentation, the monitoring observability that we need on this code."

Carolyn: Well, I love what you say in the blog. I'm just going to read directly from the blog here. The software folks needed to feel like they were part of a larger team, the mission, right?

Bill: Yes.

The Onus of the User Experience

Carolyn: That was responsible for the end-users experience. So we take the onus off of the passengers, to let me know if there's an engine out and we own that. And it seems like, especially within government, the onus of the user experience has been on the end-user a lot and partly because where else are they going to go, right?

Bill: Exactly right. Trouble tickets, that was the way that the software folks knew when their application was down, was when a trouble ticket was issued by a user. "Hey, this is not working." And so it's like the pilot, when a passenger, raises his hand, "Hey, one of the engines is out." That shouldn't be the first time you hear that.

You should beat your users to the punch in terms of knowing what's going down when a disc drive is filling up or when some application is having problems and it has to be taken the entire perspective, from the code to the end-user. We have to be observant of what's going on.

And so exactly right. It's really, really critically important that we take measure of what's happening in our infrastructure and how our users are affected.

Carolyn: So, you mentioned that at the VA, you saw a lack of APM, the lack of APM exposed a cultural crack. What do you mean by that? Unpack the cultural crack that you saw?

Missing to Monitor Applications May Result in Cultural Crack

Bill: Right. So as the deputy secretary for DevOps, I saw both sides. I saw the development and then I saw the operations. And when you have a lot of the metrics and the instrumentation on the ops side, and very little on the dev side, to me, it exposes a responsibility gap.

And to your point, this is a joint responsibility and the whole idea of product management, which is something that the VA has pivoted to as opposed to project management, when you move to product management. That product idea includes everybody, your user, the inside developers, your operations. It is the manifestation in a lot of ways of this DevOps or agile way of life and way of developing and operating code.

So the cultural divide that I saw was that the software folks and I'm one of them by the way, would write the code, and what I call drop and run. They'd write the code and they would expect the operations teams to run it. Well, that's great.

And when things break, the software folks are the ones, in some cases, if there's a bug or some security exposure or something, they're the ones that get called. But the responsibility was pretty much on the operations side of the fence. So from a cultural perspective, I wanted to balance that. So everybody had a role, Dev, Ops and the user, nothing like having a great champion, a business champion who owns the operational responsibility of the outcomes, for example, of a specific piece of software.

But if you have that great team and they're bound together, that's the essence of a product team, as opposed to a project team where you have milestones and you have an end, right?

We’re All in This Together

Bill: Every project has an end, but that's not, in the software business, that's exactly the point. It does not end. You as a project manager for building an application, you build the application, and your responsibility doesn't end there and you get to move on to another project.

Yes. You get to move on to another project, but your responsibility for the previous project does not end. It endures. And you are now part of a team with the operations and the security and the end-user, to make sure that the end-user experience is good and frankly, great.

So that idea that we're all in this together, that we are a holistic team, not just chunks or pieces in a series of milestones, or that we are all in it together, forever together in service of the veterans. So that's a very passionate thing about anybody who works at the VA. Everybody that has heard me talk about that before, that working at the VA is like no other agency. Their purpose is so clear, like no other company, and it's so noble.

You feel good about working there and you get passionate about providing the services to the veterans that they've earned. So when you tie that fantastic noble objective, with the toolsets and the culture being able to deliver it, it's just a fantastic experience for me.

The Culture in the Veterans Affairs

Mark: Bill, do you feel like, obviously the culture helps with the mission because your end users are veterans and you want to support veterans, et cetera. So, I completely get that. Do you think having everything under your purview when you were there, helped make that happen, as opposed to maybe some other organizations within government?

Bill: Yes. I think yes. And I don't get the credit for that. We have some fantastic leadership, that moves VA in that direction. Frankly, we had some fantastic technology support. VA has a great digital service team there. They brought a lot of great new ideas. APM was one of them. I give them credit for that.

My friend, Steve Vito says, "Lead with your ears, not with your mouth. Everybody was born with two ears, one mouth, use them in that ratio." But in order to adopt those ideas, you have to be open to listening to them. Then from that perspective, having a DevOps organization, now it's DevSecOps organization, you do have all the levers in front of you to knit the culture and the toolsets and the objectives together. Frankly, that was really one of the reasons why we could do what we were able to do in VA and why they've continued to move ahead in a lot of great ways since that point.

I think a lot of other agencies, they absolutely should look at this DevOps model or the DevSecOps model and consider that. The other thing that's different that the VA had, that all the other agencies don't, is that we have a strong CIO in the VA, and they had the financial accountability and authority that was viewed in the CIO role, by the Clinger-Cohen Act.

The Challenge to Monitor Applications in DoD

Bill: A lot of other agencies don't have that single accountability from a budget perspective. So when you have the culture, you have the purpose and you have the ability to control the finances with a single governance model, that makes your life a whole lot easier. A lot of agencies don' have those very same authorities and powers imbued into the organization. So I think those are all necessary pieces of the puzzle.

Mark: Do you think that your colleagues on the DoD side of the house struggle with that? Because there's not that connection between the DevSecOps side of the house and maybe the mission owner, you wouldn't say line of business, but the mission owner?

Bill: I think so. As an IT person, it's hard, like in the DoD, for you to see the outcomes of your activity. So you might be writing a line of code, and maybe if you're in the air force, for example, you may have six or seven layers removed from still on target or some mission outcome. So it can be more difficult in organizations like DoD than in an agency like VA where the purpose is so absolutely very clear and crystal clear. We build kiosks or at least the software for kiosks that our veteran touches. That's very close to you as an IT person and you can see the outcomes directly, not so much in DoD, your outcomes are farther away and it's harder for you to see.

The Old Waterfall Model

Bill: Having said all that. I do believe that the idea of tying your operators and the users, for example, an air force pilot let's take the software person and the hardware, the operations infrastructure person, tying those three legs of the stool together, I think, produces the aha moments that you don't otherwise get when you live in your own little stovepipes. And so I would absolutely recommend that.

Mark: When you were on that side of the world, were they creating software factories at that time?

Bill: Yes, there were a few software factories, but it was still the old waterfall model. And so the software factories produced code that someone else implemented and operated. And so the software factories that the VA is building and a lot frankly, everybody's thinking about the idea of virtual software factories. But if you do that in an agile sense, in a DevOps sense, you get very different outcomes.

So let's build something today, that's fully instrumented and fully secure. But let's build and deliver something today as opposed to plan to deliver something perfect, maybe never. So the whole waterfall model I think really builds and frankly constructs a lot of these cultural boundaries that we try to erase in the DevOps.

And so, back to the APM, it doesn't matter what you build, and it doesn't matter what process you use if you don't know how it operates. If you have no observability, no insights into whether it's up or down, no understanding if it's alive or Dev. Two in the morning, I want to be able to not literally, but figuratively hear that heartbeat of the application that my code is running. And so it puts a smile on your face and you can sleep comfortably.

Applications Monitoring Is Integrated into the VA Process

Bill: But if it's not, I want to be the first to know. So that's a critical piece of operation, frankly. And it's something that ties you as a software builder to the operation in ways that frankly you don't get into in a waterfall model.

Carolyn: Do you think that the VA has now baked APM into their process? Like they're using it?

Bill: Yes. Policy-wise, yes. Tools wise. Yes, I think culturally, we still, the VA have a way to go. I mean, we made huge progress. I mean it’s night and day difference. But I think there's still a way to go there. The software inventory in a VA is huge. 800 to a thousand applications in the VA software inventory. A lot of that code is legacy code that has been and still works and runs OnPrem. And they still turn out the goods and batched jobs like it has for years.

So now you've got these applications. And I think a lot of agencies do these applications that are very successful in what they do and what they accomplish. How do you go back into those and introduce modern tools? Say like APM, how do you move, what I call the electronic alligator clips? How do you attach those to these old legacy...