Leading the Shift—a podcast from Microsoft Azure
Episode 5: Cristina Pieretti | Moody’s
[MUSIC]
Susan Etlinger (00:05):
Welcome to Leading the Shift, a podcast from Microsoft Azure where we hear from the people at the forefront of data, AI and cloud technologies. I'm your host, Susan Etlinger. Would it surprise you to learn that some of the fastest generative AI adoption is happening in older, larger, highly regulated industries like financial services? John Moody founded Moody's in 1909 to help leaders make confident decisions. Now we fast-forward to December 2023. Moody's launches Research Assistant, a first of its kind search and analytical tool powered by generative AI. The mission is the same as it was over a century ago, but the technology is brand new.
Today, my guest is Cristina Pieretti, general manager of digital insights from Moody's Analytics. I first met Cristina at an event early last spring where she shared how she and her team had already launched a gen AI product and what they were learning, and I knew we needed to have her on the show so you can hear it straight from her. Cristina, I'm so excited to have you here. Thank you so much and welcome to Leading the Shift.
Cristina Pieretti (01:10):
Thank you, Susan. The pleasure is mine. I'm super excited to be joining you today.
Susan Etlinger (01:15):
Great. We met several months ago at an event, and I remember looking at you and thinking, "Who is this woman who has already built a generative AI app for her organization?" You were so far ahead of the curve and I was just transfixed really by the story that you were telling, so I'm really excited to share that with a broader audience today. You are the GM of digital insights at Moody's, and you have been instrumental in the development of Research Assistant, which is an app that you have released to the market. Maybe you could just start by telling us how did you get from where you started to hear a little bit about your career journey? And then, let's talk about the app and what you've learned.
Cristina Pieretti (01:59):
Of course, Susan. My pleasure. First of all, I'm Venezuelan. I've been out of Venezuela for the last 25 years, and 18 of those have been in the US, which I consider my home, and 17 of those 18 years have been at Moody's. I'm an engineer by education, by training, and then I work in banking. So, that's how I got to be very familiar and passionate for both technology and financial services. And then, in Moody's, I would describe my career as a continuous chain of changes and innovation. I've been very privileged to work here and have been given the opportunity of always bringing change to the products I manage. Always, the company has been very open about new technologies, and that's actually how gen AI came to be.
Gen AI was a mandate directly from Rob Fauber, our CEO, and the president of Moody's Analytics, Steve Tulenko. They wanted to make sure when they saw the disruption that we understood what it meant, what it meant for us as a company, how we could operate more efficiently. How could it potentially disrupt our offering? They also wanted to understand how would it impact our customers? And by exploring those things and really getting very close with our technology, we identified an initial opportunity and that's how Research Assistant came to be.
Susan Etlinger (03:18):
So, let's talk about the opportunity. How did you do that? You sat in a room with some people, you had some virtual calls? Who were the stakeholders that you brought together and how did you come up with this idea?
Cristina Pieretti (03:29):
I think part of it, it's Moody's we've been working with machine learning for a long time. So, we develop a lot of quantitative models. We also collect a lot of data, so both for those models and for data collection we had been experimenting with machine learning for a long time. Actually one of my previous jobs around eight years ago, I was one of the founding members of the innovation unit at Moody's and part of my role there was how to leverage new technologies, such as machine learning in that case, to be able to collect data more efficiently out and also to automate tasks. I think it was very enlightening to us when we saw gen AI because we had done a lot of experiments to try to get, to automate some of the tasks that we're now seeing that gen AI is capable of doing.
So, we could see how relevant and how impactful that innovation was and that's why immediately we actually had a team that is focused on artificial intelligence, again, on how to make sure that we stay at the forefront of the market. We had an independent team doing that. We also had the CEO and the president very keen on understanding how we could leverage this technology. And then, as part of my current job, I'm also responsible for, first, a product that basically packages all the content and the analytics that we have at Moody's and also for Moody's.com as a platform. We saw an opportunity in, "Okay, let's explore this new technology. Let's combine it with data that we really know and can trust and with a platform that had cutting edge technology," and we knew it was ready to explore because that's what we did at the beginning, explore what could we do with this new technology?
So, that's how we started. We started very small with, I would say, it was not even a full squad. Literally from the beginning, it was the laptop of one of our developers, understanding what could the product do for us and how could it work? We were astonished by what it was doing, so we decided to put more and more resources. I think one of the things we quickly learned, and also one of the things that we had the biggest objections, is, "Okay, how are you throwing more resources into something that is so uncertain? And have we drank too much of that Kool-Aid?" I think sometimes as leaders, we have to take a stance on that. And yes, we didn't deploy all our squads, but we quickly start escalating the initiative because we saw the potential. I think it's always better to be the disruptor and the one that's driving the change than to be the disrupted one.
Susan Etlinger (06:06):
Yeah, yeah. It's a really interesting dynamic because a lot of us grew up in the world where you have an idea and then you build a business case, and then you get support for the business case, and then you get more support for the business case, and then you get a little bit of money, and then you build something. All of this feels very compressed, and one of the themes that I've been hearing in the conversations I've been having with people is that sense of at some point, you just need to take the leap into the uncertain future. So, to what extent when you say you started small, what do you mean by that?
Cristina Pieretti (06:37):
Well, what I mean, we started small. I think it was literally three or four developers. The other day, I was reflecting to the team. We have a pretty well-established product development life cycle, I would say in the unit and in Moody's, where we start as basically as you said. We have a strategy team, a product strategy team that will look at a business opportunity, size the opportunity, understand our right to win, and then define next steps. And then, from there comes product, making it a reality. We work in a squad with UX people, with engineers, with QA people, with product. And then, we do customer testing, a lot of customer testing along the way until we go and have our go-to-market initiative. I was reflecting with the team that this was a little bit the other way. We started with engineering being super excited with what the technology could do for us. Of course, these are engineers that have worked with our products for a long time, so they have a lot of business context.
So, that use case were pretty real. Then, once we started to gain traction, we realized, "Wait a minute, we need product managers for which this is not their side job, but it's their full-time job." I would say not until very recently, we now have the whole full PDLC established for this product where we have strategy working with it and we have product and we have engineering. But it definitely has been at a very different speed. The other key difference with how we develop products is usually we determine what's going to be the minimum viable product, and then we start developing. Here, we realized that it was so uncertain both for us and for our customers that we had to emphasize more on showing our customers the art of what's possible.
At a moment some months ago, I told the team, "Wait a minute. We have to maybe pause the roadmap. We have to devote some of those resources towards showing our customers the art of what's possible." And what I meant by that, it's working closely with our customers, and start experimenting with them and start doing proof of concepts with them so we could demonstrate the value to them. We could also learn in the process. Then, once that's successful and we see critical mass, and we've seen that with this new technology combined with our domain expertise, combined with the vastness of the Moody's data state, we can then develop a solution that it's going to be successful. And only then it enters the roadmap. So, it has also been a big shift in the way we approach product development.
Susan Etlinger (09:14):
What kinds of questions did customers ask, and was this a different way of engaging with them than you had in the past?
Cristina Pieretti (09:20):
Yeah, I'm probably going to give you different responses for this. One of the things I always share with our sales team, you're guaranteed to have an engaging conversation with any customers and for your customers to take the meeting with you because everyone wants to talk about gen AI. Now, once you start that conversation, there's several reactions you might have. I would say in general, I'm seeing a mandate more and more for every type of organization to adopt gen AI. Basically, what I'm hearing more as a common theme is we need to achieve efficiencies with gen AI, we need to save hours, we need to have a more effective deployment of our resources. We want to do more with less. Now, when you start talking about the implementation, then I think one of the things that plays a pivotal role depending on the profile of the institution is their risk and controls.
You see banks that are highly regulated, of course, taking a much more... I would say risk is much higher in their considerations and that's, of course, understandably going to drive a slower progress. The other phenomenon that you observe is also related to the size of the institutions. Bigger institutions, yes, they have more resources but they also have more committees, and they have an AI transformation office, and you have big technology teams and you have business teams. So, by nature, it takes more. One of the things that, for example, we've been very successful with some of those institutions, it's in being able to identify small projects. Let's start small, prove the success, and then move towards bigger initiatives as opposed to try to tackle everything at the same place because it might be that only leads to slower progress.
So, just to summarize, I think there's very different risk profiles in the different players in the industry that are going to adopt this technology. There's also, depending on how complex the organizations are and how mature, also, are they in their journey? Do they already have an AI governance framework, for example? How comfortable are they in delegating tasks to gen AI and to other data? There was a lot of comments around hallucinations at the beginning, and can I trust that data, where is it coming? That type of things.
Susan Etlinger (11:39):
Yeah. Let's actually talk about the use case because I think that also brings in the question of how you mitigate hallucinations and how you mitigate other kinds of risks that come with the technology. The use case is the Research Assistant, can you just tell us a little bit about that?
Cristina Pieretti (11:56):
Yeah. I would say it's been evolving. When we launched Research Assistant, basically what we wanted to do is be able to surface information in an efficient and quick manner to our customers so they could run analysis, peer comparisons, build reports on an efficient manner. It was both a combination of how quickly we could surface the information, how accurate was the information that we were surfacing? Because we could put the right information, the information that you were interested, in your fingertips. And then, the ability for you to incorporate that into your workflow. We actually started pretty small in terms of content. We started basically with the ratings and research that were produced by the rating agency. And there was, I think, a very logical reason first. That's the data that, in particular, my unit was used to work, so we knew the data very well.
Second is data that we can trust and that we can always trace to the source. So, that's how we started, and quickly we saw how the customers, by looking at their prompts, they wanted to know more. They didn't want to limit their research to just ratings and research, they wanted to know anything that Moody's wanted to say about a company, anything that Moody's wanted to say about an industry. They wanted to run comparisons. They wanted to know what was the impact of market trends? So, we immediately saw a need of, "Okay, this is more than just surfacing information. We have to quickly be able to first expand our coverage and our content, but also allow the customers to do more."
The second big innovation... And we've been innovating in the product, I would say we do at least a release a day with either new content, new features that we put into the product or things that we know were not working well for our customer, so we want to improve it. The second big innovation we did is we implemented something we called text-to-code. The commercial name of that is advanced query, and basically what it allows our customers, it's to input complex queries. I'm going to give you an example. Let's say you are an asset manager and you're looking to invest in companies that fit a certain profile. So, you have 10 different criteria, and you can come to Research Assistant and say, "Give me a list of companies that meet those 10 criteria." If you were to provide that to an LLM, at least when we started, the large language model would get confused because of the number of parameters.
What we're doing with this is we're basically taking that query that comes in natural language, using the LLM to write SQL query. We then go to the database, we retrieve the information, again, translate it, and voila, there is a query. That's huge, huge efficiency saving, because imagine the amount of time that an analyst or a group of analysts would have to spend to get that list ready. And with that, we have many other examples.
Then, the third part, which I think it's, to me, the one that I'm more excited about, and it's we've realized that you can actually complete a complex set of tasks with gen AI. And that's where you've probably heard about agents. So, that's where we're focusing a lot of our resources now because we see the potential of automating complete tasks. What I mean, for example, if you're thinking about an underwriting process or you're thinking about an investment process, at the end of those processes, generally you have to write a report. We see a lot of potential that with gen AI, you can automate a lot of that analysis or those reports, and of course, save a lot of hours and still have a lot of good quality or even better quality because you have all those agents gathering different types of information in real time, which potentially will lead to better insights.
Susan Etlinger (15:39):
Wow, so there's a lot to unpack there.
Cristina Pieretti (15:39):
That's a lot. Yeah.
Susan Etlinger (15:41):
No, that's fantastic. There's a lot to unpack. Let's go back to the concerns that customers had at the beginning around the integrity of the data. How did you address that?
Cristina Pieretti (15:53):
Yeah. This is, again, changing a little bit, but where we started and when we reflect in the past, would I have started the same way? I think I would have done it the same way because it allowed us to be a trusted provider for those initial customers. The way we develop this is we don't train the model. We are not a technology company. The main model we leverage is ChatGPT, although we do use some other large language models in some other parts of the process. We decided that all the responses had to be grounded on Moody's data. So, the way that worked is we basically, when we went back to the large language model with a response, we had very specific instructions of, "You cannot use any of the other information that you have been trained for." I'm going to give you a simple example.
If you ask, Moody's knows a lot about Ford, but as part of the research we publish, we don't talk about when Henry Ford was born. Now, that's something that would be available to any large language model. You ask a large language model and they can tell you immediately where he was born. If you ask Research Assistant, the idea is Research Assistant should tell you, "I don't know," because that's not information that is in the Moody's corpus of data. And when we started this, we wanted to make sure that we could stand behind every answer, that when you click on any of the answers on the footnote that is after the answers, you could get back and look at a research document from Moody's or the data point where it had came.
Now, we are still very much on that train of thought, but we have now incorporated other data sources. We always want to know where they're coming from, but we've expanded and we're now using news. Again, we want to make sure that it's a new service, that it's owned by Moody's, and we want to make sure that we can always trace it, but we now have news. We are in the process of incorporating annual reports. So, we're expanding the coverage, but front and center, one of our tenants for development in Research Assistance that we can always stand behind the data that we're surfacing in the tool.
Susan Etlinger (18:10):
So, really what you're saying, as I understand it, is for one thing, taking unstructured data, turning that into code to create a rigor for the query. But then, with the grounding in the Moody's data, you're asking the large language model in a sort of conversational way, "Don't be helpful. Don't try to be helpful. Don't give me anything extraneous, just help."
Cristina Pieretti (18:36):
Exactly that. I would joke with the developers team, "We don't want the large language model to be a mischievous case." In the case like, "I know the answer. I want to be the smarter one." No, no, no, no, no. We want you to answer just based on the data that we're fitting.
Susan Etlinger (18:51):
So, you start with that corpus of just Moody's data, and now you say you're expanding it further?
Cristina Pieretti (18:57):
Yeah.
Susan Etlinger (19:01):
I mean, that sounds like, to me, in a best practice. A best practice, especially in an area where you have highly consequential decisions that are being made based upon that data. Because you think about this in terms of making an investment decision or assessing risk, it's a little bit different from, "Are these the shoes that I should wear to the prom?" kind of a thing.
Cristina Pieretti (19:27):
Now that said, Susan, we always get asked from the clients, "Well, you're not expecting me to consume only Moody's data, right?" We use sources from many different areas, internal data, third-party data, some public information. And yes, we recognize that. The answer that we are going to give you, it's grounded on Moody's data or third-party data that we've decided to incorporate into the answer, but that's why we're not expecting customers to only consume the service from the website. We're actually offering APIs and other ways of data sharing because we recognize that at the end, especially with complex and sophisticated, institutions are going to want to integrate and do orchestration of all these responses on their side. But we do want to emphasize that the responses we give from our tools, either through API or through the web, are always going to be grounded in the Moody's data.
Susan Etlinger (20:21):
No, that makes sense. I mean, you're protecting the integrity of the data and the integrity of the responses at the same time you're giving them the choice to incorporate it with any other inputs that they have or that they'd like to use. No, I think that makes great sense. Let's talk a little bit about the performance of the app, because it's been quite successful, hasn't it?
Cristina Pieretti (20:44):
Yeah. Well, I think this is public information because our CEO generally talks about it in the earnings goals. We're north of 100 customers. We have thousands of users using it from those 100 institutions that are subscribing to the product. And then, we are also very excited about the pipeline and the roadmap that we have for this year. We are very proud to say that this has been the product with the fastest adoption we've ever seen in Moody's. I think that talks a lot about how relevant and how serious a lot of the market participants are taking this technology, but of course, how powerful the combination is of trusted data plus this relevant technology to then bring efficiency into their day-to-day.
Susan Etlinger (21:30):
That's amazing. That's amazing. I'm thinking also about what customers might've learned just simply from using Research Assistant. Not only about, for example, the specific queries that they have, but about what the opportunities around generative AI. One of the things I remember you talking about early on was some design considerations about how you built the app in order to be able to make the experience as intuitive as possible. Because of course, when you're talking to a chatbot or some kind of a query and response technology, you don't have a lot of signals around you to tell you what it is that you can or should be doing. So, how did you approach some of those things?
Cristina Pieretti (22:14):
Yeah. I think that has been subject to continuous improvement. We monitor the prompts and we monitor the success of the app to those prompts. And based on things where we... We either see customers abandoning the tool, they ask a question, they're not happy with it or they didn't get an answer and they just don't come back, or answers that weren't as accurate or successful as they could be. We started implementing some changes. For example, some of the things we quickly work on is how to identify tickers, how to make sure that we address abbreviations. If we see a prompt coming, we reply back, "Did you mean this?"
The other thing is we now have an extensive library of suggested prompts. So, I think all those things have helped our customers to get better at prompting, have made the interface more intuitive. Also, in addition to the things we've done in the UI, of course, we have a group of people that is supporting this product very closely to make sure that we not only allow our customers to use it and find it easy to do so, but we also can talk about best practices in the industry and use cases that are very popular among people with similar jobs.
Susan Etlinger (23:39):
Now it's how many months since you introduced the product?
Cristina Pieretti (23:44):
We launched in December 1st, 2023, so we're past the year mark and I would say we are even more excited with the potential that this has. I would emphasize, again, what I said. I think this is much more than a chat functionality or something. Where we are all using ChatGPT or some flavor of it, we're seeing a big, big potential in how you can use this actually as an assistant, as someone where you can rely on to do some of the repetitive tasks, or to be able to do potentially more things than you usually do because you have someone that if you give that set of instructions, can do the task in a very efficient manner and accurate manner as well.
Susan Etlinger (24:33):
No, that's great. And of course, we're hearing more and more about agentic, about agents being able to do things on our behalf. One of the examples that you gave was being able to pull from multiple sources and orchestrate an answer together. Are there other applications that you're thinking about?
Cristina Pieretti (24:53):
Yeah. I would say of course you can source from different places, I think being able to automate some of the tasks. For example, being able to automate, let's think about lead generation. You are a banker or you're a salesperson and you want to get leads either because you want to sell a product, either because you want to say, "What are M&A targets?" because you want to extend credit. I think that that's a great application because basically what you have to do is to look at companies or targets that meet a certain criteria, and then you have to learn about them and then potentially create a pitch there, go create a report. And that all can be automated.
If we go on the report theme, think about the amount of reports that we create as human beings in our different jobs. It could be a product brief, it could be an investment memo, it could be a credit memo. So, we're seeing a lot of potential there. Also, then I would say the other thing we're seeing is how can we use this and leverage to comply with certain regulations? Let's say that you have to demonstrate to the regulator that you are complying with a certain set of things. You can automate these tools to look for the information, the documents you have, and then determine if you're meeting certain criteria or not.
Susan Etlinger (26:15):
Oh, wow.
Cristina Pieretti (26:16):
So, those are some examples of use cases.
Susan Etlinger (26:18):
Yeah. No, that's a great example. As you look back on the year and you look ahead to the year to come, any kind of insights or learnings or things that you sort of just didn't expect that you want to share with a broader audience, just around what advice you have for them?
Cristina Pieretti (26:39):
I would say number one is the pace of change has been extremely fast, I think has been faster than anything we saw, for the good and for the bad. For the good because suddenly a new model came up and it solved challenges that you were having to address in other ways with the previous models. But on the other hand, it puts a lot of pressure on you to learn the new technologies, to adapt them very quickly, to skill your team so you have people that are willing to take the challenge and to learn very fast. I think it impacted many things. So, I'm going to name a few that I've seen common both for us and for our customers.
The legal and risk considerations. We've talked about it in some of our conversations before, but it's a whole different risk profile from, what is the data that you're using for it, do you have the proprietary rights to use that data? From the client side, is there any private information that their users could be sharing throughout a prompt? Then, there's, are we getting our data out to the outside world and models are going to be trained with that data, and that means that we're going to lose the future of our business? So, the legal and risk ramifications are long, are complex, and I think are probably more profound than in any of the other products I've worked with. And there's a reason for that. I think we are very proud that because of the fact that half of our company is a rating agency, we had risk and controls front and center when we started this journey. But I see it's a common theme across everyone in the industry. How do we make sure that we're doing the right thing and we have the right controls in place?
Then, I think in general, it's so transformative that even the way we approach a customer is different. There's a much larger need to understand what is it that they're trying to do in their day-to-day so you can actually help them automate the task or you can deliver a solution that is fit for purpose. You're no longer kind of a passive partner where they come and fetch for information. You're now working together with them to make sure that you get together to where you want to get. So, the whole commercialization, the sales process, that has also been very different. Then, maybe the last thing I'll point out is I think there's a lot of uncertainty about where this is going to take us, and uncertainty, it's sometimes scary. How I'm going to have the right skills to survive in this new era? How is it going to transform my job? I think those are some of the things we continue to see.
Susan Etlinger (29:36):
Cristina, given all of this change, how do you keep yourself sane?
Cristina Pieretti (29:40):
I was joking to the team yesterday. I was having a conversation with them about agents. I'm thinking about, "Okay, when you have a lot of agents, is that a super agent? Is that a squadron of agents?" At some point I told the team, "My head is literally going to explode." And why is it about to explode? Because first I have to understand the technology and understand what we really mean by an agent. You think Matrix, but it's actually not quite the agents that you would see in the Matrix. Then, what are their applications and how do we not get carried away with the emotion and we actually build something that it's adding value for our customers? But at the same time, we have the right governance framework in place and we're utilizing the right data so we can mitigate as many unforeseen risk as we can. So, it's a combination of yes, you want to be at the forefront of it and you want to be learning and innovating, but at the end, you also wanted to make it for the right reasons and you wanted to do it the right way.
Susan Etlinger (30:43):
Yeah. No, absolutely. I think that combination of the passion and the spirit of experimentation, and the rigor on the other side and how those two work in tandem, to me, that just feels like kind of a perfect way to end the conversation, because this is a time of great experimentation and it's also a time of great change and uncertainty. I think having both of those pieces working together... And then, also of course, what I really appreciate about you is the humility to say, "Yes, this is a lot." This is a lot, but you're still working through it and you're still open and you're still learning, and that's, I think, what's really inspired me to have this conversation with you today. So, thank you so, so much.
Cristina Pieretti (31:30):
Thank you. The pleasure was mine, Susan. I'm very passionate about learning, as I can see you are as well. I think if I always have an advice for people that are coming after us is never stop learning, because change is a constant. It's always going to be there, and we have to keep ourselves open to learning.
Susan Etlinger (31:49):
Fabulous. Thank you again.
Cristina Pieretti (31:51):
Great. Thank you very much.
Susan Etlinger (31:59):
Wow. Here we are in 2025, and it kind of blows my mind that we're already looking back at what we've learned, and Cristina's kind of the perfect guest to do that with. So, here are a couple of things I took away from our conversation that I think are kind of helpful if you're exploring, planning, or implementing gen AI. First, let's talk about building interest from the start. Cristina's team took multiple measures to reduce risk, including grounding responses in Moody's data, telling the model only to use data provided by Moody's and say "I don't know" when it didn't have an answer, and expanding data sources really carefully.
I was also struck by how strategically she and her team approached generative AI. Their focus on what it meant for them as a company in addition to the operational efficiencies is a really multi-layered way of thinking. And equally as important is the focus on customer engagement and feedback. The team's commitment to engaging early in the process and devoting resources to showing customers the art of the possible gave them insights that ultimately turned into product features. That's what we mean when we talk about that flywheel effective AI.
We hope you've enjoyed this episode, and as I'm sure you know, new podcasts live and die on engagement. So, please like, comment, share, tell your friends, and let us know what you'd like to hear about. We're listening. If you'd like to learn more about how people and organizations are innovating with Microsoft Azure, visit azure.microsoft.com.
The Leading the Shift podcast is a place for experts to share their insights and opinions. As students of the future of technology, Microsoft values inputs from a diverse set of voices. That said, the opinions and findings of our guests are their own, and they may not necessarily reflect Microsoft's own research or positions.
Leading the Shift is a production of Microsoft Azure. I'm your host, Susan Etlinger. Our executive producers are Hailie Meehan, Aaron Russell, Erik Williams, and me. Production support provided by the incredible team at Indigo Slate.
[END]
Episode 5: Cristina Pieretti | Moody’s
[MUSIC]
Susan Etlinger (00:05):
Welcome to Leading the Shift, a podcast from Microsoft Azure where we hear from the people at the forefront of data, AI and cloud technologies. I'm your host, Susan Etlinger. Would it surprise you to learn that some of the fastest generative AI adoption is happening in older, larger, highly regulated industries like financial services? John Moody founded Moody's in 1909 to help leaders make confident decisions. Now we fast-forward to December 2023. Moody's launches Research Assistant, a first of its kind search and analytical tool powered by generative AI. The mission is the same as it was over a century ago, but the technology is brand new.
Today, my guest is Cristina Pieretti, general manager of digital insights from Moody's Analytics. I first met Cristina at an event early last spring where she shared how she and her team had already launched a gen AI product and what they were learning, and I knew we needed to have her on the show so you can hear it straight from her. Cristina, I'm so excited to have you here. Thank you so much and welcome to Leading the Shift.
Cristina Pieretti (01:10):
Thank you, Susan. The pleasure is mine. I'm super excited to be joining you today.
Susan Etlinger (01:15):
Great. We met several months ago at an event, and I remember looking at you and thinking, "Who is this woman who has already built a generative AI app for her organization?" You were so far ahead of the curve and I was just transfixed really by the story that you were telling, so I'm really excited to share that with a broader audience today. You are the GM of digital insights at Moody's, and you have been instrumental in the development of Research Assistant, which is an app that you have released to the market. Maybe you could just start by telling us how did you get from where you started to hear a little bit about your career journey? And then, let's talk about the app and what you've learned.
Cristina Pieretti (01:59):
Of course, Susan. My pleasure. First of all, I'm Venezuelan. I've been out of Venezuela for the last 25 years, and 18 of those have been in the US, which I consider my home, and 17 of those 18 years have been at Moody's. I'm an engineer by education, by training, and then I work in banking. So, that's how I got to be very familiar and passionate for both technology and financial services. And then, in Moody's, I would describe my career as a continuous chain of changes and innovation. I've been very privileged to work here and have been given the opportunity of always bringing change to the products I manage. Always, the company has been very open about new technologies, and that's actually how gen AI came to be.
Gen AI was a mandate directly from Rob Fauber, our CEO, and the president of Moody's Analytics, Steve Tulenko. They wanted to make sure when they saw the disruption that we understood what it meant, what it meant for us as a company, how we could operate more efficiently. How could it potentially disrupt our offering? They also wanted to understand how would it impact our customers? And by exploring those things and really getting very close with our technology, we identified an initial opportunity and that's how Research Assistant came to be.
Susan Etlinger (03:18):
So, let's talk about the opportunity. How did you do that? You sat in a room with some people, you had some virtual calls? Who were the stakeholders that you brought together and how did you come up with this idea?
Cristina Pieretti (03:29):
I think part of it, it's Moody's we've been working with machine learning for a long time. So, we develop a lot of quantitative models. We also collect a lot of data, so both for those models and for data collection we had been experimenting with machine learning for a long time. Actually one of my previous jobs around eight years ago, I was one of the founding members of the innovation unit at Moody's and part of my role there was how to leverage new technologies, such as machine learning in that case, to be able to collect data more efficiently out and also to automate tasks. I think it was very enlightening to us when we saw gen AI because we had done a lot of experiments to try to get, to automate some of the tasks that we're now seeing that gen AI is capable of doing.
So, we could see how relevant and how impactful that innovation was and that's why immediately we actually had a team that is focused on artificial intelligence, again, on how to make sure that we stay at the forefront of the market. We had an independent team doing that. We also had the CEO and the president very keen on understanding how we could leverage this technology. And then, as part of my current job, I'm also responsible for, first, a product that basically packages all the content and the analytics that we have at Moody's and also for Moody's.com as a platform. We saw an opportunity in, "Okay, let's explore this new technology. Let's combine it with data that we really know and can trust and with a platform that had cutting edge technology," and we knew it was ready to explore because that's what we did at the beginning, explore what could we do with this new technology?
So, that's how we started. We started very small with, I would say, it was not even a full squad. Literally from the beginning, it was the laptop of one of our developers, understanding what could the product do for us and how could it work? We were astonished by what it was doing, so we decided to put more and more resources. I think one of the things we quickly learned, and also one of the things that we had the biggest objections, is, "Okay, how are you throwing more resources into something that is so uncertain? And have we drank too much of that Kool-Aid?" I think sometimes as leaders, we have to take a stance on that. And yes, we didn't deploy all our squads, but we quickly start escalating the initiative because we saw the potential. I think it's always better to be the disruptor and the one that's driving the change than to be the disrupted one.
Susan Etlinger (06:06):
Yeah, yeah. It's a really interesting dynamic because a lot of us grew up in the world where you have an idea and then you build a business case, and then you get support for the business case, and then you get more support for the business case, and then you get a little bit of money, and then you build something. All of this feels very compressed, and one of the themes that I've been hearing in the conversations I've been having with people is that sense of at some point, you just need to take the leap into the uncertain future. So, to what extent when you say you started small, what do you mean by that?
Cristina Pieretti (06:37):
Well, what I mean, we started small. I think it was literally three or four developers. The other day, I was reflecting to the team. We have a pretty well-established product development life cycle, I would say in the unit and in Moody's, where we start as basically as you said. We have a strategy team, a product strategy team that will look at a business opportunity, size the opportunity, understand our right to win, and then define next steps. And then, from there comes product, making it a reality. We work in a squad with UX people, with engineers, with QA people, with product. And then, we do customer testing, a lot of customer testing along the way until we go and have our go-to-market initiative. I was reflecting with the team that this was a little bit the other way. We started with engineering being super excited with what the technology could do for us. Of course, these are engineers that have worked with our products for a long time, so they have a lot of business context.
So, that use case were pretty real. Then, once we started to gain traction, we realized, "Wait a minute, we need product managers for which this is not their side job, but it's their full-time job." I would say not until very recently, we now have the whole full PDLC established for this product where we have strategy working with it and we have product and we have engineering. But it definitely has been at a very different speed. The other key difference with how we develop products is usually we determine what's going to be the minimum viable product, and then we start developing. Here, we realized that it was so uncertain both for us and for our customers that we had to emphasize more on showing our customers the art of what's possible.
At a moment some months ago, I told the team, "Wait a minute. We have to maybe pause the roadmap. We have to devote some of those resources towards showing our customers the art of what's possible." And what I meant by that, it's working closely with our customers, and start experimenting with them and start doing proof of concepts with them so we could demonstrate the value to them. We could also learn in the process. Then, once that's successful and we see critical mass, and we've seen that with this new technology combined with our domain expertise, combined with the vastness of the Moody's data state, we can then develop a solution that it's going to be successful. And only then it enters the roadmap. So, it has also been a big shift in the way we approach product development.
Susan Etlinger (09:14):
What kinds of questions did customers ask, and was this a different way of engaging with them than you had in the past?
Cristina Pieretti (09:20):
Yeah, I'm probably going to give you different responses for this. One of the things I always share with our sales team, you're guaranteed to have an engaging conversation with any customers and for your customers to take the meeting with you because everyone wants to talk about gen AI. Now, once you start that conversation, there's several reactions you might have. I would say in general, I'm seeing a mandate more and more for every type of organization to adopt gen AI. Basically, what I'm hearing more as a common theme is we need to achieve efficiencies with gen AI, we need to save hours, we need to have a more effective deployment of our resources. We want to do more with less. Now, when you start talking about the implementation, then I think one of the things that plays a pivotal role depending on the profile of the institution is their risk and controls.
You see banks that are highly regulated, of course, taking a much more... I would say risk is much higher in their considerations and that's, of course, understandably going to drive a slower progress. The other phenomenon that you observe is also related to the size of the institutions. Bigger institutions, yes, they have more resources but they also have more committees, and they have an AI transformation office, and you have big technology teams and you have business teams. So, by nature, it takes more. One of the things that, for example, we've been very successful with some of those institutions, it's in being able to identify small projects. Let's start small, prove the success, and then move towards bigger initiatives as opposed to try to tackle everything at the same place because it might be that only leads to slower progress.
So, just to summarize, I think there's very different risk profiles in the different players in the industry that are going to adopt this technology. There's also, depending on how complex the organizations are and how mature, also, are they in their journey? Do they already have an AI governance framework, for example? How comfortable are they in delegating tasks to gen AI and to other data? There was a lot of comments around hallucinations at the beginning, and can I trust that data, where is it coming? That type of things.
Susan Etlinger (11:39):
Yeah. Let's actually talk about the use case because I think that also brings in the question of how you mitigate hallucinations and how you mitigate other kinds of risks that come with the technology. The use case is the Research Assistant, can you just tell us a little bit about that?
Cristina Pieretti (11:56):
Yeah. I would say it's been evolving. When we launched Research Assistant, basically what we wanted to do is be able to surface information in an efficient and quick manner to our customers so they could run analysis, peer comparisons, build reports on an efficient manner. It was both a combination of how quickly we could surface the information, how accurate was the information that we were surfacing? Because we could put the right information, the information that you were interested, in your fingertips. And then, the ability for you to incorporate that into your workflow. We actually started pretty small in terms of content. We started basically with the ratings and research that were produced by the rating agency. And there was, I think, a very logical reason first. That's the data that, in particular, my unit was used to work, so we knew the data very well.
Second is data that we can trust and that we can always trace to the source. So, that's how we started, and quickly we saw how the customers, by looking at their prompts, they wanted to know more. They didn't want to limit their research to just ratings and research, they wanted to know anything that Moody's wanted to say about a company, anything that Moody's wanted to say about an industry. They wanted to run comparisons. They wanted to know what was the impact of market trends? So, we immediately saw a need of, "Okay, this is more than just surfacing information. We have to quickly be able to first expand our coverage and our content, but also allow the customers to do more."
The second big innovation... And we've been innovating in the product, I would say we do at least a release a day with either new content, new features that we put into the product or things that we know were not working well for our customer, so we want to improve it. The second big innovation we did is we implemented something we called text-to-code. The commercial name of that is advanced query, and basically what it allows our customers, it's to input complex queries. I'm going to give you an example. Let's say you are an asset manager and you're looking to invest in companies that fit a certain profile. So, you have 10 different criteria, and you can come to Research Assistant and say, "Give me a list of companies that meet those 10 criteria." If you were to provide that to an LLM, at least when we started, the large language model would get confused because of the number of parameters.
What we're doing with this is we're basically taking that query that comes in natural language, using the LLM to write SQL query. We then go to the database, we retrieve the information, again, translate it, and voila, there is a query. That's huge, huge efficiency saving, because imagine the amount of time that an analyst or a group of analysts would have to spend to get that list ready. And with that, we have many other examples.
Then, the third part, which I think it's, to me, the one that I'm more excited about, and it's we've realized that you can actually complete a complex set of tasks with gen AI. And that's where you've probably heard about agents. So, that's where we're focusing a lot of our resources now because we see the potential of automating complete tasks. What I mean, for example, if you're thinking about an underwriting process or you're thinking about an investment process, at the end of those processes, generally you have to write a report. We see a lot of potential that with gen AI, you can automate a lot of that analysis or those reports, and of course, save a lot of hours and still have a lot of good quality or even better quality because you have all those agents gathering different types of information in real time, which potentially will lead to better insights.
Susan Etlinger (15:39):
Wow, so there's a lot to unpack there.
Cristina Pieretti (15:39):
That's a lot. Yeah.
Susan Etlinger (15:41):
No, that's fantastic. There's a lot to unpack. Let's go back to the concerns that customers had at the beginning around the integrity of the data. How did you address that?
Cristina Pieretti (15:53):
Yeah. This is, again, changing a little bit, but where we started and when we reflect in the past, would I have started the same way? I think I would have done it the same way because it allowed us to be a trusted provider for those initial customers. The way we develop this is we don't train the model. We are not a technology company. The main model we leverage is ChatGPT, although we do use some other large language models in some other parts of the process. We decided that all the responses had to be grounded on Moody's data. So, the way that worked is we basically, when we went back to the large language model with a response, we had very specific instructions of, "You cannot use any of the other information that you have been trained for." I'm going to give you a simple example.
If you ask, Moody's knows a lot about Ford, but as part of the research we publish, we don't talk about when Henry Ford was born. Now, that's something that would be available to any large language model. You ask a large language model and they can tell you immediately where he was born. If you ask Research Assistant, the idea is Research Assistant should tell you, "I don't know," because that's not information that is in the Moody's corpus of data. And when we started this, we wanted to make sure that we could stand behind every answer, that when you click on any of the answers on the footnote that is after the answers, you could get back and look at a research document from Moody's or the data point where it had came.
Now, we are still very much on that train of thought, but we have now incorporated other data sources. We always want to know where they're coming from, but we've expanded and we're now using news. Again, we want to make sure that it's a new service, that it's owned by Moody's, and we want to make sure that we can always trace it, but we now have news. We are in the process of incorporating annual reports. So, we're expanding the coverage, but front and center, one of our tenants for development in Research Assistance that we can always stand behind the data that we're surfacing in the tool.
Susan Etlinger (18:10):
So, really what you're saying, as I understand it, is for one thing, taking unstructured data, turning that into code to create a rigor for the query. But then, with the grounding in the Moody's data, you're asking the large language model in a sort of conversational way, "Don't be helpful. Don't try to be helpful. Don't give me anything extraneous, just help."
Cristina Pieretti (18:36):
Exactly that. I would joke with the developers team, "We don't want the large language model to be a mischievous case." In the case like, "I know the answer. I want to be the smarter one." No, no, no, no, no. We want you to answer just based on the data that we're fitting.
Susan Etlinger (18:51):
So, you start with that corpus of just Moody's data, and now you say you're expanding it further?
Cristina Pieretti (18:57):
Yeah.
Susan Etlinger (19:01):
I mean, that sounds like, to me, in a best practice. A best practice, especially in an area where you have highly consequential decisions that are being made based upon that data. Because you think about this in terms of making an investment decision or assessing risk, it's a little bit different from, "Are these the shoes that I should wear to the prom?" kind of a thing.
Cristina Pieretti (19:27):
Now that said, Susan, we always get asked from the clients, "Well, you're not expecting me to consume only Moody's data, right?" We use sources from many different areas, internal data, third-party data, some public information. And yes, we recognize that. The answer that we are going to give you, it's grounded on Moody's data or third-party data that we've decided to incorporate into the answer, but that's why we're not expecting customers to only consume the service from the website. We're actually offering APIs and other ways of data sharing because we recognize that at the end, especially with complex and sophisticated, institutions are going to want to integrate and do orchestration of all these responses on their side. But we do want to emphasize that the responses we give from our tools, either through API or through the web, are always going to be grounded in the Moody's data.
Susan Etlinger (20:21):
No, that makes sense. I mean, you're protecting the integrity of the data and the integrity of the responses at the same time you're giving them the choice to incorporate it with any other inputs that they have or that they'd like to use. No, I think that makes great sense. Let's talk a little bit about the performance of the app, because it's been quite successful, hasn't it?
Cristina Pieretti (20:44):
Yeah. Well, I think this is public information because our CEO generally talks about it in the earnings goals. We're north of 100 customers. We have thousands of users using it from those 100 institutions that are subscribing to the product. And then, we are also very excited about the pipeline and the roadmap that we have for this year. We are very proud to say that this has been the product with the fastest adoption we've ever seen in Moody's. I think that talks a lot about how relevant and how serious a lot of the market participants are taking this technology, but of course, how powerful the combination is of trusted data plus this relevant technology to then bring efficiency into their day-to-day.
Susan Etlinger (21:30):
That's amazing. That's amazing. I'm thinking also about what customers might've learned just simply from using Research Assistant. Not only about, for example, the specific queries that they have, but about what the opportunities around generative AI. One of the things I remember you talking about early on was some design considerations about how you built the app in order to be able to make the experience as intuitive as possible. Because of course, when you're talking to a chatbot or some kind of a query and response technology, you don't have a lot of signals around you to tell you what it is that you can or should be doing. So, how did you approach some of those things?
Cristina Pieretti (22:14):
Yeah. I think that has been subject to continuous improvement. We monitor the prompts and we monitor the success of the app to those prompts. And based on things where we... We either see customers abandoning the tool, they ask a question, they're not happy with it or they didn't get an answer and they just don't come back, or answers that weren't as accurate or successful as they could be. We started implementing some changes. For example, some of the things we quickly work on is how to identify tickers, how to make sure that we address abbreviations. If we see a prompt coming, we reply back, "Did you mean this?"
The other thing is we now have an extensive library of suggested prompts. So, I think all those things have helped our customers to get better at prompting, have made the interface more intuitive. Also, in addition to the things we've done in the UI, of course, we have a group of people that is supporting this product very closely to make sure that we not only allow our customers to use it and find it easy to do so, but we also can talk about best practices in the industry and use cases that are very popular among people with similar jobs.
Susan Etlinger (23:39):
Now it's how many months since you introduced the product?
Cristina Pieretti (23:44):
We launched in December 1st, 2023, so we're past the year mark and I would say we are even more excited with the potential that this has. I would emphasize, again, what I said. I think this is much more than a chat functionality or something. Where we are all using ChatGPT or some flavor of it, we're seeing a big, big potential in how you can use this actually as an assistant, as someone where you can rely on to do some of the repetitive tasks, or to be able to do potentially more things than you usually do because you have someone that if you give that set of instructions, can do the task in a very efficient manner and accurate manner as well.
Susan Etlinger (24:33):
No, that's great. And of course, we're hearing more and more about agentic, about agents being able to do things on our behalf. One of the examples that you gave was being able to pull from multiple sources and orchestrate an answer together. Are there other applications that you're thinking about?
Cristina Pieretti (24:53):
Yeah. I would say of course you can source from different places, I think being able to automate some of the tasks. For example, being able to automate, let's think about lead generation. You are a banker or you're a salesperson and you want to get leads either because you want to sell a product, either because you want to say, "What are M&A targets?" because you want to extend credit. I think that that's a great application because basically what you have to do is to look at companies or targets that meet a certain criteria, and then you have to learn about them and then potentially create a pitch there, go create a report. And that all can be automated.
If we go on the report theme, think about the amount of reports that we create as human beings in our different jobs. It could be a product brief, it could be an investment memo, it could be a credit memo. So, we're seeing a lot of potential there. Also, then I would say the other thing we're seeing is how can we use this and leverage to comply with certain regulations? Let's say that you have to demonstrate to the regulator that you are complying with a certain set of things. You can automate these tools to look for the information, the documents you have, and then determine if you're meeting certain criteria or not.
Susan Etlinger (26:15):
Oh, wow.
Cristina Pieretti (26:16):
So, those are some examples of use cases.
Susan Etlinger (26:18):
Yeah. No, that's a great example. As you look back on the year and you look ahead to the year to come, any kind of insights or learnings or things that you sort of just didn't expect that you want to share with a broader audience, just around what advice you have for them?
Cristina Pieretti (26:39):
I would say number one is the pace of change has been extremely fast, I think has been faster than anything we saw, for the good and for the bad. For the good because suddenly a new model came up and it solved challenges that you were having to address in other ways with the previous models. But on the other hand, it puts a lot of pressure on you to learn the new technologies, to adapt them very quickly, to skill your team so you have people that are willing to take the challenge and to learn very fast. I think it impacted many things. So, I'm going to name a few that I've seen common both for us and for our customers.
The legal and risk considerations. We've talked about it in some of our conversations before, but it's a whole different risk profile from, what is the data that you're using for it, do you have the proprietary rights to use that data? From the client side, is there any private information that their users could be sharing throughout a prompt? Then, there's, are we getting our data out to the outside world and models are going to be trained with that data, and that means that we're going to lose the future of our business? So, the legal and risk ramifications are long, are complex, and I think are probably more profound than in any of the other products I've worked with. And there's a reason for that. I think we are very proud that because of the fact that half of our company is a rating agency, we had risk and controls front and center when we started this journey. But I see it's a common theme across everyone in the industry. How do we make sure that we're doing the right thing and we have the right controls in place?
Then, I think in general, it's so transformative that even the way we approach a customer is different. There's a much larger need to understand what is it that they're trying to do in their day-to-day so you can actually help them automate the task or you can deliver a solution that is fit for purpose. You're no longer kind of a passive partner where they come and fetch for information. You're now working together with them to make sure that you get together to where you want to get. So, the whole commercialization, the sales process, that has also been very different. Then, maybe the last thing I'll point out is I think there's a lot of uncertainty about where this is going to take us, and uncertainty, it's sometimes scary. How I'm going to have the right skills to survive in this new era? How is it going to transform my job? I think those are some of the things we continue to see.
Susan Etlinger (29:36):
Cristina, given all of this change, how do you keep yourself sane?
Cristina Pieretti (29:40):
I was joking to the team yesterday. I was having a conversation with them about agents. I'm thinking about, "Okay, when you have a lot of agents, is that a super agent? Is that a squadron of agents?" At some point I told the team, "My head is literally going to explode." And why is it about to explode? Because first I have to understand the technology and understand what we really mean by an agent. You think Matrix, but it's actually not quite the agents that you would see in the Matrix. Then, what are their applications and how do we not get carried away with the emotion and we actually build something that it's adding value for our customers? But at the same time, we have the right governance framework in place and we're utilizing the right data so we can mitigate as many unforeseen risk as we can. So, it's a combination of yes, you want to be at the forefront of it and you want to be learning and innovating, but at the end, you also wanted to make it for the right reasons and you wanted to do it the right way.
Susan Etlinger (30:43):
Yeah. No, absolutely. I think that combination of the passion and the spirit of experimentation, and the rigor on the other side and how those two work in tandem, to me, that just feels like kind of a perfect way to end the conversation, because this is a time of great experimentation and it's also a time of great change and uncertainty. I think having both of those pieces working together... And then, also of course, what I really appreciate about you is the humility to say, "Yes, this is a lot." This is a lot, but you're still working through it and you're still open and you're still learning, and that's, I think, what's really inspired me to have this conversation with you today. So, thank you so, so much.
Cristina Pieretti (31:30):
Thank you. The pleasure was mine, Susan. I'm very passionate about learning, as I can see you are as well. I think if I always have an advice for people that are coming after us is never stop learning, because change is a constant. It's always going to be there, and we have to keep ourselves open to learning.
Susan Etlinger (31:49):
Fabulous. Thank you again.
Cristina Pieretti (31:51):
Great. Thank you very much.
Susan Etlinger (31:59):
Wow. Here we are in 2025, and it kind of blows my mind that we're already looking back at what we've learned, and Cristina's kind of the perfect guest to do that with. So, here are a couple of things I took away from our conversation that I think are kind of helpful if you're exploring, planning, or implementing gen AI. First, let's talk about building interest from the start. Cristina's team took multiple measures to reduce risk, including grounding responses in Moody's data, telling the model only to use data provided by Moody's and say "I don't know" when it didn't have an answer, and expanding data sources really carefully.
I was also struck by how strategically she and her team approached generative AI. Their focus on what it meant for them as a company in addition to the operational efficiencies is a really multi-layered way of thinking. And equally as important is the focus on customer engagement and feedback. The team's commitment to engaging early in the process and devoting resources to showing customers the art of the possible gave them insights that ultimately turned into product features. That's what we mean when we talk about that flywheel effective AI.
We hope you've enjoyed this episode, and as I'm sure you know, new podcasts live and die on engagement. So, please like, comment, share, tell your friends, and let us know what you'd like to hear about. We're listening. If you'd like to learn more about how people and organizations are innovating with Microsoft Azure, visit azure.microsoft.com.
The Leading the Shift podcast is a place for experts to share their insights and opinions. As students of the future of technology, Microsoft values inputs from a diverse set of voices. That said, the opinions and findings of our guests are their own, and they may not necessarily reflect Microsoft's own research or positions.
Leading the Shift is a production of Microsoft Azure. I'm your host, Susan Etlinger. Our executive producers are Hailie Meehan, Aaron Russell, Erik Williams, and me. Production support provided by the incredible team at Indigo Slate.
[END]
