00:00:15
Speaker 1: Welcome to Inside Active, a podcast about active managers that goes beyond sound bites and headlines and looks deeper into their processes, challenges, and philosophies and security selection. I'm David cone I, lead mutual fund and active Research at Bloomberg Intelligence. Selecting active managers has been one of the biggest challenges in investing. While performance, risk, attribution, and other portfolio metrics can tell us how fun performed, they don't necessarily tell us whether that performance was the result of repeatable skill or simply favorable market conditions. For institutional investors, consultants, and financial advisors, separating genuine investment skill from luck remains one of the industry's most difficult and most important questions. Recent advances in portfolio analytics are challenging some long held assumption about how active managers should be evaluated. Rather than focusing primarily on investment outcomes, a growing body of research suggests investors may gain deeper insights by examining the decision makers, what they buy, what they sell, how they seize positions, and how consistently those decisions add value over time. So today I wanted to explore how investors should think about measuring manager's skill, why traditional performance metrics may not tell the full story, and whether a more decision based approach could improve active manager selection. So join me to discuss that is Michael Ervolini, author of Skill Versus Luck taking the guests out of Equity fund selection. Mike, thanks for joining me today.
00:01:39
Speaker 2: It's a pleasure, David, and I appreciate the opportunity to speak with you and your audience great.
00:01:45
Speaker 1: So what problem were you trying to solve when you wrote this book Skill Versus Luck?
00:01:50
Speaker 2: Fundamentally, it was helping the industry overcome conventional thinking. It's a challenge in every industry trying to move forward rather than holding fast to the current activities and beliefs we have. And as you mentioned in your intro, there's quite a few activities going on right now where people are using decision based analytics to give better insights into manager skill. That area hasn't received as much traction as it needs, and so in writing the book, I was hoping to stimulate an industry wide conversation about why we need to go beyond outcome based analytics and look at decision based analytics in combination so that people can get a deeper and richer understanding of what skills are driving the fund results, what skills are persistent, and all of that can allow investors to be more confident in selecting managers that have a greater likelihood of outperforming in the future. And really that's what cation decisions are all about, is who you want to give your money to in order to get better returns going forward.
00:03:08
Speaker 1: So you're the title of the book kind of suggests we consistently are constantly confused luck with skill. Why do you think intelligent people make that mistake so often?
00:03:19
Speaker 2: Well, two points on that one is I really believe that skill and luck are just opposite ends of a continuum, and so when a fund manager has more skill, they're less dependent upon luck for their results. So that's part one is skill and luck are involved in every decision. You just want managers that are more skilled. But a more specific answer to your question is, when you're making decisions using conventional analytics, they bring you to the threshold of understanding skill, but then you need to make a jump. You need to overinterpret the results or see in them something that you want to see. In other words, you have to guess a little bit. And I think because of that inadequacy of conventional analytics at addressing skill it makes it hard to figure out which managers are skilled and which aren't, and that has been plaguing the industry for a while. It continues to do so. But I have worked with institutional investors in particular who've been using decision based analytics, and their ability to sort out superior managers has improved as a result of this. So we can get past this. But most of the confusion has to do with working with inferior analytics as opposed to people not trying their best.
00:04:46
Speaker 1: So do you think we're kind of wired to judge outcomes rather than decisions.
00:04:50
Speaker 2: Yeah. One of the confusions that we have is how we make decisions. And what is not commonly understood is that more than ninety five percent the decisions we make every day are made completely automatically by our unconscious and so a lot of what's happening, whether it's a buying a stock, picking a manager, or ordering something off the menu, these automatic decisions generally work. But they only work really well where there's been great feedback about how good the decision is and how good the outcome has been. And so in that automatic decision environment. When you're using outcome based analytics, you see in them what you want, and so I think when asset owners or allocators are shown decision based results against outcome based results, they can take the best of both. But lacking any kind of decision based analytics and using only outcome based analytics, our natural instinct is to see in those results something that seems reason and often what we see is completely wrong.
00:06:03
Speaker 1: Okay, and you know you redefine skill as the combined effect of expert judgment and investment processes. Why did you feel the industry needed a new definition?
00:06:15
Speaker 2: Well, I think partly is because we had been looking at skill the wrong way. We've been looking at it as outcomes, and when you look at that, it's hard to figure out what's in there. So if we're going to look at decisions, then we want to focus on what is it the manager brings to the fund and the manager brings to qualities, their expert judgment and their investment process, and that's what they should be judged on. And fortunately, decision based analytics can allow you to rigorously evaluate and quantify both of those elements judgment and process, and when you understand them well you can then get a sense of how consistently good the men is making buying, selling, sizing decisions, and using that, I think what we've seen in the industry is that people then have more confidence that the manager they're working with, well not perfect, has a better than average chance about performing going forward.
00:07:19
Speaker 1: So then how do you actually separate you know, a good decision from a lucky outcome, Like, how do you go in and do that?
00:07:25
Speaker 2: Well, there's always going to be some luck. Okay, that's just we'll call that the market.
00:07:31
Speaker 1: You know, if you were.
00:07:32
Speaker 2: Lucky enough to buy the Magnificent seven some years ago and overweight them extensively, part of that was a good decision, and part of that is they've just done remarkably well. So there's a little bit of luck, or if your style is in favor and so forth. But given that we can look at individual decisions, let's say a buying decision. What we can do is over a period of time, we use ten years just for easy number. We look at every position that was purchased over that ten year period, and then we evaluate, independently from how the manager sized a position or when they sold it, on average, how did those positions do in aggregate that they generate excess return or not on their own. And so we can do that for sizing, we can do that for adding on the way up, adding on the way down. We can look at decisions broadly or very very narrowly, and the idea is to isolate individual types of decisions and evaluate if they, on average are helping to generate more excess return or in fact are taking away excess return, which is obviously a negative skill.
00:08:52
Speaker 1: Okay, and you know, one of the most interesting concepts I think in the book is the counterfactual portfolio. Can you kind of explain that idea and plan English.
00:09:01
Speaker 2: Sure, a counterfactual portfolio is simply a copy of the actual portfolio, but then one decision type is adjusted to understand its impact on the portfolio. So a very straightforward example is this, there's a fund where the manager periodically purchases stocks that are outside the benchmark of the fund, and presumably these out of benchmark purchases are intended to generate excess return or alpha. Well, one way to evaluate that is to create a copy of the fund and then go through it every day, and whenever the manager buys a position that's outside the benchmark. Just simply eliminate that from the copy or the counterfactual. And so when you're done doing that exercise, the counterfactual portfolio on every single day owns exactly what's in the actual portfolio, except it doesn't own any out of benchmark positions. And now with that counterfactual portfolio, you generate its returns and its excess return. You can do information ratios, whatever metrics you want, and compare those to the actual. Now, if we're looking at relative return, if the actual portfolio has a higher relative return than the counterfactual, that tells us owning those out of benchmark positions really helped it generated more excess return. Conversely, if the actual portfolio underperforms the counterfactual, it tells us that owning those out of benchmark positions actually hurt the results. So that's the idea of a counterfactual. You take one decision type at a time, you either eliminate it or alter it, and then compare it back to the actual and that way you're able to see whether specific decisions are additive or detracting from overall success in the fund.
00:11:15
Speaker 1: Now it makes sense, it makes a lot of sense. It's really interesting. You know. I also couldn't help but think that, you know, what you're looking at applies even beyond investing. Do you think your framework could be used to evaluate, you know, even CEOs or even sports coaches.
00:11:32
Speaker 2: I'm not sure I've thought about that. It's possible. It definitely was meant to be used across any publicly treated asset type, that's for sure, so option swaps, fixed income. But beyond that, we've never tackled anything like that, So I'm not sure it's possible, But I'm not sure.
00:11:58
Speaker 1: It's definitely interesting concept. And you know, obviously the biggest, you know, one of the biggest criticisms you've had is allocators spend way too much time evaluating performance, not enough time evaluating decisions. If you were sitting on an investment community today, what would you stop measuring and what would you start measuring?
00:12:16
Speaker 2: Well, I'm not sure I would stop measuring anything, but I would add a whole lot of decision based analytics. I would want to know how effective the manager is at buying, selling, and sizing, and then I would want to look at them from multiple perspectives. I would want to look, let's say, ten years again, how good was buying over ten years, how good was it year by year to see how consistent it was, How good is the buying across sectors, to make sure that we're not just leaning on a few sectors, but the manager's good across the board. And I might want to look at it by factor. If there's a certain element in the strategy that the managers expressing in their coversations or their marketing material about we look for stocks with these characteristics, I would want to use those factors to evaluate the buying as well. As an example, if the manager says they'd like to buy high quality companies, well, we could use any number of factors to evaluate the quality of a company balance sheet, debt, or whatever we want to do and see if in fact, higher quality balance sheet companies show up as being the stronger buys in the portfolio. So I'd want to do a lot of work like that, And when you get to that place, you know that the manager is doing what they said they're going to do, They're doing it very well every year and over time, it's really adding value to the fund. That, in combination with conventional analytics, allows you to know that over the past few years the results have been great too. The amount of risk taking was not disproportional to the return, and they're well within their peer group, and all that kind of stuff. All these conventional analytics are important, they simply don't provide enough information about skill.
00:14:11
Speaker 1: Okay, you know you obviously spent decades looking at how professional investors make decisions. What do you think the biggest misconception people have about how decisions are actually made?
00:14:22
Speaker 2: Well, I mentioned one, which is this unconscious bit thinking. Recognizing that most of your decisions are unconscious really convinces me and a lot of other people that it's imperative that you have a strong process because that can arrest some of the automatic decision making and make sure that you stay in the area where you make your best decisions. Another misconception is emotions. We hear frequently, at least I have in my career an investor or money manager or other profession I will say, well, we take all the emotions out of our decision making. We do everything here by the numbers. Well, that sounds great, but it's actually impossible. What neuroscience tells us is it's literally impossible to make a rational choice without emotions. You can add up a column of numbers without emotions because there's no judgment involved. You're just adding numbers. But when you're trying to make a choice, what discount rates to use? What fund am I ultimately going to pick for my allocation? What research am I going to use to help me get to that place? Those choices, you get to difficult spots where it's not easy to differentiate one option from another, and a little emotional spark is what helps in the tie breakers. And so understanding that your emotions play a role in your decision making is another reason you want to have a good process because it can slow you down and allow you to arrest. Automatic decision making allow you to harness emotions in a positive way. And when you embrace those two concepts and use them to help build your process, that result is you get much better decision making.
00:16:18
Speaker 1: Do you think there's you know, a danger we've become obsessed with measuring everything, you know, in other words, can skill itself become over engineered.
00:16:26
Speaker 2: I think we can overmeasure things. Yeah, we can lose sight of what we're really after. And if that happens, it's because we didn't have a good plan going in. You know, we have to understand why we're measuring skill and what we want to come out of it. But I will caution this frequently, the fear of overmeasurement can be used as an argument to hang onto convention, to not take a step forward to state where we are more comfortable in presenting these analytics. To some fund managers, sometimes the pushback is, well, you know, I don't want you coming in and messing up what I'm doing now. Well, so I don't want to see this stuff now. I think there's a little fear in there about learning what you do well and what you don't. So there's a balance point. Yes, we can't go overboard and measure too much or follow analytics too far, because no analytic is perfect. But reasonable people can take on new analytics and learn to make better decisions. And that's what it's all about. Being proportional, being reasonable and understanding that the purpose here is to make better decisions, not just to measure things or to follow analytics one hundred percent.
00:17:40
Speaker 1: You know your approach depends on analyzing thousands of decisions. Could someone argue that investing is still too uncertain to isolate skill.
00:17:49
Speaker 2: I don't think so now, because I myself have been involved in analyzing four or five six hundred funds, and there's no question that skill can be measured. And there are other companies out there that have collectively looked at perhaps eight or nine thousand funds, and so skill can be measured, can be isolated, can be quantified. It's done differently by different analytics shops, but there's no question that can be isolated. It simply can't be done using conventional analytics.
00:18:23
Speaker 1: Okay, you know Warren Buffett has said that you know, temperament matters more than IQ. Does your framework capture qualities like patients, discipline, conviction or is it only you know, kind of measurable decisions.
00:18:37
Speaker 2: No, In fact, it can help you determine some of those things. One example, let's say patients. Okay, we could look at when a manager is buys stocks and they take off relatively early after purchase. Does the manager hang on to them or does the manager sell them far too quickly and give up a lot of alpha. Now there's a patience issue. So by looking at new winners and evaluating whether they're being sold too quickly or not, that's a way of getting into patients conviction. We can look at when positions are purchased and maybe in the first few months they take a dip in price. Sometimes an early dip in price is an indication that the manager bought a bad stock. Other times it's just that the market has had a bad reaction to that company, but the fundamentals really haven't changed, and more often than not, those stocks are going to bounce back. And by evaluating that behavior, we can see whether the manager has sufficient conviction in stocks that wiggle a little bit right after they're purchased, but ultimately do really, really well. So we can look for behavioral kinds of aspects in the manager looking at the decisions they make.
00:20:02
Speaker 1: Okay, you know another thing that you know kind of really jumped out was that you know you've concluded that buying is usually the biggest drive of alpha. Why do you think buying is more important than selling. Well, if you're not.
00:20:14
Speaker 2: A good buyer, you're basically starting out in a hole, and it's hard to sell your way out of a hole and generate alpha. It can be done, but it's really hard. But if you start out buying good stocks, you can have a few mistakes on sizing and selling and still do pretty well, so it's just a question of starting off on the strongest platform possible. And that's with good buying. In general speaking, most successful managers are really really good buyers. There's a few that are not, but and I can explain why if that's of interest, But generally speaking, they're good buyers.
00:20:56
Speaker 1: So that kind of brings me to the next part is you know the you argue that selling is probably that the industry's weakest skill. You know, and when I speak to managers, I always I don't want to know just what their buying philosophy is, but you know what makes them sell? And so what do you think a lot of professionals struggle so much? Was selling?
00:21:13
Speaker 2: It's the least understood. If you were to do a quick search on equity investing strategy, everything you.
00:21:22
Speaker 1: Found would be about buying.
00:21:26
Speaker 2: When it so that's the whole industry in academy itself, when they think about the strategic decisions inside of equity investing or any other asset class, it's generally they're looking at buying. Selling is often a discipline. Well, if buying is strategic and selling is a discipline, it sort of tells you that selling is sort of the orphan of thinking in asset management, and so it's not well understood. The second thing is what is a good sell. Often people believe that a good sell is after I sell the stock the price goes down. Well, that's not necessarily a good description. A better one is A good sell is when you sell a stock and after you sell it, its return is less than the fund's average. So, in other words, you want to hang onto stocks that can lift up the fund average return, and you want to sell stocks that will take down the fund's average return, meaning that if the market is going down, holding onto a stock that only goes down five percent when a lot of other stocks are going down ten percent, well you shouldn't sell that stock even though it's priced going down. So we get confused a little bit about what a good sell is. And again it's selling a stock before it drags down the average. That's the fundamental test. And then the fact that academia and other analytics really don't look at selling particularly well. Conventional analytics are really nonexistent in their ability to help you understand selling. The decision based analytics do a real nice job at looking at selling selling when stocks are up selling when stocks are down. You can look at selling all different manners and really understand it. But most managers do not have good analytics to help them understand their selling, and that's why they try their best, but they really can't improve it because they don't have the right feedback.
00:23:42
Speaker 1: Okay, and you know, if you had to pick one investment habit, you know you think every active manager should stop doing tomorrow. What do you think that would be?
00:23:51
Speaker 2: Reaching for conviction too quickly? Conviction is an important element. When you talk to managers, they talk about it. When consultants took the managers, they ask them about their conviction. Conviction is a well bandeed word in the industry and it's important.
00:24:09
Speaker 1: But what the.
00:24:10
Speaker 2: Research shows is coming to it slowly and allowing it to develop, particularly using your analysts, using the investment committee, using your research team. Allowing conviction to develop slowly allows the manager to be more confident that they're really dealing with well airned conviction and not just bluster. And telling the difference when you're excited is really really hard. So that again brings me back to why process is so important. Getting to a discipline process where you can let your unconscious do its best job. Let your emotions help you as much as they can, but let others in your organization offer questions, criticism, or support. All of that allows the manager to develop real fiction and make their best decisions.
00:25:03
Speaker 1: Okay, you know we're obviously entering an error where AI can analyze enormous amounts of information. Do you think AI makes skill more valuable or less valuable?
00:25:16
Speaker 2: Right now? There's no question it makes skill more valuable. There's a recent research report from Lauren Cohen out of Harvard just came out in the past seven or ten days where they use some large language models to mimic a number of mutual funds over an extended period of time, and what they found out is that these largers language models were really really good at anticipating something between seventy five and eighty percent of the stocks that the fund actually purchased over time. What they also found is that the stock the large language models didn't buy were the ones that contained most of the alpha. So there's something about human judgment at the edge of the marketplace that is pretty impressive today. It allows people to really do some spectacular stuff. Will that always be the case, I don't know, but my sense is markets move so fast, humans are always going to have a role to play. One other supporting piece of evidence I can offer is back in the Great Crash and eight nine, I remember being at a conference of chief investment officers and some of the risk managers, global risk managers, and what they were saying is, at that moment they abandoned their models because they weren't trained on such an extreme situation. So to the extent things move rapidly and we see extreme activities going on, I think humans are going to be a little bit smarter because they can see and pull things together a little faster today. Again, who knows what's going to happen years to come, But right now there's some evidence that AI can make humans do even better. It's not clear that they can supplant them completely.
00:27:29
Speaker 1: Okay, And kind of on that note or a little bit, you know, kind of going away from AI. But you know, if we talk about ten years from now, if you rewrote this book in ten years, what do you think would change.
00:27:40
Speaker 2: Well, my challenge, as I said earlier, is to overcome convention. So what would be great to be able to write is that overcoming convention was not as hard and didn't take as long as I originally thought that would be a great thing to be able to write ten years. No, that's my hope. I also think we'll see, without question, a more greater adoption of decision based analytics, and I think we will see a lot more AI being used in the industry, and hopefully we'll see the opportunity for active management to become stronger, and we may find a different balance point between active and passive investing. Right now, I think there's such a challenge in identifying a skilled manager that many investors are choosing to allocate more capital passively simply because they're afraid of picking the wrong manager. But if we can make skill as transparent as it needs to be and as it can be, then I think that offers the opportunity for a different balance point between active and passive, and that's another benefit we get by going down this road of improving our skilled metrics.
00:29:07
Speaker 1: Okay, I've got just one more question before I let you go. Every investor listening right now, remember just one idea from skill versus luck. What would you hope it would be that.
00:29:18
Speaker 2: There are skilled managers out there. There are plenty of them, the difficulty is finding them. So whether you're an institutional investor, a manager search consultant, or an individual, what it's incumbent upon you to do is to begin to demand better skill metrics from whomever you work with, whether it's an internal team, if you're a pension fund, or whether it's a manager search firm, if you're a large investor, or if you're an individual, whatever platform you're using, start demanding better skill metrics. They exist, they're relatively inexpensive, and they add tremendously to your ability to make better allocation decisions. And what we're all trying to do is meet some financial need in the future, whether it's retirement or an endowment, or making investments in the community or in healthcare or whatever it is. The better we do with our investments, the better off we all are. And asking and demanding for better skill analytics can allow you to make better investment decisions. And that's certainly what I would be pushing for right now, is we need to change the industry. And it starts by saying, hey, why don't we have some different skill metrics on this platform.
00:30:44
Speaker 1: That we're using. Well, this is great. I really learned a lot and enjoyed speaking with you. And so, Mike, thank you again for joining me today. Well, thank you very much.
00:30:53
Speaker 2: It's my pleasure.
00:30:54
Speaker 1: I also want to thank our listeners. If you like the episode, please share it, subscribe and leave a review. If you'd like to see more of our research on the terminal, go to BI Fund, go for US Fund and Active Research until our next episode. This is David Cone with Inside Active
Speaker 1: Welcome to Inside Active, a podcast about active managers that goes beyond sound bites and headlines and looks deeper into their processes, challenges, and philosophies and security selection. I'm David cone I, lead mutual fund and active Research at Bloomberg Intelligence. Selecting active managers has been one of the biggest challenges in investing. While performance, risk, attribution, and other portfolio metrics can tell us how fun performed, they don't necessarily tell us whether that performance was the result of repeatable skill or simply favorable market conditions. For institutional investors, consultants, and financial advisors, separating genuine investment skill from luck remains one of the industry's most difficult and most important questions. Recent advances in portfolio analytics are challenging some long held assumption about how active managers should be evaluated. Rather than focusing primarily on investment outcomes, a growing body of research suggests investors may gain deeper insights by examining the decision makers, what they buy, what they sell, how they seize positions, and how consistently those decisions add value over time. So today I wanted to explore how investors should think about measuring manager's skill, why traditional performance metrics may not tell the full story, and whether a more decision based approach could improve active manager selection. So join me to discuss that is Michael Ervolini, author of Skill Versus Luck taking the guests out of Equity fund selection. Mike, thanks for joining me today.
00:01:39
Speaker 2: It's a pleasure, David, and I appreciate the opportunity to speak with you and your audience great.
00:01:45
Speaker 1: So what problem were you trying to solve when you wrote this book Skill Versus Luck?
00:01:50
Speaker 2: Fundamentally, it was helping the industry overcome conventional thinking. It's a challenge in every industry trying to move forward rather than holding fast to the current activities and beliefs we have. And as you mentioned in your intro, there's quite a few activities going on right now where people are using decision based analytics to give better insights into manager skill. That area hasn't received as much traction as it needs, and so in writing the book, I was hoping to stimulate an industry wide conversation about why we need to go beyond outcome based analytics and look at decision based analytics in combination so that people can get a deeper and richer understanding of what skills are driving the fund results, what skills are persistent, and all of that can allow investors to be more confident in selecting managers that have a greater likelihood of outperforming in the future. And really that's what cation decisions are all about, is who you want to give your money to in order to get better returns going forward.
00:03:08
Speaker 1: So you're the title of the book kind of suggests we consistently are constantly confused luck with skill. Why do you think intelligent people make that mistake so often?
00:03:19
Speaker 2: Well, two points on that one is I really believe that skill and luck are just opposite ends of a continuum, and so when a fund manager has more skill, they're less dependent upon luck for their results. So that's part one is skill and luck are involved in every decision. You just want managers that are more skilled. But a more specific answer to your question is, when you're making decisions using conventional analytics, they bring you to the threshold of understanding skill, but then you need to make a jump. You need to overinterpret the results or see in them something that you want to see. In other words, you have to guess a little bit. And I think because of that inadequacy of conventional analytics at addressing skill it makes it hard to figure out which managers are skilled and which aren't, and that has been plaguing the industry for a while. It continues to do so. But I have worked with institutional investors in particular who've been using decision based analytics, and their ability to sort out superior managers has improved as a result of this. So we can get past this. But most of the confusion has to do with working with inferior analytics as opposed to people not trying their best.
00:04:46
Speaker 1: So do you think we're kind of wired to judge outcomes rather than decisions.
00:04:50
Speaker 2: Yeah. One of the confusions that we have is how we make decisions. And what is not commonly understood is that more than ninety five percent the decisions we make every day are made completely automatically by our unconscious and so a lot of what's happening, whether it's a buying a stock, picking a manager, or ordering something off the menu, these automatic decisions generally work. But they only work really well where there's been great feedback about how good the decision is and how good the outcome has been. And so in that automatic decision environment. When you're using outcome based analytics, you see in them what you want, and so I think when asset owners or allocators are shown decision based results against outcome based results, they can take the best of both. But lacking any kind of decision based analytics and using only outcome based analytics, our natural instinct is to see in those results something that seems reason and often what we see is completely wrong.
00:06:03
Speaker 1: Okay, and you know you redefine skill as the combined effect of expert judgment and investment processes. Why did you feel the industry needed a new definition?
00:06:15
Speaker 2: Well, I think partly is because we had been looking at skill the wrong way. We've been looking at it as outcomes, and when you look at that, it's hard to figure out what's in there. So if we're going to look at decisions, then we want to focus on what is it the manager brings to the fund and the manager brings to qualities, their expert judgment and their investment process, and that's what they should be judged on. And fortunately, decision based analytics can allow you to rigorously evaluate and quantify both of those elements judgment and process, and when you understand them well you can then get a sense of how consistently good the men is making buying, selling, sizing decisions, and using that, I think what we've seen in the industry is that people then have more confidence that the manager they're working with, well not perfect, has a better than average chance about performing going forward.
00:07:19
Speaker 1: So then how do you actually separate you know, a good decision from a lucky outcome, Like, how do you go in and do that?
00:07:25
Speaker 2: Well, there's always going to be some luck. Okay, that's just we'll call that the market.
00:07:31
Speaker 1: You know, if you were.
00:07:32
Speaker 2: Lucky enough to buy the Magnificent seven some years ago and overweight them extensively, part of that was a good decision, and part of that is they've just done remarkably well. So there's a little bit of luck, or if your style is in favor and so forth. But given that we can look at individual decisions, let's say a buying decision. What we can do is over a period of time, we use ten years just for easy number. We look at every position that was purchased over that ten year period, and then we evaluate, independently from how the manager sized a position or when they sold it, on average, how did those positions do in aggregate that they generate excess return or not on their own. And so we can do that for sizing, we can do that for adding on the way up, adding on the way down. We can look at decisions broadly or very very narrowly, and the idea is to isolate individual types of decisions and evaluate if they, on average are helping to generate more excess return or in fact are taking away excess return, which is obviously a negative skill.
00:08:52
Speaker 1: Okay, and you know, one of the most interesting concepts I think in the book is the counterfactual portfolio. Can you kind of explain that idea and plan English.
00:09:01
Speaker 2: Sure, a counterfactual portfolio is simply a copy of the actual portfolio, but then one decision type is adjusted to understand its impact on the portfolio. So a very straightforward example is this, there's a fund where the manager periodically purchases stocks that are outside the benchmark of the fund, and presumably these out of benchmark purchases are intended to generate excess return or alpha. Well, one way to evaluate that is to create a copy of the fund and then go through it every day, and whenever the manager buys a position that's outside the benchmark. Just simply eliminate that from the copy or the counterfactual. And so when you're done doing that exercise, the counterfactual portfolio on every single day owns exactly what's in the actual portfolio, except it doesn't own any out of benchmark positions. And now with that counterfactual portfolio, you generate its returns and its excess return. You can do information ratios, whatever metrics you want, and compare those to the actual. Now, if we're looking at relative return, if the actual portfolio has a higher relative return than the counterfactual, that tells us owning those out of benchmark positions really helped it generated more excess return. Conversely, if the actual portfolio underperforms the counterfactual, it tells us that owning those out of benchmark positions actually hurt the results. So that's the idea of a counterfactual. You take one decision type at a time, you either eliminate it or alter it, and then compare it back to the actual and that way you're able to see whether specific decisions are additive or detracting from overall success in the fund.
00:11:15
Speaker 1: Now it makes sense, it makes a lot of sense. It's really interesting. You know. I also couldn't help but think that, you know, what you're looking at applies even beyond investing. Do you think your framework could be used to evaluate, you know, even CEOs or even sports coaches.
00:11:32
Speaker 2: I'm not sure I've thought about that. It's possible. It definitely was meant to be used across any publicly treated asset type, that's for sure, so option swaps, fixed income. But beyond that, we've never tackled anything like that, So I'm not sure it's possible, But I'm not sure.
00:11:58
Speaker 1: It's definitely interesting concept. And you know, obviously the biggest, you know, one of the biggest criticisms you've had is allocators spend way too much time evaluating performance, not enough time evaluating decisions. If you were sitting on an investment community today, what would you stop measuring and what would you start measuring?
00:12:16
Speaker 2: Well, I'm not sure I would stop measuring anything, but I would add a whole lot of decision based analytics. I would want to know how effective the manager is at buying, selling, and sizing, and then I would want to look at them from multiple perspectives. I would want to look, let's say, ten years again, how good was buying over ten years, how good was it year by year to see how consistent it was, How good is the buying across sectors, to make sure that we're not just leaning on a few sectors, but the manager's good across the board. And I might want to look at it by factor. If there's a certain element in the strategy that the managers expressing in their coversations or their marketing material about we look for stocks with these characteristics, I would want to use those factors to evaluate the buying as well. As an example, if the manager says they'd like to buy high quality companies, well, we could use any number of factors to evaluate the quality of a company balance sheet, debt, or whatever we want to do and see if in fact, higher quality balance sheet companies show up as being the stronger buys in the portfolio. So I'd want to do a lot of work like that, And when you get to that place, you know that the manager is doing what they said they're going to do, They're doing it very well every year and over time, it's really adding value to the fund. That, in combination with conventional analytics, allows you to know that over the past few years the results have been great too. The amount of risk taking was not disproportional to the return, and they're well within their peer group, and all that kind of stuff. All these conventional analytics are important, they simply don't provide enough information about skill.
00:14:11
Speaker 1: Okay, you know you obviously spent decades looking at how professional investors make decisions. What do you think the biggest misconception people have about how decisions are actually made?
00:14:22
Speaker 2: Well, I mentioned one, which is this unconscious bit thinking. Recognizing that most of your decisions are unconscious really convinces me and a lot of other people that it's imperative that you have a strong process because that can arrest some of the automatic decision making and make sure that you stay in the area where you make your best decisions. Another misconception is emotions. We hear frequently, at least I have in my career an investor or money manager or other profession I will say, well, we take all the emotions out of our decision making. We do everything here by the numbers. Well, that sounds great, but it's actually impossible. What neuroscience tells us is it's literally impossible to make a rational choice without emotions. You can add up a column of numbers without emotions because there's no judgment involved. You're just adding numbers. But when you're trying to make a choice, what discount rates to use? What fund am I ultimately going to pick for my allocation? What research am I going to use to help me get to that place? Those choices, you get to difficult spots where it's not easy to differentiate one option from another, and a little emotional spark is what helps in the tie breakers. And so understanding that your emotions play a role in your decision making is another reason you want to have a good process because it can slow you down and allow you to arrest. Automatic decision making allow you to harness emotions in a positive way. And when you embrace those two concepts and use them to help build your process, that result is you get much better decision making.
00:16:18
Speaker 1: Do you think there's you know, a danger we've become obsessed with measuring everything, you know, in other words, can skill itself become over engineered.
00:16:26
Speaker 2: I think we can overmeasure things. Yeah, we can lose sight of what we're really after. And if that happens, it's because we didn't have a good plan going in. You know, we have to understand why we're measuring skill and what we want to come out of it. But I will caution this frequently, the fear of overmeasurement can be used as an argument to hang onto convention, to not take a step forward to state where we are more comfortable in presenting these analytics. To some fund managers, sometimes the pushback is, well, you know, I don't want you coming in and messing up what I'm doing now. Well, so I don't want to see this stuff now. I think there's a little fear in there about learning what you do well and what you don't. So there's a balance point. Yes, we can't go overboard and measure too much or follow analytics too far, because no analytic is perfect. But reasonable people can take on new analytics and learn to make better decisions. And that's what it's all about. Being proportional, being reasonable and understanding that the purpose here is to make better decisions, not just to measure things or to follow analytics one hundred percent.
00:17:40
Speaker 1: You know your approach depends on analyzing thousands of decisions. Could someone argue that investing is still too uncertain to isolate skill.
00:17:49
Speaker 2: I don't think so now, because I myself have been involved in analyzing four or five six hundred funds, and there's no question that skill can be measured. And there are other companies out there that have collectively looked at perhaps eight or nine thousand funds, and so skill can be measured, can be isolated, can be quantified. It's done differently by different analytics shops, but there's no question that can be isolated. It simply can't be done using conventional analytics.
00:18:23
Speaker 1: Okay, you know Warren Buffett has said that you know, temperament matters more than IQ. Does your framework capture qualities like patients, discipline, conviction or is it only you know, kind of measurable decisions.
00:18:37
Speaker 2: No, In fact, it can help you determine some of those things. One example, let's say patients. Okay, we could look at when a manager is buys stocks and they take off relatively early after purchase. Does the manager hang on to them or does the manager sell them far too quickly and give up a lot of alpha. Now there's a patience issue. So by looking at new winners and evaluating whether they're being sold too quickly or not, that's a way of getting into patients conviction. We can look at when positions are purchased and maybe in the first few months they take a dip in price. Sometimes an early dip in price is an indication that the manager bought a bad stock. Other times it's just that the market has had a bad reaction to that company, but the fundamentals really haven't changed, and more often than not, those stocks are going to bounce back. And by evaluating that behavior, we can see whether the manager has sufficient conviction in stocks that wiggle a little bit right after they're purchased, but ultimately do really, really well. So we can look for behavioral kinds of aspects in the manager looking at the decisions they make.
00:20:02
Speaker 1: Okay, you know another thing that you know kind of really jumped out was that you know you've concluded that buying is usually the biggest drive of alpha. Why do you think buying is more important than selling. Well, if you're not.
00:20:14
Speaker 2: A good buyer, you're basically starting out in a hole, and it's hard to sell your way out of a hole and generate alpha. It can be done, but it's really hard. But if you start out buying good stocks, you can have a few mistakes on sizing and selling and still do pretty well, so it's just a question of starting off on the strongest platform possible. And that's with good buying. In general speaking, most successful managers are really really good buyers. There's a few that are not, but and I can explain why if that's of interest, But generally speaking, they're good buyers.
00:20:56
Speaker 1: So that kind of brings me to the next part is you know the you argue that selling is probably that the industry's weakest skill. You know, and when I speak to managers, I always I don't want to know just what their buying philosophy is, but you know what makes them sell? And so what do you think a lot of professionals struggle so much? Was selling?
00:21:13
Speaker 2: It's the least understood. If you were to do a quick search on equity investing strategy, everything you.
00:21:22
Speaker 1: Found would be about buying.
00:21:26
Speaker 2: When it so that's the whole industry in academy itself, when they think about the strategic decisions inside of equity investing or any other asset class, it's generally they're looking at buying. Selling is often a discipline. Well, if buying is strategic and selling is a discipline, it sort of tells you that selling is sort of the orphan of thinking in asset management, and so it's not well understood. The second thing is what is a good sell. Often people believe that a good sell is after I sell the stock the price goes down. Well, that's not necessarily a good description. A better one is A good sell is when you sell a stock and after you sell it, its return is less than the fund's average. So, in other words, you want to hang onto stocks that can lift up the fund average return, and you want to sell stocks that will take down the fund's average return, meaning that if the market is going down, holding onto a stock that only goes down five percent when a lot of other stocks are going down ten percent, well you shouldn't sell that stock even though it's priced going down. So we get confused a little bit about what a good sell is. And again it's selling a stock before it drags down the average. That's the fundamental test. And then the fact that academia and other analytics really don't look at selling particularly well. Conventional analytics are really nonexistent in their ability to help you understand selling. The decision based analytics do a real nice job at looking at selling selling when stocks are up selling when stocks are down. You can look at selling all different manners and really understand it. But most managers do not have good analytics to help them understand their selling, and that's why they try their best, but they really can't improve it because they don't have the right feedback.
00:23:42
Speaker 1: Okay, and you know, if you had to pick one investment habit, you know you think every active manager should stop doing tomorrow. What do you think that would be?
00:23:51
Speaker 2: Reaching for conviction too quickly? Conviction is an important element. When you talk to managers, they talk about it. When consultants took the managers, they ask them about their conviction. Conviction is a well bandeed word in the industry and it's important.
00:24:09
Speaker 1: But what the.
00:24:10
Speaker 2: Research shows is coming to it slowly and allowing it to develop, particularly using your analysts, using the investment committee, using your research team. Allowing conviction to develop slowly allows the manager to be more confident that they're really dealing with well airned conviction and not just bluster. And telling the difference when you're excited is really really hard. So that again brings me back to why process is so important. Getting to a discipline process where you can let your unconscious do its best job. Let your emotions help you as much as they can, but let others in your organization offer questions, criticism, or support. All of that allows the manager to develop real fiction and make their best decisions.
00:25:03
Speaker 1: Okay, you know we're obviously entering an error where AI can analyze enormous amounts of information. Do you think AI makes skill more valuable or less valuable?
00:25:16
Speaker 2: Right now? There's no question it makes skill more valuable. There's a recent research report from Lauren Cohen out of Harvard just came out in the past seven or ten days where they use some large language models to mimic a number of mutual funds over an extended period of time, and what they found out is that these largers language models were really really good at anticipating something between seventy five and eighty percent of the stocks that the fund actually purchased over time. What they also found is that the stock the large language models didn't buy were the ones that contained most of the alpha. So there's something about human judgment at the edge of the marketplace that is pretty impressive today. It allows people to really do some spectacular stuff. Will that always be the case, I don't know, but my sense is markets move so fast, humans are always going to have a role to play. One other supporting piece of evidence I can offer is back in the Great Crash and eight nine, I remember being at a conference of chief investment officers and some of the risk managers, global risk managers, and what they were saying is, at that moment they abandoned their models because they weren't trained on such an extreme situation. So to the extent things move rapidly and we see extreme activities going on, I think humans are going to be a little bit smarter because they can see and pull things together a little faster today. Again, who knows what's going to happen years to come, But right now there's some evidence that AI can make humans do even better. It's not clear that they can supplant them completely.
00:27:29
Speaker 1: Okay, And kind of on that note or a little bit, you know, kind of going away from AI. But you know, if we talk about ten years from now, if you rewrote this book in ten years, what do you think would change.
00:27:40
Speaker 2: Well, my challenge, as I said earlier, is to overcome convention. So what would be great to be able to write is that overcoming convention was not as hard and didn't take as long as I originally thought that would be a great thing to be able to write ten years. No, that's my hope. I also think we'll see, without question, a more greater adoption of decision based analytics, and I think we will see a lot more AI being used in the industry, and hopefully we'll see the opportunity for active management to become stronger, and we may find a different balance point between active and passive investing. Right now, I think there's such a challenge in identifying a skilled manager that many investors are choosing to allocate more capital passively simply because they're afraid of picking the wrong manager. But if we can make skill as transparent as it needs to be and as it can be, then I think that offers the opportunity for a different balance point between active and passive, and that's another benefit we get by going down this road of improving our skilled metrics.
00:29:07
Speaker 1: Okay, I've got just one more question before I let you go. Every investor listening right now, remember just one idea from skill versus luck. What would you hope it would be that.
00:29:18
Speaker 2: There are skilled managers out there. There are plenty of them, the difficulty is finding them. So whether you're an institutional investor, a manager search consultant, or an individual, what it's incumbent upon you to do is to begin to demand better skill metrics from whomever you work with, whether it's an internal team, if you're a pension fund, or whether it's a manager search firm, if you're a large investor, or if you're an individual, whatever platform you're using, start demanding better skill metrics. They exist, they're relatively inexpensive, and they add tremendously to your ability to make better allocation decisions. And what we're all trying to do is meet some financial need in the future, whether it's retirement or an endowment, or making investments in the community or in healthcare or whatever it is. The better we do with our investments, the better off we all are. And asking and demanding for better skill analytics can allow you to make better investment decisions. And that's certainly what I would be pushing for right now, is we need to change the industry. And it starts by saying, hey, why don't we have some different skill metrics on this platform.
00:30:44
Speaker 1: That we're using. Well, this is great. I really learned a lot and enjoyed speaking with you. And so, Mike, thank you again for joining me today. Well, thank you very much.
00:30:53
Speaker 2: It's my pleasure.
00:30:54
Speaker 1: I also want to thank our listeners. If you like the episode, please share it, subscribe and leave a review. If you'd like to see more of our research on the terminal, go to BI Fund, go for US Fund and Active Research until our next episode. This is David Cone with Inside Active
