Fuld transskription af Dave Mabe - AI Overconfidence in Systematic Trading

he finally convinced me to start to use Claude Code. I was like, oh my gosh, this is so much better for a variety of reasons. I mean, the main reason was, of course, that, you know, with ChatGPT, you have to give it s- you have to basically copy and paste all the context to get useful information from it, right? The more context you give it, the better. Jonathan Stark: Mm-hmm. Dave Mabe: Well, with Claude Code, it runs right in your GitHub repo, so it ha- it like, it literally has all the context without you doing anything. So I was like, oh man, this is mind blowing from day one. Jonathan Stark: Isn't it wild what a difference it is to not have to cut and paste? Like, think of how lazy we are. Like, Dave Mabe: Yeah. Jonathan Stark: like, oh, I gotta copy this over here and put it on my desktop or put it in a Google Doc and then put it over here. And- and it- it is so frustrating that it is a complete game- game changer as soon as it can start to access like a local file system. It's Dave Mabe: Yeah. Jonathan Stark: just mind blowing what an improvement that is. Anyway, but please continue. Dave Mabe: Yeah, so I absolutely agree with that. And it was, I mean, it doesn't sound like a lot. Hey, okay, copy paste. You can just do that. It's no problem, right? Jonathan Stark: Yeah, yeah. Dave Mabe: But yeah, when I remember like when you, when you first start Claude in a GitHub repo, you type in that basically has it, you know, it goes and scans the, the repo to see what's there and comes up with this Claude.md file from its, you know, from its exploration of the code base. I was like, oh man, this is so good. It's just so much better. So, yes, I mean, I was a, I was hooked right away. Jonathan Stark: Mm-hmm. Dave Mabe: And it just gotten deeper and deeper into using it, and yeah, it's been, I mean, it's been game-changing for sure. Jonathan Stark: Mm, okay. So, in- in- well, in what ways? I mean, what would be a, um, a simple example of something that was just impossible or prevented of, you know, just like too much of a pain to do in the past? Unit Tests and AI-Assisted Development --- Dave Mabe: Yeah, so, so a quick story. When I was the CTO of Trade Ideas, I remember I hire and fire people. One of the guys I hired was a developer who was so passionate about unit testing, Jonathan Stark: Mhm. Dave Mabe: which, you know, if you know much about unit tests, they're not exactly sexy, they're not exactly fun to deal with. I was like, man, that's interesting. I hadn't really done much with unit testing. I sort of knew what it was, but man, Claude will write unit tests out the wazoo, which lets me get so much more confident in what I create. It's, it's actually nuts. I mean, I've got probably, I've got over 3,000 unit tests that it's written, uh, so it's every time I do a release, e- each time that I make a new feature and, and this, you know, accumulates over time, I get more and more confident just sending it out once those unit tests pass. It's Jonathan Stark: Right. Dave Mabe: kind of nuts. Jonathan Stark: Yeah, I don't know if, I don't know how clear it was at the beginning, but, but MabeKit is a software application, or a, a suite of software applications. Dave Mabe: Yes. Jonathan Stark: That people, that, that doesn't live in the cloud, so you really, you know, it's like downloadable software, so you really don't want to have bugs in it. Dave Mabe: Right. Jonathan Stark: Yeah. Dave Mabe: Yeah. Jonathan Stark: Okay, cool. Dave Mabe: So that's been just- Jonathan Stark: So unit tests and the level of confidence that you get from that, and, uh, I'm, I'm curious, like, how often- I mean, that's a lot of tests. How often does it happen that it catches something? I'm just out of curiosity. Does it happen a lot or not that often? Dave Mabe: Uh, I would say it happens a lot, um, and, you know, that's what they're there for to- when you add a feature that you- you know, you don't realize you broke something. Jonathan Stark: Yeah. Dave Mabe: How many times has that happened? You- you do a release and you're like, "Oh wow, I didn't even realize it affected that. Jonathan Stark: Right. Dave Mabe: And it's so hard to c- come up with these all at once. It- it's really much better to come up with them as you make changes to the software over time. And, you know, 3,000 unit tests sounds like, wow, I wouldn't, like, all those can't be doing useful things, but like I look at Claude and what it does, and they're very, there is, they're, they're always for something very specific and something useful. So it's not just like, you know, trivial things that don't really mean anything. It's really good at what it comes up with. And part of that is when I, you know, the guy that really recommended that I use this, he has a very specific Claude on Claude.md file that initializes the way he uses it, and he shared that with me, and it basically implements this workflow where don't do anything until you do a failing unit test first. So that whole workflow is, I think, dramatically better than the way other people use Claude. It's just, I just see that everywhere where, like, here's a good example. Like, I haven't run out of tokens in a long time, and I'm on the cheapest plan, and yet I hear other people who are on the $200 plan and running out of tokens. Like, well, how is that possible? Because I'm using it all the time. I mean, there's- it doesn't make any s- like, how would I even run out of tokens? But I think it's- I think that most people are having it do too much and not really understanding the details, and so they have it go all sort of on this wild goose chase, and it eventually, you know, finds the right or the, you know, maybe a right way to do things that- that you asked for, but I think my workflow is just, you know, if I see, you know, I've got- I have enough experience to where I s- if I see it doing something wrong, I'll stop it, said, "This is wrong, don't do that, do it this way, I know it's

the right way, don't- don't bother with these other things you're trying to do. Jonathan Stark: Mm. So this is a really tactical dev specific question, but how do you, how specific are you, or how much thinking do you do? Like, I- I haven't used Claw- I can't remember, I've used Claude Code, uh, a little bit, and it's got different modes, it's like planning mode, and, um, do you- do you use, like, different modes like that? I know this is- this is making me sound like I don't know what I'm talking about, but, um- Dave Mabe: Yeah. So, Jonathan Stark: Yeah, go ahead. From Manual Approval to Auto Mode --- Dave Mabe: It's interesting because w- several months ago, I would not turn on auto mode, which is like you give it the command or you- you give it the context, tell it what to do, and then it comes back and says, "Okay, I'm getting ready to check your file system, is that okay?" And you click okay, you know, enter. So with each thing it was doing, it was- I would specifically give it permission to do it. And- and then there's this plan mode where sometimes I would say, "Okay, well, let's plan this first, and then we'll start the work just to make sure that the plan is good. And a few months ago, I started using auto mode exclusively, which basically says it doesn't do any- like, it's planning already in there, and it doesn't- it- it has a, uh, sort of a parser to figure out if it has permission or if it should have permission to do the things that it's trying to do, so you don't have to go and hit enter every time it's at- it wants to try to do something. Jonathan Stark: Yeah. Dave Mabe: And it's way better. Jonathan Stark: Yeah, I'm sure. Well, so Dave Mabe: It's way better. Jonathan Stark: Yeah, until- until it- yes, takes all your money or whatever. I mean, like- like, I don't actually think that, but that- that's what someone would say. And- and when I was using what use- what's it, uh, what's the non-auto mode called? Dave Mabe: Uh... Jonathan Stark: I don't even know, but Dave Mabe: Yeah, good question. Jonathan Stark: Approval mode, whatever it's called. Dave Mabe: Manual mode. Jonathan Stark: Manual, right. So when I f- I think that must be the default, because when I first started using it, that's my- was my experience too, and within 5 minutes you stop reading what it's asking you. Dave Mabe: Right, yeah. Auto Mode and the Trade Client --- Jonathan Stark: You just hit enter. So I was like, why not just go to, you know, auto mode, YOLO mode? And the reason I'm not that familiar with Claude Code is because I use my OpenClaw for everything. I just have it prototype everything, but I'm not shipping software, I'm just doing little apps for myself. So, I think Claude Code is much better fit for that sort of thing, unit tests, all of that stuff, working locally, all that, it's great. Okay, so I feel like there are two big halves to, two big components or almost modules to MabeKit that people would be able to relate to, and the first one is the

trade client, which is basically clicking the buttons for you. So, like, here's a set of rules that were created from a strategy, and and here's access to an API that will allow you to, uh, I don't know what the right terminology, basically like, uh, place trades, like make a trade or buy or sell, basically, right? Dave Mabe: Right, but I- I- I- the way I refer to it is, uh, it's trading instructions. So you- you- it's set up to receive events that you, uh, you know, events that happen in the market. So, you know, stock XYZ hits a new high, that may be an event that your strategy is looking for, and that generates an event, and then the trade client is set up where you can basically have infinite flexibility to take that trading event and then enter the trading instructions around it so that the trade client automatically enters the order in exactly the right way with all your exit orders, uh, you know, your stop order, how long it's the, the, the position is going to be live, like everything about what you would do manually, it's defined in that, in, in the set of trading instructions. Jonathan Stark: Okay, so in developer English, what's a position and what's a stop order? Dave Mabe: Uh, a position is when you have, you know, you enter an order and you enter a position, so you buy 100 shares of Google. Jonathan Stark: Mhm. Dave Mabe: You have a position. Jonathan Stark: Okay. Dave Mabe: And, and then a stop order is, so let's say you, in your strategy, maybe you buy new highs for, uh, in a certain situation. Jonathan Stark: Yeah, so Google Dave Mabe: Well- Jonathan Stark: goes to an, an all-time high. If that happens, well, that would be a weird time to buy it, but, but something like that. Dave Mabe: Actually, it's not a weird time to buy. It's a very pretty popular trading strategy. Jonathan Stark: Oh. Dave Mabe: Buy new highs. Jonathan Stark: Oh, well, good thing I'm not a day trader. So, okay, so buy new highs, all right, is maybe it's directional and like it's the beginning of a, a run or something. Okay, I'm not going to try and know what I'm talking about here. So you tell me that, so now I understand what it is to have a position, an event happens in the market. You've got access to real time data coming from wherever, some API says this just happened, and you're like, ooh, that's one of the things I've been listening for. What am I supposed to do when that happens? Places the order. Now you have the position, and then the stop order is what? Dave Mabe: So, of course, not all the positions you take are going to be profitable, right? Some of it, some of them are going to go against you, and you're going to, you have to have a stop order to get out before you lose too much. So that will have been, you will have already done the research to figure out what the optimal stop is across all these, across your entire strategy. And then, you know, as the trade goes in and you take the position, then it automatically puts the stop at your predefined level so that if it gets hit, you know you want to get out. So it's basically protection for, you know, a large loss. Jonathan Stark: Mm. And is that something that you, that, are you listening for that event as well in the trade client, or is that something that you, when you take the position, you can put a, you send a stop order with it? So that's getting handled on the Dave Mabe: Right. So Jonathan Stark: On their side. Dave Mabe: Yeah, so the bro- brokers typically have these order types, a variety of order types that you can use to do this. So that usually lives on on your broker, your broker server, and, uh, you know, it's handled there. So the trade client's just sort of watching that. It's not, and it it's sending it and then watching what happens to it, but it's not, you know, uh, you know, watching for market data and actually implementing the stop locally. Jonathan Stark: Yeah, that would be scary. Dave Mabe: Yeah, it's, um, a lot of people do it. It's, I like my default to be on the server away from, you know, like just today my power went out before the market opened, and, uh, yeah, that so that's, that's scary. Jonathan Stark: Okay, that's a great example. That's a real world example. Dave Mabe: Yeah. Jonathan Stark: Okay, so in this, the whole trade client, so the software you built that, uh, is listening for events or, or however they're listening for events, it gets sent into the trade client. It's like, okay, great, this thing happened, buy Google, and if it goes below this, then sell it. Uh, okay, great. Now, is AI involved in any of that? I think the answer is no. Dave Mabe: No. Jonathan Stark: Right. So you're just, so for the trade client, you're just using AI to build it. Dave Mabe: Correct. The Strategy Cruncher and Backtesting --- Jonathan Stark: Cool. All right. Now let's switch over to the other side of the, the ballroom over here, the other side of the fence, and talk about the Cruncher. Dave Mabe: Yeah. So the Strategy Cruncher is basically my tool set that I've created over the last 20 years to trade well for myself, and it, it's packaged up as an offering for other traders to use. Jonathan Stark: What's, yeah, and what is, I mean, I, you mentioned back testing already. Let's, let's, let's go a little bit deeper into that and like maybe not necessarily a technical standpoint, but your worldview around it. Dave Mabe: Right. So I feel like there's really only two ways to get confident enough in a strategy to trade, to trade it with significant, significant size. One is just experience over and over and over, day in and day out, trading the strategy, seeing how it works, sort of getting a feel for the, you know, the edge cases, just getting a feel for h- how it operates. That takes time, right? That's d- that that takes, can take a long time for you to gain the experience enough to gain confidence to trade it with significant size. Jonathan Stark: What, so before before we move on from there, can you say,

can you explain a strategy in layman's terms? Dave Mabe: Right, so just what you mentioned earlier, buying new highs, that's a strategy. Now, but that's, that's the sort of a starting point for a strategy. Um, so you've got this theory that, okay, yeah, um, maybe you've seen somewhere online that people trade this and they think it works, or they, you think that they're doing well. You can, uh, so you have a strategy that says, okay, I'm gonna buy each time a stock, um, hits a new high. Of course, you can't do that for every one of them, you're only doing that in certain situations. So figuring all that out and then figuring out what to do once the new high happens and how to actually trade it, uh, all that is encompassed in a strategy. So it's like a, uh, i- it's like a sort of like a f- a tiny little program or logic for, you know, if then, you know, if this, then do this Jonathan Stark: Mm. Dave Mabe: for a trade. Jonathan Stark: So it's kind of like- it's kind of like a giant pile of rules. Dave Mabe: Exactly, yeah, that's a good way to put it. Jonathan Stark: Okay. All right, so- and it's so many rules that it- it almost doesn't make sense to call it that. It's so strategy makes more sense. Like, here's my strategy, this is my- I've got this idea for a strategy, maybe you saw one and you want to modify it. Sounds like they kind of proliferate around the internet and Reddit and stuff. So, okay, now you- and you did say earlier, so I've got this idea, I think if I buy new highs, the maximum dollar amount of this, and if I, you know, and- and sell when it hits a threshold or not, whatever the rules are, a whole bunch of rules, and maybe it's, uh, whatever, whatever your rules are. Then you take that rule set, the strategy, and you run it against historical data, and that's a backtest. Dave Mabe: Right. You're basically simulating that strategy over an historical period. So you can see, okay, if I would have, you know, gone back 10 years, started trading this strategy exactly as I've defined it, from that point forward, how would it have done? Jonathan Stark: Mhm. Dave Mabe: So Jonathan Stark: So you're telling me if I bought Amazon in 2000 or whenever they started, 2006? Dave Mabe: Exactly, yeah. Trading Versus Investing --- Jonathan Stark: So, OK, so that would be a dumb thing to backtest because it's too, what is that? Why is that? How would you characterize a, um, because this the whole idea, and I think this is going to get to a technical question that I think AI is going to be the answer, even though it sounds like a non-technical question. It's kind of easy to just look at historical data and be like, well, I should have bought Google, and I should have bought Amazon, and I should have bought Apple when they were really low. So that's, so I guess what I'm saying is the historical data at a certain level of abstraction is useless. Dave Mabe: Right, so. Jonathan Stark: It needs to be more like patterns, not like particular companies. Dave Mabe: Right. So what you're describing when you, you know, go, go back in time and buy Amazon 2000, you're making an investment. You're investing in that company, and I view that as very different than trading, like short-term trading. Jonathan Stark: Oh. I see. This has never come up before. Okay, keep going. Dave Mabe: So, uh, you know, i- investors look to buy a company that maybe never sell it or sell it, you know, hold it for years. Jonathan Stark: Mhm. Dave Mabe: Where with day trading or short-term trading, you're holding for small amounts of time. Jonathan Stark: Mhm. Dave Mabe: So all my strategies are day trading strategies. So at the beginning and end of every day, I'm in 100% cash. So- Jonathan Stark: So, right, so just to- for the dear listener, like, you have no money in the market overnight. Dave Mabe: Right. Jonathan Stark: So in the morning, your trade client takes over, buys a bunch of stuff, depending on the events that happen, and then when the market close, right before the market closes, or wh- or throughout the day at some point, but by the time the market closes, all your money is back out. Dave Mabe: Right. Jonathan Stark: And that makes total sense, right? You're not investing in the companies, you're sort of trying to predict the market, trying to Dave Mabe: Right. Jonathan Stark: predict the moves. Dave Mabe: And the reason I'm drawn to that is I had a couple trades very early on, like way back in the, you know, early 2000s, where I would, uh, I was basically i- i- investing or like holding a little bit longer, and I bought a company, I remember it was RFMD. It was a company based in North Carolina, where I'm from. Everybody's like, yeah, this this company's, you know, typical stock tip, hey, this company's going to the moon. I held it, it released earnings, it beat earnings. I was like, man, this is great, it's getting ready to go up, and then it tanked. I was like, okay, well, how can I possibly predict that? There's no way I can have any sort of edge doing that. And the other, so I realized, okay, I need to, I'm not never going to be able to predict that. I need to find some real edge. And Jonathan Stark: Edge being I'm going to beat an index fund or something. Dave Mabe: Right. And the other thing I realized is, man, all the news comes out after hours when you can't trade. Jonathan Stark: Yeah, yeah, yeah. Dave Mabe: So there's a whole lot of risk there. Jonathan Stark: Well, what's after hours trading? Dave Mabe: There, well, at the time there wasn't that much of it. There's a lot more of it now, but still, news comes out after hours where things are- there's less trading that can happen. Sometimes you just can't trade, so you're stuck. Jonathan Stark: Yeah, yeah, yeah. Dave Mabe: So- Jonathan Stark: That's a fun night. Dave Mabe: Yeah, so I like not having to worry about that at all. Jonathan Stark: Yeah, yeah, yeah. It's funny because I, you know, before we started working together, I was not familiar with

anything other than what you just described, which now I know is called discretionary trading or investing. Dave Mabe: Mhm. Jonathan Stark: And I was like, I don't like the sound of that at all, for the exact reason that you just described. Dave Mabe: Yeah. Jonathan Stark: Um, so not to sell people on systematic day trading, but, uh, but okay. So it's cool, it's cool because for a software developer, from an emotional standpoint and from a really a capability standpoint, like you're natural, you know, is former CTO, you've got all of these, um, skills that kind of push you in this direction, where it's like, I don't want to trust my gut instinct about what Jensen Wang is going to do. Dave Mabe: Right. Turning Market Patterns into Strategies --- Jonathan Stark: Whatever, what Jensen's going to do. So, so it's amazing that you kind of get out of that with through this process. But OK, so the the real question is, though, how would you, how do you use AI, because I know that's where we're going. How do you use AI to come up with s- strategies that are, I don't know if statistically significant is the right word, but they're- it's kind of- but you're just playing a- a- a probability game here, where, yes, you- you're not going to go back and invest in Amazon or whatever, you're going to buy s- stuff and then sell it before the end of the day. So where do you come up with- does Collide help you come up with the initial strategy, or do you have some intui- like, intuition must come into it somewhere? Dave Mabe: Right. So, yeah, the more- I mean, usually people have a little bit of experience before they start going down a systematic trading route. Jonathan Stark: Mhm. Dave Mabe: Uh, so they- so they have some theories about how the market works, or they've seen some strategies that they feel like work, and- or maybe they see something on the chart, a stock chart, which is a, you know, visual representation of the price over time, eh, of one stock, and they come up with a theory about, OK, well, this is- looks pretty interesting. What happened? You know, obviously, I could look at this one chart and see, wow, OK, I would buy right there before it, you know, has a huge meteoric rise. But the question is not how could I have gotten in this particular symbol, but okay, is this a pattern that I could create a strategy from? And so I've got several skills that Claude is able to use that, and one of the ones I just created recently, which is really, uh, helpful, is it can take a chart and come up with basically 3 different methods for getting into the, the, the pattern that i- i- appears on the stock, on that chart. So it's basically 3 different starting points for a potential strategy. Yeah. Jonathan Stark: Yeah, what signal to listen for, for the buy. Dave Mabe: Right, and that's a, it's a really, it seems like it should be really easy. Hey, look at this chart and come up with code that trades stuff like this, but it's actually quite hard to, uh, to, to do

that. Jonathan Stark: Doesn't sound easy to me. It sounds like, it sounds very hard, but I do have the intuition or and the experience that at least the frontier models are unbelievable at spotting patterns that the naked eye can't see. AI's Dangerous Overconfidence --- Dave Mabe: Right. So, but what they aren't good at Jonathan Stark: Mhm. Dave Mabe: is like, so they're not good at trading itself, like Claude's never made a trade, right? And so I think a lot of people get caught up, I see a lot of traders in this situation where because Claude is so good at programming, Jonathan Stark: Yeah. Dave Mabe: they think it's like some oracle that's good at everything. Jonathan Stark: Yeah, it's a god. Dave Mabe: Right, so they, uh, they end up having Claude do all sorts of things, like, you know, the classic example is, OK, Claude, you know, give me a strategy that works. Jonathan Stark: Well, that would be the right- that's where I would start. Dave Mabe: Right, of course. Jonathan Stark: Make me a millionaire. Dave Mabe: Right. Well, Claude- and the problem is Claude is going to answer that in a way that sounds pretty appealing, it sounds believable. Jonathan Stark: Right. Yeah, Claude, start me a business that I can sell in 3 years for 5 million dollars. All right. Dave Mabe: Yeah. Jonathan Stark: Where do you live? OK. Start a landscape business, blah blah blah. Dave Mabe: Right. Jonathan Stark: Sounds plausible. Dave Mabe: I mean, it turns into a cheerleader really fast. Jonathan Stark: Yeah. Dave Mabe: And it's like, you know, start my 3-year business, and it's like, oh yeah, let's, let's do it, Jonathan, let's go. Right? It's, it's so overconfident, and it, the problem is if you're not discerning about these problems that you're asking it to, it to do, Jonathan Stark: Yeah. Dave Mabe: it sounds plausible, it sounds like it's the expert, right? Jonathan Stark: Yeah, you're convinced. Dave Mabe: Absolutely. Jonathan Stark: You convinced me, Claude. Okay, so, oof. So it's funny 'cause I've had, uh, I have this same shape conversation this around this specific thing, the sort of overconfidence and the cheerleader thing. Oh, I've been having it a lot lately in lots of different fields, you know, cyber everything from, you know, cybersecurity to marketing and and, you know, in trading and whatever. It's just like everywhere. It's like it is really good at X and you begin to trust it, but you forget that it only built trust with you around X, and then here comes Y, and it stinks at Y. Dave Mabe: Yeah. Jonathan Stark: Or the other version of this is you're amazing at X, and you're not amazing at Y, and since you, you're like, it's, I guess it's kind of the flip side of the same coin. It's like with, with X, you forget how much you're guy- actually guiding it because it gives you this output that's, um, 50% of the time it's stupid, 50% of the time it's amazing. Dave Mabe: Mm-hmm. Jonathan Stark: But you can tell the difference because you're good at X. Dave Mabe: Yeah. Jonathan Stark: And so you forget about all the stupid stuff that you told it not to do, and in fact, it probably goes down fewer stupid paths because you guided it onto the right road in the first place. Dave Mabe: Right. Jonathan Stark: And then you're like, Claude's amazing, you- but you're discounting your own expertise, and then you ask it to do s- plan a vacation to Greece, and it- and it's- and it- you don't know, you've never been there, you don't know anything about it, and you're forced to trust it. So, OK. Dave Mabe: Have you- have you heard the term, I think it's Gilman's amnesia? Jonathan Stark: Yes. Dave Mabe: It's just- I- I can't stop thinking about this one, uh, about Claude now. Jonathan Stark: Yeah, I wrote an email about it recently, where when I was a little kid, I went to a rock show, like a- a paleontology kind of rock show, and because I was a nerd even then, and a reporter was there with a photographer, and they interviewed me, took my picture, put it in the paper the next day, and it had like 6 major errors. Dave Mabe: Yeah. Jonathan Stark: And I was like, and for I didn't get the, the, I think, I think it's Gelman amnesia. I don't remember the exact name. I didn't get Gelman amnesia. I never trusted the newspaper again. But most people are like, oh, this one article in the newspaper I'm reading that I know a lot about is totally wrong. It's completely, or at least it's 101 level simplistic stupidness. But all these other articles are expert level genius articles. Dave Mabe: Yeah. Jonathan Stark: That don't have typos or don't have the names wrong or, you know, don't have a correction in tomorrow's edition, which I'll never see anyway. And I mean, when you're on a daily deadline, and I'm from a newspaper family, uh, a lot of stuff's wrong, but anyway, that's the, that's the, I think that was the original use case, but it's the exact same thing, or the original, uh, impetus for that study was newspapers, but Dave Mabe: Mhm. Jonathan Stark: Now it's, it is totally true with Claude and or or any of them, any of even the most advanced ones, it is so bad at stuff that I'm good at. Dave Mabe: Yeah. Jonathan Stark: So why is it, why do I think it's so good at stuff I'm bad at? Because I can't tell the difference. Dave Mabe: Yeah. The Backtesting Software Trap --- Jonathan Stark: Anyway, so I'm just sort of, I'm sort of s- singing your song here. So what's the, what are the common failure modes for folks you're, not just in general, but folks you actually work with, like folks you talk to? Dave Mabe: Mhm. Jonathan Stark: What are the failure modes for people who are, you know, pretty serious? Dave Mabe: Right. Jonathan Stark: And maybe have been doing it for a long time, but Claude is still rel- obviously it's relatively new to everyone, but they're maybe even newer to them. Dave Mabe: Yeah, so by far the most, the biggest mistake I see, or the most frequent mistake I see now is people using Claude to create

their own backtesting software. So there's lots of backtesting software out there that works, and a much better approach is have- teach Claude how to use the existing backtesting software, but, you know, the first thing- Jonathan Stark: Why- why is that? Because intuitively, they probably have the- or- or it certainly intuitively in my part, I might have fallen into the same trap because I know it's good at making software. So why- why do you think it doesn't work? Dave Mabe: I think it- you- like back testing software is more complicated than Claude thinks it is, and there's all these edge cases, there's stock splits, there's reverse splits, there's corporate actions. Jonathan Stark: Oh wow. Dave Mabe: There's all sorts of things that can skew your back test, like can skew your data Jonathan Stark: Mm. Dave Mabe: if you, if they're not accounted for and, and, um, just not, not anticipating that and the first thing that Claude's gonna do is, yeah, we can, we can write a backtester, Jonathan. Yeah, no problem. Jonathan Stark: No problem. Dave Mabe: And then, you know, 2 months later, they realize they're, they've got this big pile of technical debt that they don't even realize, and they're just kind of stuck. Jonathan Stark: And, I mean, unlike perhaps a non-technical founder who's building, you know, what they think is the next greatest to-do app. Someone who's do, uh, using it to build their own backtester is losing money. Dave Mabe: Right. Jonathan Stark: Most, almost certainly. Dave Mabe: Yeah. Jonathan Stark: So they notice. No, it's not just technical debt, it's debt. Dave Mabe: Yeah, that's true. I mean, it's, it's so the, a good example of this is, um, you know, imagine somebody, just imagine a new Claude user you're excited about, you know, creating software. Somebody sends you an Excel spreadsheet, and you're like, wow, I don't want to install Excel. I'm just going to have Claude build Excel for me so I can read this file. That's essentially what they're doing. Like, it's the same thing. Jonathan Stark: They just totally Dave Mabe: You would never do that. Jonathan Stark: Yeah, Dave Mabe: you would never do that. Jonathan Stark: They're just totally underestimating how complicated a back- backtesting Dave Mabe: Right. Jonathan Stark: software is. And- and in fact, you just- just what you said before about, like, stock splits, I was just like, oh my, you know, and instantly I was- I was like, you know, the, the, uh, 2x2 matrix of like known unknowns and unknown unknowns. You said like 3 unknown unknowns right off, like right off the top of your head, and I was just like, oh, I would have got owned by that. That would have taken me for 3 to 6 months of complete wasted time and money. Dave Mabe: Yeah. I see a lot of traders making that mistake, and, but I, I s- I do see some traders recognizing that and not making that mistake, so there's growing numbers of both of those people, and, but I think that there's- there's more that are, you know, creating their own backtester with Claude, unfortunately. Jonathan Stark: Mm. Well, it sounds like the kind of- kind of error that corrects itself. Dave Mabe: Yeah, that- that's true. Operational AI Inside MabeKit --- Jonathan Stark: If they don't- if they don't quit. Okay, so where else, where in any of your, the current sort of suite of apps that you've got deployed to people, what ones, if any, do you have AI as part of the process where s- where you or s- or your users, maybe they've got their own API key or something, do you have anything where you're using AI operationally during the, uh, app usage? Dave Mabe: Yeah, so there's, there is one piece that it's quite good at, but it takes, well, actually there's a couple piece, couple places that I use it. One is, um, the backtest that I've encouraged people to use and, uh, uh, people that, you know, come to me typically use is called Amibroker. It has its own language that you use to create backtests with, and it's a little, because it's its own language, it's, um, uh, it's unfamiliar with, for people. And so I have a, a, uh, what I call an AFL generator. AFL is the, the, the Amibroker function language. So basically, I've got a concise form that you could do, you can enter data in for describing your, the strategy you want to create, and it'll create the backtest for you in the AFL language that will run in Amibroker. So it's a big time saver. I mean, if you ask Claude, okay, create a strat, you know, create the strategy in AFL, it'll do it, but there's like 10 ways to do that, and there, you know, 9 of them are bad. So giving it a good, all the context for what I know about creating strategies and doing it the right way is important and is very helpful. So that that's one, uh, that's one area that I use it. The other one I've just added recently, and it was way, it was a lot more work than I thought it was going to be. So it's an MCP for the Cruncher. So the basically the Cruncher, you submit your backtest, it's going to tell you, it's basically going to do an optimization for you to tell you ways to improve the strategy. So it's going to give you a whole litany of rules you could add to improve your strategy. And that, if you, the first time you use it, it's, it's not, new users may have questions about, okay, well, which rule do I decide? how do I, how do I, which rule should I apply to the strategy? How should I apply it? How should I run it? Jonathan Stark: Yeah, it's like data, but not information or not knowledge. Dave Mabe: Right. So I tried to capture all of my intuition and experience in making that judgment Jonathan Stark: Mhm. Dave Mabe: in an MCP. So I thought it was going to be pretty easy, but it took way longer than I thought because Claude is consistently trying to be more helpful than it should be. Jonathan Stark: Okay. Dave Mabe: So basically, I was just like coming up with all sorts of guardrails telling it what not to do. Jonathan Stark: Mhm. Dave Mabe: And I- so

it's- it distills down to what I'm allowing it to do in the right context, and it's- it's quite helpful, but man, it took a long time to get Claude out of its own way. Encoding Expert Judgment --- Jonathan Stark: It kind of sounds like you're trying to document your gut instinct. Dave Mabe: Yeah, that- it was- th- that's what I was doing, and it's, um, it- it took a while to get there, but it- it works pretty well, but it- it- there's a lot of, uh, creating guardrails for it. Jonathan Stark: Yeah, that's fa- I- so I ha- I actually have a- I don't know if we- I don't think we've talked about this. I have a parallel situation where I was like, and, you know, perhaps this perhaps of interest for anybody who's trying to develop an app that's of this category, where you're trying to document your good instincts so that an AI can do it on your behalf, but I think there's a lot of people out there thinking that. I had the same experience. It's really hard. Um, the one I was trying- I was working on was, um, you know, record a sale, you know, software developer, somebody has a consultant, has a sales call with a potential client. Upload the transcript or create a transcript. You can anonymize it if you want. Upload it, and it will apply my thinking to the conversation that you had to score it and say, you know, how did you do? And if it wasn't A plus, like, what could you do better next time? And it is brutally hard. Dave Mabe: Yeah. Jonathan Stark: It's like, it's just possible enough, it's just good enough to make you think like, this could work, I just Dave Mabe: Right. Jonathan Stark: haven't given it enough of my gut instinct yet. And, you know, you've been doing what you've been doing over 20 years, same here, and it's a longer list than you think. Dave Mabe: Yeah. Yeah, a lot of the stuff you just internalize and don't even think about. It's just completely natural for you to make decisions like you do, and you have Jonathan Stark: Yeah. Dave Mabe: been for so long that Jonathan Stark: Yeah. Dave Mabe: it's- it's hard to quantify that and Jonathan Stark: Yeah, it's certainly hard to articulate it. Dave Mabe: Yeah. Jonathan Stark: Right. Yeah, because you could say, you know, you could say, present me with a bunch of scenarios, and you're like, oh, I would pick, you know, n- n options or potential choices, and then like, you know, A, B, C, none of the above, and it- I'll be like, B, and then it's like, okay, why? And I'm like, I don't know. Dave Mabe: Yeah. Jonathan Stark: Past experience, it's hard to say. I just know it's crazy. Dave Mabe: Yeah, it's just a lot of trial and error. Um, I mean, it's just hard to generalize that type of knowledge, I think. Jonathan Stark: Mm. Yeah. But I bet you, I mean, I'm sure my brain is doing the same thing at some point. I mean, maybe this is a bigger conversation about neuro- you know, neuroscience or something, but I do feel like it's like a plinko board in there where it's hitting the, you know, the- the situation is hitting a bunch of s- familiar patterns that are maybe not always combined in the same way, but here- here's 5 patterns that I've seen before, never all combined, but it leads me to believe that the- the most beneficial move would be price it high, don't take on the client, give it to someone else, price it low. Uh, and it- there's at least a half a dozen, but it's probably more like 30 factors that I'm considering in a situation like that, and it would be so cool to be able to move that over into so that I just- it just outsource it. It would be so Dave Mabe: Yeah. Jonathan Stark: cool to be able to outsource that. Dave Mabe: Well, he- here's what I- I was in at that point, and I think a lot- probably a lot of listeners, I would imagine, are sort of perfectionists, and I was at that point with this because I realized it wasn't perfect. I realized I could do a lot more testing to try to make it a little bit more perfect, but I was- I was kind of getting tired of it. I was like, well, let me just go forward with this as is and just see what happens. And the people that use it love it. Like, it was way better than I thought it was going to be. I thought, you know, I thought I'd be so embarrassed by some of the things it was doing, but it was way better than the default or the median interpretation of- of the users that were using it. They love it. So it was- Jonathan Stark: Mm. Dave Mabe: That was, um, that was kind of eye-opening for me. Guidance Instead of Perfect Answers --- Jonathan Stark: Well, let me ask the tough question, because I'm in the same situation, but you're farther down the path. It's giving them answers to a question, and they love that. They love that they're getting confident answers to a question, but the proof's in the pudding. So how do you- so it's one thing to, like, try to make it as good as you, 100% as good as you, 80% good as you is- is better than nothing. But for me, and I don't- I don't think I'm being perfectionist, and you must have really wrestled with this because it is direct- well, I guess mine's directly applicable to money too, but when I see it do one thing that is completely, it's almost like the magnitude of the, of the badness. So if it's like pretty good all the time and it's never horribly wrong, I'd probably be pretty comfortable with, you know, reviewing sales calls and scoring them and, and maybe even generating a proposal based on the sales call. And if they're, if they were pretty good every time, I'd be like, yeah, that's pretty good, and I could say to people, it's pretty good. Dave Mabe: Mm-hmm. Jonathan Stark: You know, I, you know, if we were working together privately, it'd probably be better, but it's pretty good, and, you know, it's part of your subscription, and you could just, you know, it's better than nothing. But for that 10% chance that it's horribly wrong. Dave Mabe: Mm-hmm. Jonathan Stark: I'm like, oh, so it's, it feels like, as I say that out loud, I feel like this

is just a layering upon layering upon layering of guardrails or something, but how did you, you know, again, I'm sure there are tons of people listening that are having these same ideas. How do I outsource my expertise to something so I don't have to do it every time, and I can sell it for a fraction of what I would normally have to charge? So how did you tune it? Like, what was the, what was the process through which you tuned it to the point where you were like, this is pretty good? Dave Mabe: Yeah, I actually changed the scope of it a bit in a way that I think was really helpful. So when I first started with this project, I was thinking- I was thinking that it would come up with the quote unquote answer, like the answer. Jonathan Stark: Mhm. Dave Mabe: But what I realized is actually it doesn't need to do that. I- it- and there is no such, like, the answer that it needs to come up with. It's much better to sort of guide, like, almost ask them the right questions in response Jonathan Stark: Mm. Dave Mabe: and give them guidance on how to, um, answer it for themselves or direction, like, some options for them to take. Jonathan Stark: Oh, that's really good. That's really good. It's more consultative. Dave Mabe: Right. And- and I realized that, you know, it's- there- there is no the answer, like, you're not gonna be able to come up with the answer that's- that is perfectly accurate, right? Jonathan Stark: Yeah, there'd be no way to validate it. You just Dave Mabe: Right. Jonathan Stark: have to play it out, right? Dave Mabe: So, but giving people some guidance on, uh, some options to take or, like, throwing some questions back at them that make them th- think, "Oh, wow, yeah, that's right. I didn't- hadn't thought about that. Jonathan Stark: Mm. Dave Mabe: It's- it's really valuable. Jonathan Stark: Yeah. And- and I'm sure this is true with you, there's usually a, you know, a dozen or two dozen anti-patterns that are just classic blunders. Dave Mabe: Mm-hmm. Jonathan Stark: That you can just like, no, that like, that's a bad idea. Like, that's not a good idea. The chances of that playing out are really bad. There's lots of evidence that that won't work. But it's- it's sort of like a hypothesis. Like, someone could come to me with, uh, like a positioning statement or something. I'm like, that's not, that's not a positioning statement. Like, I can see that that is not a good, it's not positioning, it's nothing. But when I see one that does look like a positioning statement, that doesn't mean it's gonna work, but I can tell when it's not one. You know what I'm saying? Dave Mabe: Mm-hmm. Jonathan Stark: It's kind of like, um, it's kind of, this is, this is reaching way back, but there's the con- there's a concept of, um, well-formed XML, Dave Mabe: Mm. Jonathan Stark: valid XML, and then there's XML that conforms to the grammar that is assigned to it. Dave Mabe: Mm-hmm. Jonathan Stark: And- and it can tell, like, you can tell, it- it's feels like the

same kind of thing that's super nerdy, but, um, I can tell when something's in the wrong form. It's like someone's making a category error. Dave Mabe: Mm-hmm. Jonathan Stark: So- so you can rule out a whole bunch of stuff, which lowers the- the- the problem space by dramatically l- s- decreases the problem space. And then you get to the other things, exactly like you just said, which is you have it ask them questions, and they kind of decide for themselves, given these- given this sort of, um, guard- yeah, guardrails. I just have to say guardrails again. Given these guardrails, like, what is your answer to these? Okay, you know, given these 3 things, here's what I would do, but it's not guaranteed to work. Dave Mabe: Yeah, I mean, the- a lot of questions like this are just so open- open-ended. Jonathan Stark: Mhm. Dave Mabe: And there's- I- I think people get stuck just because there's- there's a thousand different ways to do it, and maybe they haven't even thought of, you know, a- a quarter of them. Jonathan Stark: Mhm. Dave Mabe: And- but just giving them some, like, have- I- I like the way you put it, like, reducing the problem space is really, I mean, that's very valuable to be able to do. Jonathan Stark: Oh my God, it's a game changer. Dave Mabe: Yeah. Jonathan Stark: It's not a silver bullet, but it really makes the decision making process easier. Dave Mabe: And it s- it sort of gives them more, ultimately it gives them more confidence in what they're doing. Because if you, if somebody came to you and you gave them the answer and that was it, like that, you s- they still have to come to it on their own and really understand and internalize it. AI Adoption in Trading --- Jonathan Stark: Yeah, but am I standing in front of a Ferrari or not? So, OK, inside joke. Anyway, the the let's zoom out because we're we're I know we both have to run. You're, if not the first, certainly one of the only day trader tech people I've had on the show, you know, so so you're in a domain that is highly unusual for this particular show, and yet Claude is, it sounds like, I mean, this is the question really, how flooded is your space with people using AI to try and make their lives better? Dave Mabe: Oh wow, it's completely flooded. Mm. It's, yeah, everybody's using it. Jonathan Stark: Do you see, if you had to speculate, do you think that they're, you know, these mistakes that they're making, these anti-patterns that they're, they're, these traps they're falling into, do you think you're gonna, we're gonna see, you know, a, a, a res- not a recession economically, but like a receding of people are gonna get bored of it, or it doesn't really work, and they decided to go back to gut instinct? Or do you think even if there's a small kind of drawdown in the attention in this group, is this, is this the inevitable future? Dave Mabe: I, so I, I bet there'll be a lot fewer tokens used over the next couple years among traders in my space, and I think that because I, I, I,

I, I see a lot of traders will make these dashboards that are sort of fun to look at, but they're not really contributing to the bottom line, and I- Jonathan Stark: Mhm. Dave Mabe: And so I see a lot of traders doing projects that don't really matter that much, but are fun to look at. Jonathan Stark: Mm. Dave Mabe: And they feel like they're really powerful, but I- I think there's only going to be a certain number of things that you can have Claude do that really can contribute to your profit and loss. Jonathan Stark: Mhm. Dave Mabe: So I think over time, people will hone in on those- those more, and it'll be, um, you know how it's, it's almost kind of a brag now that you use so many tokens, but maybe they'll, maybe that won't be the case in a few months, and it'll be more like you'd be bragging about how discerning you are about, you know, which projects to work on, or you're, you know, bragging on your, your project taste. Jonathan Stark: Yeah. Yeah, it's, I'm seeing a similar pattern in the, in the straight up traditional developer space where people are, you know, you go, I was just talking to someone, they said they, you know, went to a get together over the weekend with a bunch of software developers, and they're all showing like, oh, look what I, look what I built over the weekend, and they're all showing their, it's like show and tell on their phone, these apps they built for themselves. It's like, that's going to get old. Dave Mabe: Yeah. Jonathan Stark: It, it reminds me of the, it's a little bit like when I would go to a barbecue in 2009 and people found out I was a mobile consultant, and they're like, I've got an idea for an app. Like, nobody says that anymore. It's like, it's just this gold rush type of mentality almost, but with no gold. So, I mean, well, I guess there was no gold in the App Store rush either, really. I mean, there were some big winners, but that was about it. And I guess it's probably the same thing here, where it's just going to become silly when you go to enough of these barbecues. Dave Mabe: Yeah. Jonathan Stark: That it's like, oh wow, you've got an idea for an app? Wow. And now it's, now it's, wow, you built an app? Wow. Yeah, it's, it's going to be, it's, I mean, it's going to be wild, but it's interesting to hear that you, that's your intuition, is that people are going to start to recognize that the, the fancy looking dashboards aren't really, aren't really helping. Dave Mabe: Yeah, a good example is I just saw a news, uh, article recently about Goldman Sachs taking Claude away from their traders. Jonathan Stark: Mhm. Dave Mabe: I think just because I, I assume they're coming to this conclusion where, okay, you're, you're, you're using it a lot that we can see that in the token spend, Jonathan Stark: Mm. Dave Mabe: but, but is it translating into profits for us? That's going to be the question, and Jonathan Stark: Yeah. Dave Mabe: it sounds like that answer to for themselves because probably traders, it's it's got to be a very specific kind of trader that can really get a ton of value from from Claude. Jonathan Stark: Yeah, it's not a one shot. Dave Mabe: Right. Experts, AI, and the Road Ahead --- Jonathan Stark: Yeah. Well, I I welcome the pendulum swing back for sure, for a variety of reasons, and this conversation, it's probably just me, it's probably me starting the conversation with different people, but I've been having this conversation a lot about, you know, AI in the hands of an expert is really powerful. Dave Mabe: Mhm. Jonathan Stark: AI in the hands of a non-expert is really dangerous, if it's a- if it's for anything important. You know, if- if you're deciding where to go for dinner, fine, whatever, but if it's- if it's something important, like it's a big business, strategic business decision or something, it's like you really, it's, it's a little scary. So I think people are going to maybe learn the hard way, um, but at the same time, on the other side of that, just for the, for the dear listener, I do think there will be, first of all, I don't think it's going to go away. Dave Mabe: No. Jonathan Stark: And I think there is a future in which 100x of whatever we're doing now becomes the new norm. S- t- t- in the, the analogy I keep reaching for, which is maybe just eight dates me and people don't even, doesn't even resonate, but when the first TV shows came on the air, there was basically somebody put a TV camera in a ra- radio studio, in a radio station, and they just had people doing a radio show with a camera on them. And that's what I feel like we're doing right now, is we're optimizing all the things that we've already been doing and, you know, paving all the cow paths, so to speak, but we're not, there's not a lot of, I haven't seen a lot of things where people are blazing new territory, like, like I'm doing a thing that would have been impossible before, before. And, and just using enough imagination and thinking big enough to say like, well, what if, what if it's like the compute's basically free right now? I don't think that's going to last, but it's basically free right. Or for 200 bucks a month, you basically have unlimited compute for anything you conceiv- conceivably would want to do. What's the biggest thing I could do with that? Instead of thinking, like, how could I cut costs and automate things in my existing business? Like, I- I feel like they're not- it is really hard to think like that when- when a, uh, bottleneck or a constraint just vanishes. Dave Mabe: Yeah. Jonathan Stark: Tomorrow, it's- you can teleport. Safe teleportation is here tomorrow. Eh, well, I don't even know what to think, like, what do you Dave Mabe: Yeah. Jonathan Stark: use it? You know, it's like Superman is real. He can fly you anywhere you want. Like, what, what, okay, how do I use this new capability in a way that's not just a dumb version of the old, like an optimization of the dumb stuff I've been doing? Dave Mabe: Yeah, yeah, I think the, I- I- I think the people that have the skills to use it really well are gonna find out the stuff that is gonna make a difference. I mean, I think it's- it's just a matter of time, it has to be. Jonathan Stark: Yeah, I- I would agree with that, yeah. Well, cool, we gotta run, but, uh, this has been amazing. I feel like we could keep talking for another hour, but, um- Dave Mabe: Yeah, I'm sure we could. Thank you. Jonathan Stark: Yeah, well, thanks for coming on. It's just super fascinating. Tell folks where they can find out more about what you're doing, maybe find out about MabeKit or whatever else you got going on. Dave Mabe: Sure, you can go to my website, davemabe.com. Also have a free backtesting course if you're interested in backtesting at all at betterbacktesting.com. All those come right from my email. You can hit reply and just go, go right to my inbox. Jonathan Stark: Fantastic. All right, folks, that's it for this week. I'm Jonathan Stark, and I hope you join me again next time for Ditching Hourly. Bye.