WEBVTT
00:00:05.759 --> 00:00:07.759
Welcome to the Dashboard Effect Podcast.
00:00:07.839 --> 00:00:08.640
I'm Brick Thompson.
00:00:08.880 --> 00:00:09.759
And I'm Landon Oakes.
00:00:09.919 --> 00:00:10.560
Hey, Landon.
00:00:10.640 --> 00:00:14.800
So we thought today we'd talk a bit about vibe coding using AI.
00:00:15.759 --> 00:00:27.440
I've been doing some personally over the last few months, especially since about I don't know, Thanksgiving, Christmas time last year when I think it was Opus 4.5 came out.
00:00:27.679 --> 00:00:28.559
It was pretty darn good.
00:00:28.640 --> 00:00:31.039
All of a sudden Claude Code was really good.
00:00:31.199 --> 00:00:34.000
I don't do any coding really professionally at all.
00:00:34.159 --> 00:00:36.640
Haven't for you know a couple decades at least.
00:00:36.799 --> 00:00:38.240
Not probably a couple decades.
00:00:38.479 --> 00:00:40.960
But all of a sudden, I'm doing all kinds of coding.
00:00:41.039 --> 00:00:43.600
And it's really fun to be able to do that.
00:00:43.840 --> 00:00:49.119
And then when Opus 4.6 came out from Anthropic, I think it was February 6th.
00:00:49.200 --> 00:00:51.920
Uh same time GPT 5.4 came out.
00:00:52.159 --> 00:00:57.280
Anyway, I was I was still doing the vibe coding there using uh cloud code, still fantastic.
00:00:57.600 --> 00:01:00.880
Now, just yesterday we got GPT 5.5 out.
00:01:01.039 --> 00:01:04.400
Codex, you know, the the OpenAI app is updated.
00:01:04.640 --> 00:01:06.640
I'll probably be spending my weekend doing that.
00:01:06.799 --> 00:01:08.719
So anyway, I thought we were just talking about it.
00:01:08.799 --> 00:01:15.519
We're doing uh not really vibe coding here at Blue Margin, but we are using AI tools to help us build our stuff.
00:01:15.680 --> 00:01:17.680
I know you've been experimenting quite a lot.
00:01:17.920 --> 00:01:18.000
Yeah.
00:01:18.239 --> 00:01:19.599
Um maybe maybe we can start there.
00:01:19.680 --> 00:01:26.640
We just talk about what we're doing, what we're finding is helpful, where it's working well, where we still have to exercise a lot of care.
00:01:27.280 --> 00:01:28.400
Yeah, definitely.
00:01:28.719 --> 00:01:29.200
Let's see.
00:01:29.359 --> 00:01:45.040
So and we've we've done several things recently from all the way from uh you know automating, not necessarily automating, but vibecoding a uh API notebook that that pulls data out of out of a source, um, which has been interesting.
00:01:45.200 --> 00:01:56.400
We've you know created synthetic data sets and demos, demo data sets, because one of the struggles is you want a data set that replicates a business, but it you also want one that's updating and new.
00:01:56.640 --> 00:01:56.799
Yeah.
00:01:57.040 --> 00:02:02.719
Um so you can kind of create your own now without a ton of effort, which has been really cool.
00:02:03.040 --> 00:02:05.680
Um Yeah, several more that I'm probably not thinking of.
00:02:06.079 --> 00:02:06.400
It is cool.
00:02:06.560 --> 00:02:16.639
So you you manage our platform team, and so you're you and your guys manage all of our clients' data lake houses, but a big part of that is pulling in new data sources.
00:02:16.960 --> 00:02:36.879
So creating uh you know a really robust data pipeline that can run daily or multiple times a day and has a graceful fail mode if something goes wrong so it doesn't leave the data lake in bad shape and sets off alerts and you know makes it easy for the platform engineers, well, easier, to go in and fix it.
00:02:37.120 --> 00:02:41.680
But also right up front, when a new data source comes from one of our clients, we've got to make that connection.
00:02:41.840 --> 00:02:48.080
So we've got a series of Jupyter notebooks, depending on what type of data source it is, so we can make that connection.
00:02:48.319 --> 00:02:53.039
In some cases, um, I think I don't know, somewhere like a hundred data sources.
00:02:53.120 --> 00:02:57.439
We have something we've used before, so we at least something have something we can start with.
00:02:57.599 --> 00:03:02.400
Um sometimes we get brand new stuff though, like uh like a totally novel API.
00:03:02.639 --> 00:03:06.159
It's not uncommon for API documentation to not be great.
00:03:06.400 --> 00:03:06.719
Yeah.
00:03:06.960 --> 00:03:10.879
And so, yeah, that's a perfect place to use vibe coding.
00:03:10.960 --> 00:03:12.879
Um, but we want to start with our standards.
00:03:12.960 --> 00:03:16.560
We have best practices, we want to do all those things I was talking about.
00:03:16.719 --> 00:03:18.800
So how do you how do you start?
00:03:18.960 --> 00:03:23.039
How do you get uh your I don't I think you guys are using VS Code with Claude.
00:03:23.360 --> 00:03:24.719
Yeah, if I'm if I recall.
00:03:24.800 --> 00:03:32.240
So how do you get it to know what your target architecture is and make sure it does a good job and doesn't go off and do something kind of crazy?
00:03:32.639 --> 00:03:33.120
Yeah, yeah.
00:03:33.199 --> 00:03:34.479
So it's still a work in progress.
00:03:34.639 --> 00:03:38.879
We're still refining it, which I'm sure all things AI are going to be that way.
00:03:39.120 --> 00:03:41.199
But um right now we're using skills.
00:03:41.360 --> 00:03:49.199
So we've you know kind of trained skills to uh learn what our current best practice template is.
00:03:49.280 --> 00:03:51.680
So what some of our best Python engineers made.
00:03:51.840 --> 00:03:55.759
Um we gave it that and we said, hey, this is the stuff you need to follow.
00:03:55.840 --> 00:03:58.479
Here are the things you shouldn't do, here are the things you should do.
00:03:58.719 --> 00:04:04.080
Um it's actually quite interesting because we we ended up trying to put everything in one skill.
00:04:04.319 --> 00:04:07.840
And that so that includes like how do I handle all these edge cases?
00:04:07.919 --> 00:04:09.520
How do I go out and find documentation?
00:04:09.599 --> 00:04:11.199
What do I do if I can't find the documentation?
00:04:11.360 --> 00:04:11.520
Yeah.
00:04:14.639 --> 00:04:14.800
Okay.
00:04:15.120 --> 00:04:15.280
Yeah.
00:04:15.439 --> 00:04:18.399
And we found some weird behavior with that.
00:04:18.639 --> 00:04:20.800
It's almost like we just gave it too much.
00:04:20.959 --> 00:04:21.120
Yeah.
00:04:21.360 --> 00:04:29.680
So one of the things we're experimenting with now, which seems a little bit more promising, is having discrete skills that can call each other.
00:04:30.079 --> 00:04:30.720
Oh, that's okay.
00:04:30.879 --> 00:04:34.160
Do you have like an orchestrator skill that knows which skills to call?
00:04:34.720 --> 00:04:43.120
So, like, for example, I'm imagining you can tell me if this is what you're doing or not, but um you have a new API, you can have a skill that goes out and reads the API documentation.
00:04:43.439 --> 00:04:43.680
Exactly.
00:04:43.920 --> 00:04:54.399
For example, that's actually what what led us to the idea because we are having a lot of lot of trouble with the API documentation, because not all documentation is AI friendly.
00:04:54.480 --> 00:04:59.199
I just can't read it because they put it in JavaScript or there's some weird backend on it.
00:04:59.279 --> 00:05:00.720
So Yeah, interesting.
00:05:00.959 --> 00:05:06.879
And some of these APIs have sort of secret off-the-menu stuff that you have to know to really make it work well.
00:05:07.120 --> 00:05:07.199
Yeah.
00:05:07.360 --> 00:05:07.519
Yeah.
00:05:07.839 --> 00:05:08.079
Yeah.
00:05:08.560 --> 00:05:13.040
And then sometimes the APIs don't work well, so you end up going to the old ODBC or something.
00:05:13.279 --> 00:05:13.360
Yeah.
00:05:13.680 --> 00:05:14.560
Anyway, yeah.
00:05:14.879 --> 00:05:19.600
So um I know you've got goals around bringing down the amount of time it'll take to create one of those.
00:05:19.920 --> 00:05:41.360
One of the interesting things uh to me that uh you told me you found out was that it's not necessarily the hands-on keyboard time, but there's a lot of time around that, getting access to the data source, getting permissions set up right, getting the OAuth set up, you know, uh the system accounts, whatever you need, um uh as well as you know, looking at code and and doing reviews and all that.
00:05:41.600 --> 00:05:48.000
So the hands-on keyboard time can come down significantly, but you can still end up spending days.
00:05:48.160 --> 00:05:50.560
Like may maybe hands-on keyboard is half a day now.
00:05:50.720 --> 00:05:50.800
Yeah.
00:05:51.040 --> 00:05:56.240
But you can still spend days or even a couple weeks getting it done because of all that ancillary stuff.
00:05:56.480 --> 00:05:56.639
Yeah.
00:05:56.800 --> 00:06:03.040
And I sort of wonder if that's going to be the case with a lot of business agentics stuff for a while.
00:06:03.360 --> 00:06:03.519
Yeah.
00:06:03.759 --> 00:06:04.959
I think it probably will, actually.
00:06:05.360 --> 00:06:06.879
Yeah, I would I wouldn't be surprised.
00:06:06.959 --> 00:06:09.759
You know, it's whenever you need to coordinate people, really.
00:06:09.839 --> 00:06:12.639
That's where a lot of time a lot of times our roadblocks come from.
00:06:13.040 --> 00:06:13.920
So Yeah.
00:06:14.079 --> 00:06:15.199
Yeah, it's true.
00:06:15.519 --> 00:06:18.639
It's gonna it's gonna be interesting to see how all of that plays out.
00:06:18.879 --> 00:06:30.720
Like if you tried to build a skill that said, okay, contact the client, get them to set up the author, it's just not gonna work because you need to talk through that very uh variable thing that happens, I think.
00:06:31.120 --> 00:06:31.600
Yeah, yeah.
00:06:31.680 --> 00:06:38.079
And it's you know, we we've seen just hundreds and hundreds and hundreds of sources, and it's just always so completely different.
00:06:38.319 --> 00:06:38.560
Yeah.
00:06:38.800 --> 00:06:41.439
You know, sometimes there's nothing even online about it, right?
00:06:41.600 --> 00:06:41.680
Yeah.
00:06:42.079 --> 00:06:43.199
We've run into that before.
00:06:43.439 --> 00:06:46.480
People just don't talk about it online.
00:06:46.639 --> 00:06:51.279
So the AI doesn't know, doesn't isn't trained on it, it doesn't know where to go find anything about it.
00:06:51.519 --> 00:06:52.879
Kind of runs into a dead end.
00:06:53.120 --> 00:06:53.439
Yeah.
00:06:53.759 --> 00:06:57.839
Do you think does Claude Code do a pretty good job of following our best practices?
00:06:58.800 --> 00:07:00.160
It's I don't know.
00:07:00.240 --> 00:07:02.560
I'd say like 80-20.
00:07:02.959 --> 00:07:05.360
I might my team might say I'm being too generous to Claude.
00:07:05.600 --> 00:07:05.920
Really?
00:07:06.000 --> 00:07:06.319
Yeah.
00:07:06.800 --> 00:07:16.000
Um they uh they have some of their times where it's like, oh well, it just went over here and edited that thing that I didn't want it to edit, right?
00:07:16.240 --> 00:07:16.319
Right.
00:07:16.480 --> 00:07:19.759
And so that's part of one of the reasons why we actually switched to VS Code.
00:07:19.920 --> 00:07:22.319
Um from just using the terminal.
00:07:22.639 --> 00:07:23.519
Yeah, exactly.
00:07:23.839 --> 00:07:29.680
Because one nice thing is that as long as you don't tell it, yeah, just go crazy, edit whatever the heck you want.
00:07:29.839 --> 00:07:34.879
Um, you say, Let me approve edits, it will show you a side-by-side comparison.
00:07:35.040 --> 00:07:38.160
Here's your version, here's what I'm gonna change, and it highlights the changes for you.
00:07:38.240 --> 00:07:41.360
You know, like those of you who might have used um Git compare.
00:07:41.600 --> 00:07:45.040
It's that essentially you're comparing what the AI did to what you have.
00:07:45.199 --> 00:07:45.600
Yeah.
00:07:45.839 --> 00:07:54.240
And so makes it way easier to make sure it doesn't go off the rails and edit something completely different, along with sneaking in, you know, the editing you asked for.
00:07:54.560 --> 00:07:54.959
I hate that.
00:07:55.040 --> 00:08:01.279
Oh, and that was the problem, you know, six months, nine months ago, whenever I would play with vibe coding, because I have been for, I don't know, the last year or so.
00:08:02.000 --> 00:08:08.639
But not that consistently, maybe for the first half of last year, till we got up to the holiday season, because it just wasn't that good.
00:08:08.879 --> 00:08:09.040
Yeah.
00:08:09.199 --> 00:08:17.199
You know, it would produce something that was close, and then you'd risk r request fixes, and it would maybe make the fix but break something else.
00:08:17.360 --> 00:08:20.319
And it was just sort of it was like playing whack-a-mole.
00:08:20.399 --> 00:08:24.480
And then all of a sudden, around Thanksgiving, Christmas time, it stopped feeling like that.
00:08:24.560 --> 00:08:27.519
And then around February, it really stopped feeling like that.
00:08:27.920 --> 00:08:28.160
Yeah.
00:08:28.319 --> 00:08:28.560
Yeah.
00:08:28.800 --> 00:08:29.920
It's gotten so much better.
00:08:30.240 --> 00:08:30.720
Yeah.
00:08:31.040 --> 00:08:35.039
So uh well, you're gonna have to try GPT 5.5, the codex.
00:08:35.200 --> 00:08:35.279
Yeah.
00:08:35.919 --> 00:08:37.600
Uh and uh and let me know how that is.
00:08:37.840 --> 00:08:39.120
I'll be playing with that too.
00:08:39.360 --> 00:08:58.159
Another thing I know you were working on, so we've got I'm doing work for clients to connect Claude and other um agents to a data lakehouse, to their data lake house, and be able to do ad hoc queries for executives to be able to do, you know, get quick answers on financial data or operational data.
00:08:58.559 --> 00:09:03.360
And it seems like that would be easy, but um it's easy to get wrong answers, actually.
00:09:03.600 --> 00:09:24.480
And so we've been building something called a platinum layer, which is to take the gold layer out of the medallion architecture and then uh do some things to it, denormalize it even more, rename columns, rename tables, make sure all of our joins are done in a consistent manner, and then actually stack a bunch of markdown files into the data lake as well with context.
00:09:24.720 --> 00:09:36.240
But then in addition to that, um, we started building custom MCP servers for different clients that that have a lot more instructions about how to go and get an answer for the agent.
00:09:36.399 --> 00:09:40.879
Um, it's been really interesting, and you've been doing some work to build a demo for that.
00:09:41.039 --> 00:09:41.200
Yeah.
00:09:41.360 --> 00:09:44.399
And part of that was sort of vibe coding a data set, right?
00:09:44.480 --> 00:09:48.720
Because we don't want to use our data or any of our clients' data, obviously, um to demo.
00:09:48.879 --> 00:09:51.759
So we had to come up with completely synthetic data.
00:09:52.000 --> 00:09:58.799
And there's things like um AdventureWorks out there or Contoso or whatever, you know, the Microsoft thing.
00:09:58.879 --> 00:10:00.000
So there's there's a lot of stuff.
00:10:00.080 --> 00:10:01.919
But but you actually started from scratch.
00:10:02.399 --> 00:10:03.919
Why don't you talk about that a little bit?
00:10:04.000 --> 00:10:04.799
I thought it was pretty cool.
00:10:05.200 --> 00:10:06.000
Yeah, definitely.
00:10:06.159 --> 00:10:10.000
So as the as the idea to start from scratch actually came from AI itself.
00:10:10.240 --> 00:10:13.919
Um, I I I just started asking it like, how do we get around this problem?
00:10:14.080 --> 00:10:15.440
Help me brainstorm this, please.
00:10:15.679 --> 00:10:18.720
Got that idea from it, and then was like, yes, let's go for it.
00:10:18.960 --> 00:10:34.799
Um so I basically vibe coded it's it's nothing crazy, but it's a it's a SQL script that builds out tables, generates random data, generates customer sales reps, products, et cetera, targets, um which then you just kind of run it.
00:10:34.960 --> 00:10:42.399
It takes like 20 minutes because you know, it it worked worked at first.
00:10:42.799 --> 00:10:48.080
And then where I got myself in trouble is I started trying to tweak it to try to make it seem more real.
00:10:48.320 --> 00:10:50.159
It's really hard to make like seasonality.
00:10:50.320 --> 00:10:51.759
Well, seasonality actually is easy.
00:10:51.840 --> 00:10:52.720
That's that's not hard.
00:10:52.960 --> 00:10:53.120
Okay.
00:10:53.360 --> 00:10:57.519
But like a customer leaving and then maybe returning a year later, right?
00:10:57.600 --> 00:10:59.679
Stories like that, really, really hard to find.
00:10:59.840 --> 00:11:00.240
Yeah.
00:11:00.480 --> 00:11:07.679
Um, so I uh was trying to prompt it to do that, and I ended up destroying pretty much just wiping out the thing.
00:11:07.919 --> 00:11:10.240
I mean, I didn't wreck it completely, but it made zero sense.
00:11:10.320 --> 00:11:19.440
Like I'd go ask, because I use Claude to connect to our MCP server to talk to the data, and I'd be like, hey, tell me our you know sales on this customer versus their target.
00:11:19.600 --> 00:11:22.320
And it's like, well, they are 2,000% above their target.
00:11:22.399 --> 00:11:24.080
You should talk to the person who set these targets.
00:11:24.240 --> 00:11:25.360
I don't know what they were thinking.
00:11:25.600 --> 00:11:26.639
Like, oh, oops.
00:11:28.559 --> 00:11:38.159
So that was when I learned that I really want Git to back up my code base when I'm Oh, you weren't you didn't have that in a repository?
00:11:38.480 --> 00:11:39.200
I had to start from scratch.
00:11:39.519 --> 00:11:40.000
Oh, that sucks.
00:11:40.320 --> 00:11:41.279
Yeah, it was painful.
00:11:41.759 --> 00:11:44.399
And we use we use uh Azure DevOps, right?
00:11:44.639 --> 00:11:44.799
Yeah.
00:11:45.120 --> 00:11:48.080
Are you s are you starting to use Git now that you're doing more Vibe code?
00:11:48.559 --> 00:11:49.360
I still use DevOps.
00:11:49.519 --> 00:11:51.279
I mean it's Git on the end.
00:11:51.759 --> 00:11:56.320
Um Yeah, I would like to get more into Git, but we have we have DevOps licenses.
00:11:56.480 --> 00:11:57.519
It's kind of it's free for us, right?
00:11:57.840 --> 00:11:59.039
So it is Git on DevOps.
00:11:59.279 --> 00:12:01.519
So I just I guess I mean GitHub.
00:12:01.679 --> 00:12:01.759
Yeah.
00:12:02.000 --> 00:12:02.159
Yeah.
00:12:02.240 --> 00:12:02.399
Yeah.
00:12:02.799 --> 00:12:05.200
That's all my hobby stuff I've been doing on that.
00:12:05.519 --> 00:12:05.759
Yeah.
00:12:06.159 --> 00:12:06.399
All right.
00:12:06.480 --> 00:12:07.759
So then how did you get it?
00:12:08.000 --> 00:12:09.039
So you started over, I guess.
00:12:09.440 --> 00:12:10.559
Started from scratch, essentially.
00:12:10.639 --> 00:12:10.720
Okay.
00:12:11.039 --> 00:12:20.080
Pre-prompted, just just wipe wiped it all out, started from scratch, connected it to a repo, um, so I can commit changes and roll back.
00:12:20.320 --> 00:12:20.399
Yeah.
00:12:20.720 --> 00:12:22.559
But yeah, it was a good learning experience for sure.
00:12:22.639 --> 00:12:28.480
Aaron Powell So one of the problems with sample data sets is that they're static usually, almost always.
00:12:28.720 --> 00:12:34.399
And so as you're using them to test and so on, it's for data that's six months old or two years old.
00:12:34.960 --> 00:12:38.799
So have you figured out a way to sort of keep adding fresh records?
00:12:39.200 --> 00:12:40.320
Yeah, yeah, definitely.
00:12:40.559 --> 00:12:42.480
So that was one of the key requirements I gave it.
00:12:42.639 --> 00:12:43.919
You know, I that's one thing too.
00:12:44.000 --> 00:12:46.799
Like when you Vive Code, you gotta start with really clear.
00:12:47.120 --> 00:12:49.759
This is what it needs, this is what the end product needs to look like.
00:12:49.840 --> 00:12:51.600
Here are the things that it needs to do, et cetera.
00:12:52.080 --> 00:12:53.279
The clearer you can be, the better.
00:12:53.679 --> 00:13:05.120
It created a pretty intense, I mean it's not intense in terms of like sloppy, it's actually really good uh procedure that you call and it goes through several different conditions.
00:13:05.279 --> 00:13:14.240
It has seasonality built in and has customer targets and segments and kind of weights where their purchases go depending on attributes on the customer.
00:13:14.480 --> 00:13:20.720
Um and you can tell it, I want to generate a thousand orders a day, I want to generate 10 orders a day, et cetera.
00:13:21.039 --> 00:13:21.360
That's cool.
00:13:21.679 --> 00:13:30.159
And so it was it was pretty cool to see because you know, with Python, I immediately attached to this is so easy to, you know, like coding typical coding.
00:13:30.480 --> 00:13:33.519
I know SQL's coding, but kind of in a different bit.
00:13:34.240 --> 00:13:36.720
Um immediately attached to how you can do that.
00:13:36.960 --> 00:13:44.080
With SQL, it's always been a little harder to figure out how do I get this thing to write good SQL because you need that data context of it.
00:13:44.320 --> 00:13:47.919
Um data and you almost you have to have a business context too, right?
00:13:48.320 --> 00:13:48.480
Exactly.
00:13:48.879 --> 00:13:54.480
So you know, here's here's the conditions I'm trying to simulate so that I can have an interesting demo.
00:13:54.799 --> 00:13:55.840
Yeah, exactly.
00:13:56.159 --> 00:14:00.879
But basically it it's uh you know, I consider myself a really strong SQL developer.
00:14:00.960 --> 00:14:02.080
I've been doing it for years and years.
00:14:02.480 --> 00:14:04.879
You are truly an expert, yeah.
00:14:05.360 --> 00:14:08.480
And yeah, the stuff in there was was insane, you know.
00:14:08.639 --> 00:14:13.919
It was stuff that I knew, but the way it put it all together and in creative ways was was really cool to see.
00:14:14.240 --> 00:14:14.720
Oh, that's cool.
00:14:14.879 --> 00:14:16.559
How much debugging did you have to do?
00:14:16.879 --> 00:14:17.840
On that proc?
00:14:18.399 --> 00:14:18.799
Yeah.
00:14:19.360 --> 00:14:19.679
Yeah.
00:14:19.840 --> 00:14:21.519
I went back and forth with Claude.
00:14:22.320 --> 00:14:25.919
I don't have a number, but like probably three hours or so back and forth.
00:14:26.240 --> 00:14:34.080
But the thing is, you know, I'd have to go go in there, then I'd have to run everything, um, wait for it to populate, and then check the results.
00:14:34.240 --> 00:14:36.879
Sometimes I'd export CSVs and send it back to Claude.
00:14:36.960 --> 00:14:41.360
I'd ask it like, hey, write me a query that will summarize this so we can make sure this makes sense.
00:14:41.600 --> 00:14:44.240
And I just paste the query in, you know, and run it.
00:14:44.559 --> 00:14:47.919
Did you have it create a test uh test suite to run?
00:14:48.399 --> 00:14:58.559
I did not for that one specifically, mainly because I need to still figure out how to connect AI to Well, I guess we do know how to do it now.
00:14:59.600 --> 00:15:01.440
To the data later.
00:15:02.639 --> 00:15:03.919
Which right now, yes, I can do that.
00:15:04.000 --> 00:15:04.879
So that's a great idea.
00:15:05.039 --> 00:15:09.519
Next time I'm gonna have it just run its own tests, get its own data back, adjust it.
00:15:09.679 --> 00:15:09.840
Yeah.
00:15:10.159 --> 00:15:10.720
It's a great idea.
00:15:12.000 --> 00:15:15.840
So what's next for you with the team here in terms of uh AI coding?
00:15:16.320 --> 00:15:23.360
Yeah, so you know, we're just trying to get as efficient as we possibly can with uh our day-to-day work, right?
00:15:23.519 --> 00:15:33.919
So a couple of things, you know, we're redesigning our standard templates, um, we're making it a bit more modular so that AI has less to edit.
00:15:34.159 --> 00:15:34.320
Right.
00:15:34.480 --> 00:15:34.639
Okay.
00:15:34.799 --> 00:15:36.639
Um there's a couple of driving forces on that.
00:15:36.720 --> 00:15:40.399
You know, obviously the less we need to edit, the less context it has to use on it.
00:15:40.639 --> 00:15:41.039
Yeah.
00:15:41.200 --> 00:15:42.879
Um more targeted it can be.
00:15:43.120 --> 00:15:50.240
And then also, you know, if prices increase like crazy for some reason, we're not we're not completely underwater.
00:15:50.639 --> 00:15:50.960
Right.
00:15:51.360 --> 00:15:59.519
Um and then the other one that I really want to do too that I've heard success about is um getting just automated bug fixes, right?
00:15:59.600 --> 00:16:12.879
So pipeline for a client fails overnight, it's going to automatically go look at the code in our repos, um, make what it thinks is the fix, put a push uh pull request in, and then in the morning we just look at it, right?
00:16:13.120 --> 00:16:15.519
We'll still want that human in the loop for for now.
00:16:16.080 --> 00:16:17.360
Um that's pretty cool.
00:16:17.679 --> 00:16:23.360
But I know some you can do a code review too with a recommended, you know, I've I've reviewed the pull request.
00:16:23.519 --> 00:16:23.679
Yeah.
00:16:23.840 --> 00:16:25.759
I recommend that this one be approved.
00:16:25.840 --> 00:16:26.480
Here's why.
00:16:26.639 --> 00:16:29.120
Yeah you still might you still might look at it, but it might speed you up.
00:16:29.440 --> 00:16:33.759
Yeah, if we got trusty enough with it, yeah, maybe we have a reviewer that reviews all the requests.
00:16:33.919 --> 00:16:34.080
Yeah.
00:16:34.399 --> 00:16:38.159
If it's ends up being good enough, it can just automatically push it and rerun it.
00:16:38.240 --> 00:16:38.799
I'd be crazy.
00:16:39.120 --> 00:16:43.120
I think these tools are getting really good on coding where you can almost get there.
00:16:43.360 --> 00:16:48.240
I mean you have to obviously be super careful around anything that has to do with security, exactly.
00:16:48.480 --> 00:16:51.120
Or that's super important um operational data.
00:16:51.200 --> 00:16:54.799
You don't want to put bad data in the data lake in the lake house.
00:16:54.960 --> 00:17:08.720
Um but because it's funny, I was reading an article about why do LLMs hallucinate so much just in free text conversations or summarizing dictation or whatever, but in code it really nails it.
00:17:08.799 --> 00:17:19.200
And and some of the theories were that, well, there's so much code out there that you can train way, way more than sort of random um prose that's out there.
00:17:19.359 --> 00:17:20.880
And it's way more structured.
00:17:21.279 --> 00:17:28.000
And the languages are very confined, like the the total vocabulary and the syntax is not that big.
00:17:28.240 --> 00:17:29.920
So they can get really good at it.
00:17:30.079 --> 00:17:41.599
And so it's not like the problem they're seeing in law where you know 75% of big legal briefs written with uh with AI have hallucinated stuff in them, made-up cases.
00:17:41.680 --> 00:17:45.359
And I mean it just happened again to another big law firm, amazingly.
00:17:45.599 --> 00:17:46.799
Um it's it's different than that.
00:17:46.960 --> 00:17:56.960
Not that it can't do things wrong or start going down a bad path or introduce some some bug, but it's amazing how good it's gotten this last, I don't know, six months.
00:17:57.039 --> 00:17:57.119
Yeah.
00:17:57.440 --> 00:17:59.519
Just as really complete step change.
00:17:59.920 --> 00:18:01.200
Oh, yeah, I completely agree.
00:18:01.359 --> 00:18:04.079
I mean, it's got public repos out there all over the place.
00:18:04.319 --> 00:18:07.440
Stack Overflow is a gold mine of info now.
00:18:07.920 --> 00:18:08.160
Yeah.
00:18:08.319 --> 00:18:08.480
Yeah.
00:18:08.640 --> 00:18:12.480
I've heard they're actually um selling their data now as training data.
00:18:12.799 --> 00:18:15.519
Oh, that's why models are that's where you used to go.
00:18:16.079 --> 00:18:17.440
If you had a question, that's where you went.
00:18:17.680 --> 00:18:18.079
Yeah, exactly.
00:18:19.200 --> 00:18:20.720
I used to go there every single day.
00:18:21.039 --> 00:18:22.640
I'd always have Stack Overflow open.
00:18:22.720 --> 00:18:26.960
I haven't opened it in four or five, probably more than that, probably like six months at this point.
00:18:27.279 --> 00:18:28.160
Yeah, I'm not surprised.
00:18:28.880 --> 00:18:30.799
I just get answers faster, you know, less digging.
00:18:31.279 --> 00:18:47.359
I think next time I'd love to talk about sort of culturally what it's been like to get the team on board with that because there's a there's a wide range of reactions to from oh, I can't wait to do all write all my code that way to gosh, one of the things I really enjoy about my job is crafting code.
00:18:47.599 --> 00:18:48.799
Like it's almost an art.
00:18:48.960 --> 00:18:50.319
So I don't really want to give that up.
00:18:50.480 --> 00:18:52.079
So anyway, we'll talk about that next time.
00:18:52.240 --> 00:18:52.799
Yeah, absolutely.
00:18:53.039 --> 00:18:55.279
But uh good discussion and uh talk to you soon.
00:18:55.440 --> 00:18:56.079
Yeah, thank you.
00:18:56.160 --> 00:18:56.559
Appreciate it.
00:18:56.799 --> 00:18:57.200
Talk to you later.
00:00:05.759 --> 00:00:07.759
Welcome to the Dashboard Effect Podcast.
00:00:07.839 --> 00:00:08.640
I'm Brick Thompson.
00:00:08.880 --> 00:00:09.759
And I'm Landon Oakes.
00:00:09.919 --> 00:00:10.560
Hey, Landon.
00:00:10.640 --> 00:00:14.800
So we thought today we'd talk a bit about vibe coding using AI.
00:00:15.759 --> 00:00:27.440
I've been doing some personally over the last few months, especially since about I don't know, Thanksgiving, Christmas time last year when I think it was Opus 4.5 came out.
00:00:27.679 --> 00:00:28.559
It was pretty darn good.
00:00:28.640 --> 00:00:31.039
All of a sudden Claude Code was really good.
00:00:31.199 --> 00:00:34.000
I don't do any coding really professionally at all.
00:00:34.159 --> 00:00:36.640
Haven't for you know a couple decades at least.
00:00:36.799 --> 00:00:38.240
Not probably a couple decades.
00:00:38.479 --> 00:00:40.960
But all of a sudden, I'm doing all kinds of coding.
00:00:41.039 --> 00:00:43.600
And it's really fun to be able to do that.
00:00:43.840 --> 00:00:49.119
And then when Opus 4.6 came out from Anthropic, I think it was February 6th.
00:00:49.200 --> 00:00:51.920
Uh same time GPT 5.4 came out.
00:00:52.159 --> 00:00:57.280
Anyway, I was I was still doing the vibe coding there using uh cloud code, still fantastic.
00:00:57.600 --> 00:01:00.880
Now, just yesterday we got GPT 5.5 out.
00:01:01.039 --> 00:01:04.400
Codex, you know, the the OpenAI app is updated.
00:01:04.640 --> 00:01:06.640
I'll probably be spending my weekend doing that.
00:01:06.799 --> 00:01:08.719
So anyway, I thought we were just talking about it.
00:01:08.799 --> 00:01:15.519
We're doing uh not really vibe coding here at Blue Margin, but we are using AI tools to help us build our stuff.
00:01:15.680 --> 00:01:17.680
I know you've been experimenting quite a lot.
00:01:17.920 --> 00:01:18.000
Yeah.
00:01:18.239 --> 00:01:19.599
Um maybe maybe we can start there.
00:01:19.680 --> 00:01:26.640
We just talk about what we're doing, what we're finding is helpful, where it's working well, where we still have to exercise a lot of care.
00:01:27.280 --> 00:01:28.400
Yeah, definitely.
00:01:28.719 --> 00:01:29.200
Let's see.
00:01:29.359 --> 00:01:45.040
So and we've we've done several things recently from all the way from uh you know automating, not necessarily automating, but vibecoding a uh API notebook that that pulls data out of out of a source, um, which has been interesting.
00:01:45.200 --> 00:01:56.400
We've you know created synthetic data sets and demos, demo data sets, because one of the struggles is you want a data set that replicates a business, but it you also want one that's updating and new.
00:01:56.640 --> 00:01:56.799
Yeah.
00:01:57.040 --> 00:02:02.719
Um so you can kind of create your own now without a ton of effort, which has been really cool.
00:02:03.040 --> 00:02:05.680
Um Yeah, several more that I'm probably not thinking of.
00:02:06.079 --> 00:02:06.400
It is cool.
00:02:06.560 --> 00:02:16.639
So you you manage our platform team, and so you're you and your guys manage all of our clients' data lake houses, but a big part of that is pulling in new data sources.
00:02:16.960 --> 00:02:36.879
So creating uh you know a really robust data pipeline that can run daily or multiple times a day and has a graceful fail mode if something goes wrong so it doesn't leave the data lake in bad shape and sets off alerts and you know makes it easy for the platform engineers, well, easier, to go in and fix it.
00:02:37.120 --> 00:02:41.680
But also right up front, when a new data source comes from one of our clients, we've got to make that connection.
00:02:41.840 --> 00:02:48.080
So we've got a series of Jupyter notebooks, depending on what type of data source it is, so we can make that connection.
00:02:48.319 --> 00:02:53.039
In some cases, um, I think I don't know, somewhere like a hundred data sources.
00:02:53.120 --> 00:02:57.439
We have something we've used before, so we at least something have something we can start with.
00:02:57.599 --> 00:03:02.400
Um sometimes we get brand new stuff though, like uh like a totally novel API.
00:03:02.639 --> 00:03:06.159
It's not uncommon for API documentation to not be great.
00:03:06.400 --> 00:03:06.719
Yeah.
00:03:06.960 --> 00:03:10.879
And so, yeah, that's a perfect place to use vibe coding.
00:03:10.960 --> 00:03:12.879
Um, but we want to start with our standards.
00:03:12.960 --> 00:03:16.560
We have best practices, we want to do all those things I was talking about.
00:03:16.719 --> 00:03:18.800
So how do you how do you start?
00:03:18.960 --> 00:03:23.039
How do you get uh your I don't I think you guys are using VS Code with Claude.
00:03:23.360 --> 00:03:24.719
Yeah, if I'm if I recall.
00:03:24.800 --> 00:03:32.240
So how do you get it to know what your target architecture is and make sure it does a good job and doesn't go off and do something kind of crazy?
00:03:32.639 --> 00:03:33.120
Yeah, yeah.
00:03:33.199 --> 00:03:34.479
So it's still a work in progress.
00:03:34.639 --> 00:03:38.879
We're still refining it, which I'm sure all things AI are going to be that way.
00:03:39.120 --> 00:03:41.199
But um right now we're using skills.
00:03:41.360 --> 00:03:49.199
So we've you know kind of trained skills to uh learn what our current best practice template is.
00:03:49.280 --> 00:03:51.680
So what some of our best Python engineers made.
00:03:51.840 --> 00:03:55.759
Um we gave it that and we said, hey, this is the stuff you need to follow.
00:03:55.840 --> 00:03:58.479
Here are the things you shouldn't do, here are the things you should do.
00:03:58.719 --> 00:04:04.080
Um it's actually quite interesting because we we ended up trying to put everything in one skill.
00:04:04.319 --> 00:04:07.840
And that so that includes like how do I handle all these edge cases?
00:04:07.919 --> 00:04:09.520
How do I go out and find documentation?
00:04:09.599 --> 00:04:11.199
What do I do if I can't find the documentation?
00:04:11.360 --> 00:04:11.520
Yeah.
00:04:14.639 --> 00:04:14.800
Okay.
00:04:15.120 --> 00:04:15.280
Yeah.
00:04:15.439 --> 00:04:18.399
And we found some weird behavior with that.
00:04:18.639 --> 00:04:20.800
It's almost like we just gave it too much.
00:04:20.959 --> 00:04:21.120
Yeah.
00:04:21.360 --> 00:04:29.680
So one of the things we're experimenting with now, which seems a little bit more promising, is having discrete skills that can call each other.
00:04:30.079 --> 00:04:30.720
Oh, that's okay.
00:04:30.879 --> 00:04:34.160
Do you have like an orchestrator skill that knows which skills to call?
00:04:34.720 --> 00:04:43.120
So, like, for example, I'm imagining you can tell me if this is what you're doing or not, but um you have a new API, you can have a skill that goes out and reads the API documentation.
00:04:43.439 --> 00:04:43.680
Exactly.
00:04:43.920 --> 00:04:54.399
For example, that's actually what what led us to the idea because we are having a lot of lot of trouble with the API documentation, because not all documentation is AI friendly.
00:04:54.480 --> 00:04:59.199
I just can't read it because they put it in JavaScript or there's some weird backend on it.
00:04:59.279 --> 00:05:00.720
So Yeah, interesting.
00:05:00.959 --> 00:05:06.879
And some of these APIs have sort of secret off-the-menu stuff that you have to know to really make it work well.
00:05:07.120 --> 00:05:07.199
Yeah.
00:05:07.360 --> 00:05:07.519
Yeah.
00:05:07.839 --> 00:05:08.079
Yeah.
00:05:08.560 --> 00:05:13.040
And then sometimes the APIs don't work well, so you end up going to the old ODBC or something.
00:05:13.279 --> 00:05:13.360
Yeah.
00:05:13.680 --> 00:05:14.560
Anyway, yeah.
00:05:14.879 --> 00:05:19.600
So um I know you've got goals around bringing down the amount of time it'll take to create one of those.
00:05:19.920 --> 00:05:41.360
One of the interesting things uh to me that uh you told me you found out was that it's not necessarily the hands-on keyboard time, but there's a lot of time around that, getting access to the data source, getting permissions set up right, getting the OAuth set up, you know, uh the system accounts, whatever you need, um uh as well as you know, looking at code and and doing reviews and all that.
00:05:41.600 --> 00:05:48.000
So the hands-on keyboard time can come down significantly, but you can still end up spending days.
00:05:48.160 --> 00:05:50.560
Like may maybe hands-on keyboard is half a day now.
00:05:50.720 --> 00:05:50.800
Yeah.
00:05:51.040 --> 00:05:56.240
But you can still spend days or even a couple weeks getting it done because of all that ancillary stuff.
00:05:56.480 --> 00:05:56.639
Yeah.
00:05:56.800 --> 00:06:03.040
And I sort of wonder if that's going to be the case with a lot of business agentics stuff for a while.
00:06:03.360 --> 00:06:03.519
Yeah.
00:06:03.759 --> 00:06:04.959
I think it probably will, actually.
00:06:05.360 --> 00:06:06.879
Yeah, I would I wouldn't be surprised.
00:06:06.959 --> 00:06:09.759
You know, it's whenever you need to coordinate people, really.
00:06:09.839 --> 00:06:12.639
That's where a lot of time a lot of times our roadblocks come from.
00:06:13.040 --> 00:06:13.920
So Yeah.
00:06:14.079 --> 00:06:15.199
Yeah, it's true.
00:06:15.519 --> 00:06:18.639
It's gonna it's gonna be interesting to see how all of that plays out.
00:06:18.879 --> 00:06:30.720
Like if you tried to build a skill that said, okay, contact the client, get them to set up the author, it's just not gonna work because you need to talk through that very uh variable thing that happens, I think.
00:06:31.120 --> 00:06:31.600
Yeah, yeah.
00:06:31.680 --> 00:06:38.079
And it's you know, we we've seen just hundreds and hundreds and hundreds of sources, and it's just always so completely different.
00:06:38.319 --> 00:06:38.560
Yeah.
00:06:38.800 --> 00:06:41.439
You know, sometimes there's nothing even online about it, right?
00:06:41.600 --> 00:06:41.680
Yeah.
00:06:42.079 --> 00:06:43.199
We've run into that before.
00:06:43.439 --> 00:06:46.480
People just don't talk about it online.
00:06:46.639 --> 00:06:51.279
So the AI doesn't know, doesn't isn't trained on it, it doesn't know where to go find anything about it.
00:06:51.519 --> 00:06:52.879
Kind of runs into a dead end.
00:06:53.120 --> 00:06:53.439
Yeah.
00:06:53.759 --> 00:06:57.839
Do you think does Claude Code do a pretty good job of following our best practices?
00:06:58.800 --> 00:07:00.160
It's I don't know.
00:07:00.240 --> 00:07:02.560
I'd say like 80-20.
00:07:02.959 --> 00:07:05.360
I might my team might say I'm being too generous to Claude.
00:07:05.600 --> 00:07:05.920
Really?
00:07:06.000 --> 00:07:06.319
Yeah.
00:07:06.800 --> 00:07:16.000
Um they uh they have some of their times where it's like, oh well, it just went over here and edited that thing that I didn't want it to edit, right?
00:07:16.240 --> 00:07:16.319
Right.
00:07:16.480 --> 00:07:19.759
And so that's part of one of the reasons why we actually switched to VS Code.
00:07:19.920 --> 00:07:22.319
Um from just using the terminal.
00:07:22.639 --> 00:07:23.519
Yeah, exactly.
00:07:23.839 --> 00:07:29.680
Because one nice thing is that as long as you don't tell it, yeah, just go crazy, edit whatever the heck you want.
00:07:29.839 --> 00:07:34.879
Um, you say, Let me approve edits, it will show you a side-by-side comparison.
00:07:35.040 --> 00:07:38.160
Here's your version, here's what I'm gonna change, and it highlights the changes for you.
00:07:38.240 --> 00:07:41.360
You know, like those of you who might have used um Git compare.
00:07:41.600 --> 00:07:45.040
It's that essentially you're comparing what the AI did to what you have.
00:07:45.199 --> 00:07:45.600
Yeah.
00:07:45.839 --> 00:07:54.240
And so makes it way easier to make sure it doesn't go off the rails and edit something completely different, along with sneaking in, you know, the editing you asked for.
00:07:54.560 --> 00:07:54.959
I hate that.
00:07:55.040 --> 00:08:01.279
Oh, and that was the problem, you know, six months, nine months ago, whenever I would play with vibe coding, because I have been for, I don't know, the last year or so.
00:08:02.000 --> 00:08:08.639
But not that consistently, maybe for the first half of last year, till we got up to the holiday season, because it just wasn't that good.
00:08:08.879 --> 00:08:09.040
Yeah.
00:08:09.199 --> 00:08:17.199
You know, it would produce something that was close, and then you'd risk r request fixes, and it would maybe make the fix but break something else.
00:08:17.360 --> 00:08:20.319
And it was just sort of it was like playing whack-a-mole.
00:08:20.399 --> 00:08:24.480
And then all of a sudden, around Thanksgiving, Christmas time, it stopped feeling like that.
00:08:24.560 --> 00:08:27.519
And then around February, it really stopped feeling like that.
00:08:27.920 --> 00:08:28.160
Yeah.
00:08:28.319 --> 00:08:28.560
Yeah.
00:08:28.800 --> 00:08:29.920
It's gotten so much better.
00:08:30.240 --> 00:08:30.720
Yeah.
00:08:31.040 --> 00:08:35.039
So uh well, you're gonna have to try GPT 5.5, the codex.
00:08:35.200 --> 00:08:35.279
Yeah.
00:08:35.919 --> 00:08:37.600
Uh and uh and let me know how that is.
00:08:37.840 --> 00:08:39.120
I'll be playing with that too.
00:08:39.360 --> 00:08:58.159
Another thing I know you were working on, so we've got I'm doing work for clients to connect Claude and other um agents to a data lakehouse, to their data lake house, and be able to do ad hoc queries for executives to be able to do, you know, get quick answers on financial data or operational data.
00:08:58.559 --> 00:09:03.360
And it seems like that would be easy, but um it's easy to get wrong answers, actually.
00:09:03.600 --> 00:09:24.480
And so we've been building something called a platinum layer, which is to take the gold layer out of the medallion architecture and then uh do some things to it, denormalize it even more, rename columns, rename tables, make sure all of our joins are done in a consistent manner, and then actually stack a bunch of markdown files into the data lake as well with context.
00:09:24.720 --> 00:09:36.240
But then in addition to that, um, we started building custom MCP servers for different clients that that have a lot more instructions about how to go and get an answer for the agent.
00:09:36.399 --> 00:09:40.879
Um, it's been really interesting, and you've been doing some work to build a demo for that.
00:09:41.039 --> 00:09:41.200
Yeah.
00:09:41.360 --> 00:09:44.399
And part of that was sort of vibe coding a data set, right?
00:09:44.480 --> 00:09:48.720
Because we don't want to use our data or any of our clients' data, obviously, um to demo.
00:09:48.879 --> 00:09:51.759
So we had to come up with completely synthetic data.
00:09:52.000 --> 00:09:58.799
And there's things like um AdventureWorks out there or Contoso or whatever, you know, the Microsoft thing.
00:09:58.879 --> 00:10:00.000
So there's there's a lot of stuff.
00:10:00.080 --> 00:10:01.919
But but you actually started from scratch.
00:10:02.399 --> 00:10:03.919
Why don't you talk about that a little bit?
00:10:04.000 --> 00:10:04.799
I thought it was pretty cool.
00:10:05.200 --> 00:10:06.000
Yeah, definitely.
00:10:06.159 --> 00:10:10.000
So as the as the idea to start from scratch actually came from AI itself.
00:10:10.240 --> 00:10:13.919
Um, I I I just started asking it like, how do we get around this problem?
00:10:14.080 --> 00:10:15.440
Help me brainstorm this, please.
00:10:15.679 --> 00:10:18.720
Got that idea from it, and then was like, yes, let's go for it.
00:10:18.960 --> 00:10:34.799
Um so I basically vibe coded it's it's nothing crazy, but it's a it's a SQL script that builds out tables, generates random data, generates customer sales reps, products, et cetera, targets, um which then you just kind of run it.
00:10:34.960 --> 00:10:42.399
It takes like 20 minutes because you know, it it worked worked at first.
00:10:42.799 --> 00:10:48.080
And then where I got myself in trouble is I started trying to tweak it to try to make it seem more real.
00:10:48.320 --> 00:10:50.159
It's really hard to make like seasonality.
00:10:50.320 --> 00:10:51.759
Well, seasonality actually is easy.
00:10:51.840 --> 00:10:52.720
That's that's not hard.
00:10:52.960 --> 00:10:53.120
Okay.
00:10:53.360 --> 00:10:57.519
But like a customer leaving and then maybe returning a year later, right?
00:10:57.600 --> 00:10:59.679
Stories like that, really, really hard to find.
00:10:59.840 --> 00:11:00.240
Yeah.
00:11:00.480 --> 00:11:07.679
Um, so I uh was trying to prompt it to do that, and I ended up destroying pretty much just wiping out the thing.
00:11:07.919 --> 00:11:10.240
I mean, I didn't wreck it completely, but it made zero sense.
00:11:10.320 --> 00:11:19.440
Like I'd go ask, because I use Claude to connect to our MCP server to talk to the data, and I'd be like, hey, tell me our you know sales on this customer versus their target.
00:11:19.600 --> 00:11:22.320
And it's like, well, they are 2,000% above their target.
00:11:22.399 --> 00:11:24.080
You should talk to the person who set these targets.
00:11:24.240 --> 00:11:25.360
I don't know what they were thinking.
00:11:25.600 --> 00:11:26.639
Like, oh, oops.
00:11:28.559 --> 00:11:38.159
So that was when I learned that I really want Git to back up my code base when I'm Oh, you weren't you didn't have that in a repository?
00:11:38.480 --> 00:11:39.200
I had to start from scratch.
00:11:39.519 --> 00:11:40.000
Oh, that sucks.
00:11:40.320 --> 00:11:41.279
Yeah, it was painful.
00:11:41.759 --> 00:11:44.399
And we use we use uh Azure DevOps, right?
00:11:44.639 --> 00:11:44.799
Yeah.
00:11:45.120 --> 00:11:48.080
Are you s are you starting to use Git now that you're doing more Vibe code?
00:11:48.559 --> 00:11:49.360
I still use DevOps.
00:11:49.519 --> 00:11:51.279
I mean it's Git on the end.
00:11:51.759 --> 00:11:56.320
Um Yeah, I would like to get more into Git, but we have we have DevOps licenses.
00:11:56.480 --> 00:11:57.519
It's kind of it's free for us, right?
00:11:57.840 --> 00:11:59.039
So it is Git on DevOps.
00:11:59.279 --> 00:12:01.519
So I just I guess I mean GitHub.
00:12:01.679 --> 00:12:01.759
Yeah.
00:12:02.000 --> 00:12:02.159
Yeah.
00:12:02.240 --> 00:12:02.399
Yeah.
00:12:02.799 --> 00:12:05.200
That's all my hobby stuff I've been doing on that.
00:12:05.519 --> 00:12:05.759
Yeah.
00:12:06.159 --> 00:12:06.399
All right.
00:12:06.480 --> 00:12:07.759
So then how did you get it?
00:12:08.000 --> 00:12:09.039
So you started over, I guess.
00:12:09.440 --> 00:12:10.559
Started from scratch, essentially.
00:12:10.639 --> 00:12:10.720
Okay.
00:12:11.039 --> 00:12:20.080
Pre-prompted, just just wipe wiped it all out, started from scratch, connected it to a repo, um, so I can commit changes and roll back.
00:12:20.320 --> 00:12:20.399
Yeah.
00:12:20.720 --> 00:12:22.559
But yeah, it was a good learning experience for sure.
00:12:22.639 --> 00:12:28.480
Aaron Powell So one of the problems with sample data sets is that they're static usually, almost always.
00:12:28.720 --> 00:12:34.399
And so as you're using them to test and so on, it's for data that's six months old or two years old.
00:12:34.960 --> 00:12:38.799
So have you figured out a way to sort of keep adding fresh records?
00:12:39.200 --> 00:12:40.320
Yeah, yeah, definitely.
00:12:40.559 --> 00:12:42.480
So that was one of the key requirements I gave it.
00:12:42.639 --> 00:12:43.919
You know, I that's one thing too.
00:12:44.000 --> 00:12:46.799
Like when you Vive Code, you gotta start with really clear.
00:12:47.120 --> 00:12:49.759
This is what it needs, this is what the end product needs to look like.
00:12:49.840 --> 00:12:51.600
Here are the things that it needs to do, et cetera.
00:12:52.080 --> 00:12:53.279
The clearer you can be, the better.
00:12:53.679 --> 00:13:05.120
It created a pretty intense, I mean it's not intense in terms of like sloppy, it's actually really good uh procedure that you call and it goes through several different conditions.
00:13:05.279 --> 00:13:14.240
It has seasonality built in and has customer targets and segments and kind of weights where their purchases go depending on attributes on the customer.
00:13:14.480 --> 00:13:20.720
Um and you can tell it, I want to generate a thousand orders a day, I want to generate 10 orders a day, et cetera.
00:13:21.039 --> 00:13:21.360
That's cool.
00:13:21.679 --> 00:13:30.159
And so it was it was pretty cool to see because you know, with Python, I immediately attached to this is so easy to, you know, like coding typical coding.
00:13:30.480 --> 00:13:33.519
I know SQL's coding, but kind of in a different bit.
00:13:34.240 --> 00:13:36.720
Um immediately attached to how you can do that.
00:13:36.960 --> 00:13:44.080
With SQL, it's always been a little harder to figure out how do I get this thing to write good SQL because you need that data context of it.
00:13:44.320 --> 00:13:47.919
Um data and you almost you have to have a business context too, right?
00:13:48.320 --> 00:13:48.480
Exactly.
00:13:48.879 --> 00:13:54.480
So you know, here's here's the conditions I'm trying to simulate so that I can have an interesting demo.
00:13:54.799 --> 00:13:55.840
Yeah, exactly.
00:13:56.159 --> 00:14:00.879
But basically it it's uh you know, I consider myself a really strong SQL developer.
00:14:00.960 --> 00:14:02.080
I've been doing it for years and years.
00:14:02.480 --> 00:14:04.879
You are truly an expert, yeah.
00:14:05.360 --> 00:14:08.480
And yeah, the stuff in there was was insane, you know.
00:14:08.639 --> 00:14:13.919
It was stuff that I knew, but the way it put it all together and in creative ways was was really cool to see.
00:14:14.240 --> 00:14:14.720
Oh, that's cool.
00:14:14.879 --> 00:14:16.559
How much debugging did you have to do?
00:14:16.879 --> 00:14:17.840
On that proc?
00:14:18.399 --> 00:14:18.799
Yeah.
00:14:19.360 --> 00:14:19.679
Yeah.
00:14:19.840 --> 00:14:21.519
I went back and forth with Claude.
00:14:22.320 --> 00:14:25.919
I don't have a number, but like probably three hours or so back and forth.
00:14:26.240 --> 00:14:34.080
But the thing is, you know, I'd have to go go in there, then I'd have to run everything, um, wait for it to populate, and then check the results.
00:14:34.240 --> 00:14:36.879
Sometimes I'd export CSVs and send it back to Claude.
00:14:36.960 --> 00:14:41.360
I'd ask it like, hey, write me a query that will summarize this so we can make sure this makes sense.
00:14:41.600 --> 00:14:44.240
And I just paste the query in, you know, and run it.
00:14:44.559 --> 00:14:47.919
Did you have it create a test uh test suite to run?
00:14:48.399 --> 00:14:58.559
I did not for that one specifically, mainly because I need to still figure out how to connect AI to Well, I guess we do know how to do it now.
00:14:59.600 --> 00:15:01.440
To the data later.
00:15:02.639 --> 00:15:03.919
Which right now, yes, I can do that.
00:15:04.000 --> 00:15:04.879
So that's a great idea.
00:15:05.039 --> 00:15:09.519
Next time I'm gonna have it just run its own tests, get its own data back, adjust it.
00:15:09.679 --> 00:15:09.840
Yeah.
00:15:10.159 --> 00:15:10.720
It's a great idea.
00:15:12.000 --> 00:15:15.840
So what's next for you with the team here in terms of uh AI coding?
00:15:16.320 --> 00:15:23.360
Yeah, so you know, we're just trying to get as efficient as we possibly can with uh our day-to-day work, right?
00:15:23.519 --> 00:15:33.919
So a couple of things, you know, we're redesigning our standard templates, um, we're making it a bit more modular so that AI has less to edit.
00:15:34.159 --> 00:15:34.320
Right.
00:15:34.480 --> 00:15:34.639
Okay.
00:15:34.799 --> 00:15:36.639
Um there's a couple of driving forces on that.
00:15:36.720 --> 00:15:40.399
You know, obviously the less we need to edit, the less context it has to use on it.
00:15:40.639 --> 00:15:41.039
Yeah.
00:15:41.200 --> 00:15:42.879
Um more targeted it can be.
00:15:43.120 --> 00:15:50.240
And then also, you know, if prices increase like crazy for some reason, we're not we're not completely underwater.
00:15:50.639 --> 00:15:50.960
Right.
00:15:51.360 --> 00:15:59.519
Um and then the other one that I really want to do too that I've heard success about is um getting just automated bug fixes, right?
00:15:59.600 --> 00:16:12.879
So pipeline for a client fails overnight, it's going to automatically go look at the code in our repos, um, make what it thinks is the fix, put a push uh pull request in, and then in the morning we just look at it, right?
00:16:13.120 --> 00:16:15.519
We'll still want that human in the loop for for now.
00:16:16.080 --> 00:16:17.360
Um that's pretty cool.
00:16:17.679 --> 00:16:23.360
But I know some you can do a code review too with a recommended, you know, I've I've reviewed the pull request.
00:16:23.519 --> 00:16:23.679
Yeah.
00:16:23.840 --> 00:16:25.759
I recommend that this one be approved.
00:16:25.840 --> 00:16:26.480
Here's why.
00:16:26.639 --> 00:16:29.120
Yeah you still might you still might look at it, but it might speed you up.
00:16:29.440 --> 00:16:33.759
Yeah, if we got trusty enough with it, yeah, maybe we have a reviewer that reviews all the requests.
00:16:33.919 --> 00:16:34.080
Yeah.
00:16:34.399 --> 00:16:38.159
If it's ends up being good enough, it can just automatically push it and rerun it.
00:16:38.240 --> 00:16:38.799
I'd be crazy.
00:16:39.120 --> 00:16:43.120
I think these tools are getting really good on coding where you can almost get there.
00:16:43.360 --> 00:16:48.240
I mean you have to obviously be super careful around anything that has to do with security, exactly.
00:16:48.480 --> 00:16:51.120
Or that's super important um operational data.
00:16:51.200 --> 00:16:54.799
You don't want to put bad data in the data lake in the lake house.
00:16:54.960 --> 00:17:08.720
Um but because it's funny, I was reading an article about why do LLMs hallucinate so much just in free text conversations or summarizing dictation or whatever, but in code it really nails it.
00:17:08.799 --> 00:17:19.200
And and some of the theories were that, well, there's so much code out there that you can train way, way more than sort of random um prose that's out there.
00:17:19.359 --> 00:17:20.880
And it's way more structured.
00:17:21.279 --> 00:17:28.000
And the languages are very confined, like the the total vocabulary and the syntax is not that big.
00:17:28.240 --> 00:17:29.920
So they can get really good at it.
00:17:30.079 --> 00:17:41.599
And so it's not like the problem they're seeing in law where you know 75% of big legal briefs written with uh with AI have hallucinated stuff in them, made-up cases.
00:17:41.680 --> 00:17:45.359
And I mean it just happened again to another big law firm, amazingly.
00:17:45.599 --> 00:17:46.799
Um it's it's different than that.
00:17:46.960 --> 00:17:56.960
Not that it can't do things wrong or start going down a bad path or introduce some some bug, but it's amazing how good it's gotten this last, I don't know, six months.
00:17:57.039 --> 00:17:57.119
Yeah.
00:17:57.440 --> 00:17:59.519
Just as really complete step change.
00:17:59.920 --> 00:18:01.200
Oh, yeah, I completely agree.
00:18:01.359 --> 00:18:04.079
I mean, it's got public repos out there all over the place.
00:18:04.319 --> 00:18:07.440
Stack Overflow is a gold mine of info now.
00:18:07.920 --> 00:18:08.160
Yeah.
00:18:08.319 --> 00:18:08.480
Yeah.
00:18:08.640 --> 00:18:12.480
I've heard they're actually um selling their data now as training data.
00:18:12.799 --> 00:18:15.519
Oh, that's why models are that's where you used to go.
00:18:16.079 --> 00:18:17.440
If you had a question, that's where you went.
00:18:17.680 --> 00:18:18.079
Yeah, exactly.
00:18:19.200 --> 00:18:20.720
I used to go there every single day.
00:18:21.039 --> 00:18:22.640
I'd always have Stack Overflow open.
00:18:22.720 --> 00:18:26.960
I haven't opened it in four or five, probably more than that, probably like six months at this point.
00:18:27.279 --> 00:18:28.160
Yeah, I'm not surprised.
00:18:28.880 --> 00:18:30.799
I just get answers faster, you know, less digging.
00:18:31.279 --> 00:18:47.359
I think next time I'd love to talk about sort of culturally what it's been like to get the team on board with that because there's a there's a wide range of reactions to from oh, I can't wait to do all write all my code that way to gosh, one of the things I really enjoy about my job is crafting code.
00:18:47.599 --> 00:18:48.799
Like it's almost an art.
00:18:48.960 --> 00:18:50.319
So I don't really want to give that up.
00:18:50.480 --> 00:18:52.079
So anyway, we'll talk about that next time.
00:18:52.240 --> 00:18:52.799
Yeah, absolutely.
00:18:53.039 --> 00:18:55.279
But uh good discussion and uh talk to you soon.
00:18:55.440 --> 00:18:56.079
Yeah, thank you.
00:18:56.160 --> 00:18:56.559
Appreciate it.
00:18:56.799 --> 00:18:57.200
Talk to you later.
