TRANSCRIPT 🔗
WEBVTT
00:00:37.432 --> 00:00:43.292
Today we got Andrew Atkinson, a software engineer, author, and consultant who's currently working at Aura Frames.
00:00:43.332 --> 00:00:43.951
Welcome to the show
00:00:45.213 --> 00:00:45.694
Thanks, Brendan.
00:00:45.852 --> 00:00:46.432
Glad to be here
00:00:47.311 --> 00:00:48.131
Yeah, glad to have you.
00:00:48.701 --> 00:00:49.421
What have you been up to?
00:00:49.421 --> 00:00:55.682
We we met a long, well, it feels like a long time ago, I guess it really wasn't, at the Sin City Ruby.
00:00:55.771 --> 00:01:02.752
And, you know, we've done some work together after that, but I haven't talked to you in, like, at least a year and a half, maybe longer.
00:01:02.811 --> 00:01:03.252
What have you been up to?
00:01:04.159 --> 00:01:04.569
Yeah.
00:01:04.670 --> 00:01:08.590
More or less the same stuff, you know, working with Postgres and Rails professionally.
00:01:08.590 --> 00:01:15.519
And then I think at that time I was finishing up the book, and the book's been now published for two years.
00:01:16.780 --> 00:01:35.429
and so it's been… You know, I kinda had like a promotional period that was pretty in-intensive last year, and then it's been-- year has been pretty chill with, you know, just kinda doing what I do, like writing about projects periodically online, and that brings in some traffic and, you know, that sorta helps with promotion.
00:01:35.429 --> 00:01:40.929
But kinda s- sales on the book side are just at a, a trickle level, and that's fine.
00:01:40.980 --> 00:01:46.650
It's fun to still have it as a, kind of a, a background investment area.
00:01:47.150 --> 00:01:51.560
But yeah lately I've have been working more as a, as an employee.
00:01:51.700 --> 00:01:57.540
I did a stint as running my own consultancy working with companies like Locable and other companies.
00:01:58.510 --> 00:02:06.679
And actually was able to find a really good fit with a company where I'd done some consulting work and there was mutual interest in bringing me on as an employee.
00:02:07.140 --> 00:02:09.860
So I've been working with Aura Frames now for about a year.
00:02:10.290 --> 00:02:16.620
A lot of the same sort of stuff, like working with Rails and Postgres, and uh, continues to be a good fit so, excited to talk about more of that today.
00:02:18.233 --> 00:02:25.463
And if I remember, I, I remember reading the, I think it was pr- it had to be one of your blog posts about the work you were doing at Aura, right?
00:02:25.483 --> 00:02:32.293
And, you know, they have the big influx of demand right around Christmas Day, right?
00:02:32.769 --> 00:02:33.110
Yep
00:02:33.274 --> 00:02:38.564
so, so what is it like know, trying to solve that problem, right?
00:02:38.564 --> 00:02:46.053
Like, i- is it… W- was it… I don't know how much you can talk about this or not, and feel free to deflect, but you know, was it a, is it a Postgres thing?
00:02:46.053 --> 00:02:49.514
Was it Postgres and, like, you know, scaling
Load full transcript
you know, the database?
00:02:49.533 --> 00:02:58.204
Like, how did you even, you know, sort of start to solve a problem with that much, you know, fluc- you know, dynamic… What's the word I'm looking for, right?
00:02:58.204 --> 00:03:01.524
Like, some days, like, the, the traffic is really low, and then all of a sudden it's absurd.
00:03:01.846 --> 00:03:02.396
Right.
00:03:03.156 --> 00:03:05.315
Yeah, like, sometimes like elasticity.
00:03:05.346 --> 00:03:12.096
Well, that's not exactly the surging demand, but yeah, there's lots of ways to talk about it, like, load or surging demand or, you know.
00:03:12.105 --> 00:03:21.505
But basically a four to five times increase on a single day, you know, which actually stresses everything, but most of that stress tends to be on the database.
00:03:21.505 --> 00:03:28.846
We need to capture all that right activity for in our-- in the case of Aura Frames, like new users setting up new frames on Christmas Day.
00:03:28.846 --> 00:03:31.105
Usually they've them as gifts.
00:03:31.115 --> 00:03:47.005
So yeah, historically for the company its you know, I think around 10 years of existence Christmas Day and Mother's Day are two of the big peak load days that really a lot of the architecture for the infrastructure is designed for, you know, to be able to handle.
00:03:47.375 --> 00:03:59.915
And then the rest of the year, the company or, you know, outside of peak events, which the company hopes there's more and more of course, like, because it means that the business is successful and healthy and… But there are those two big ones.
00:03:59.915 --> 00:04:09.435
Outside of the rest of the year the company also wants to, you know, have economical infrastructure costs, so the-- we need to be able to scale things down as well.
00:04:09.446 --> 00:04:20.016
So yeah, prior to me being part of the company I'd say the main pressure point has-- it seems like it was typically the database because it's not, you know, as easily horizontally scalable.
00:04:20.605 --> 00:04:38.505
Like as far as the web app instances we- we're big users of a variety of AWS services, and the web app instances, we can put in an auto-scaling group on AW- AWS and have those scale based on CPU so that we can bring in more instances as needed, and those can scale up and down.
00:04:38.555 --> 00:04:45.066
And that way, you know, we both have capacity when needed, but we can also scale down to control costs.
00:04:46.016 --> 00:04:46.246
Right.
00:04:46.246 --> 00:04:46.516
Yeah.
00:04:46.627 --> 00:04:47.307
with Postgres.
00:04:47.398 --> 00:04:47.937
Right
00:04:48.420 --> 00:04:54.850
so yeah the, the approach usually is like, you know, trying to understand what are the main that we might run into.
00:04:54.850 --> 00:04:57.270
And this would be true of anyone, you know, scaling Postgres.
00:04:57.280 --> 00:05:04.790
Like, it's or, CPU or memory pressure or whatever it might be, what- whatever the main bottlenecks are.
00:05:05.509 --> 00:05:10.280
And then you know, designing solutions that would those bottlenecks.
00:05:10.290 --> 00:05:22.710
So in the case of peak load, you know, we historically prior to… And I wrote about this publicly, so, I can provide these blog post links for the podcast as well.
00:05:22.740 --> 00:05:28.990
But and the company, you know, was able to take a look and review the posts as well, so they were fine with what was shared in them.
00:05:29.780 --> 00:05:41.960
prior to last year's Christmas, at the time of this recording, 2025's Christmas the Order Frames was still running on a single Postgres instance, a single primary for the main application API traffic.
00:05:42.400 --> 00:06:03.330
There were some other instances for auxiliary things like background work and things like that, but the main API was still a single instance scaled up to the biggest instance size that AWS offers, and with the fastest available storage type and, you know, the, the most that we, that the company could purchase for provisioned IOPS.
00:06:03.340 --> 00:06:19.480
So we could have much CPUs, memory and available to handle that peak you know, use vertical scaling to scale up in advance of that load, then afterwards when things settle down back to the normal levels, scale back down.
00:06:20.240 --> 00:06:33.160
And that, you know, that's a typical path is with cloud-based, Postgres is, you know, you're able to move instance classes like that, you know, whether it's AWS or another cloud provider.
00:06:34.400 --> 00:06:44.569
you know, that's pretty sensible because you just gotta click a few buttons and pay a little more money and, you know, you're not investing a huge amount of time in engineering hours and development and maintenance and s- that sort of thing.
00:06:45.389 --> 00:06:46.920
I think it's a pretty sensible path.
00:06:47.420 --> 00:06:48.280
But then, Yeah.
00:06:48.280 --> 00:06:53.590
What had happened at 20… Sorry, I said that wrong before.
00:06:54.319 --> 00:06:58.860
was a single instance through Christmas 2024, and then we put a new solution in place for last year.
00:06:59.330 --> 00:06:59.509
So
00:06:59.622 --> 00:06:59.932
Yep
00:07:01.379 --> 00:07:02.810
which was two Christmases ago,
00:07:03.848 --> 00:07:04.169
Ja, ja
00:07:04.800 --> 00:07:21.069
that's where Aura frames the, the, you know, again, on the largest instance available, peak load, lots of load testing in advance with simulated increased load from what production levels were the previous year, which is a, a cool capability that infrastructure team has built.
00:07:21.819 --> 00:07:35.139
Despite all of that, on the actual Christmas 2024, there ended up being… It was around a two to three-hour total time period where there was a mix of… There, there was basically an outage.
00:07:35.759 --> 00:08:09.790
And one of the main issues was related to replay of replica logs fast enough replicas and how… I wrote up the specific details on a Postgres oriented post, but the… I guess
the summary would be a replication-related failure mode where data was accumulating faster on a dedicated replica volume and due to a kind of a more or less a misconfiguration, wasn't replicated fast enough and it caused the volume to fill.
00:08:10.540 --> 00:08:16.889
This is a, a failure mode that other large high write volume Postgres folks have encountered.
00:08:17.720 --> 00:08:20.100
And so there's various mitigation strategies and that sort of thing.
00:08:20.100 --> 00:08:34.850
And the team… I wasn't part of the team at that point but the team had a, a really nice write-up on, you know, internally on the, as a postmortem about what happened and what the resolution was and what the tuning advice, you know, the tuning config changes were going forward.
00:08:37.559 --> 00:08:49.240
but yeah, I mean, more or less it was that peak write traffic and being able to replay that to a an async replica fast enough without running out of disk space as those changes need to be replayed.
00:08:50.039 --> 00:08:52.669
And so, more or
00:08:52.840 --> 00:08:53.960
Is that like a buffer then?
00:08:53.980 --> 00:08:54.840
Is that how that works?
00:08:54.840 --> 00:09:03.580
Like it puts it in a buffer as it's writing and then that bu- like, I, I don't know if I'm using the right terms here, but that buffer filled up the available memory before it could, you know, put it into the database.
00:09:03.580 --> 00:09:05.779
Is that sort of what you're saying?
00:09:06.077 --> 00:09:09.508
I mean, it's, it has to do with Postgres' write-ahead logging system.
00:09:09.508 --> 00:09:27.298
So to make sure that all the changes durable, the changes are written to you know, buffers that are periodically then flushed to disk, and then those files are then replayed or, you know, replicated to a, a second follower instance, and those write-ahead log changes are replayed on that follower instance.
00:09:29.118 --> 00:09:51.427
The durability piece for the write-ahead logs would be more of the primary instance, but there was a config change with, I guess I should have refreshed my own knowledge of the nuanced details of this, but there was a config change in the version of Postgres used 14 on RDS related to how it preserved those write-ahead logs.
00:09:52.138 --> 00:10:04.488
And there was a benefit to it, but the cost to it was this possibility of filling up the, the storage space where the write-ahead logs are stored.
00:10:05.738 --> 00:10:17.538
And because this write-ahead log storage trans, you know, replication or replay is so critical AWS offers a thing with RDS called a dedicated log volume, and that was in use in this case.
00:10:17.888 --> 00:10:22.048
So there that… Actually, like a separate dedicated disk to store these write-ahead logs.
00:10:22.608 --> 00:10:39.288
But there was also a change in 14 that, meant that the, could fill the volume if in the case that the repla- the replica lag was too high.
00:10:40.018 --> 00:10:41.368
And so that's what occurred.
00:10:42.697 --> 00:10:51.457
So the replica lag, you know, we wanna keep a, a reachable secondary replica or follower database in sync with the primary.
00:10:51.827 --> 00:11:00.618
A lot of the businesses run using the replica, like things that don't need to access the primary database, could be admin database or other sorts of functionality.
00:11:01.348 --> 00:11:04.138
So that's continually kept up to date.
00:11:04.378 --> 00:11:11.238
But I believe what the team did at the time to sort of recover things was essentially rebuild a new replica.
00:11:11.298 --> 00:11:19.898
The replica that was in place wasn't gonna be caught up and in order to recover the system I think that's what I would do in this situation.
00:11:19.898 --> 00:11:20.258
I'd have
00:11:20.300 --> 00:11:20.620
Yeah.
00:11:20.658 --> 00:11:33.508
back and review the details, but es-essentially, and again, that would be, I guess, to, you know, to zoom back out a bit, there was a, a Postgres-related failure mode that not have occurred in the following year.
00:11:34.567 --> 00:11:48.858
there, there wasn't a tremendous amount of confidence that the following year's increased load, because the company's been… By the way, the company's been successful and business is growing annually, so which is great.
00:11:48.918 --> 00:11:50.827
So we have to plan for
00:11:51.028 --> 00:11:53.608
I think I got three frames now f- between the family.
00:11:53.830 --> 00:11:54.060
Yeah.
00:11:54.060 --> 00:11:54.340
Awesome.
00:11:54.350 --> 00:11:54.810
That's great to
00:11:54.978 --> 00:11:55.158
Yeah.
00:11:55.240 --> 00:12:01.930
Yeah, and I mean, the company continues to, you know, reach new retail locations and sell online and enter new countries.
00:12:02.220 --> 00:12:12.639
So we forecast increased frames, increased users, and increased activity, which all, you know, on the infrastructure team translates into more pressure on all the pieces.
00:12:12.720 --> 00:12:16.250
But for me specifically, more on the database is what I would think about.
00:12:16.740 --> 00:12:22.370
So we think about like, well, what are our new, our, you know, our new write volume at peak and that kind of thing.
00:12:22.840 --> 00:12:29.159
So, that in year, so now we're getting back to like when I joined in on the party.
00:12:29.750 --> 00:12:48.121
Last year Aura initially brought me in as a consultant to help review some CPU usage issues related to the single primary instance that, that wasn't actually the original failure mode on the Christmas day, but the failure mode Christmas day was something we discussed and was kind of in the background as context.
00:12:48.542 --> 00:13:10.701
We also found though that other activities like which are critical for Postgres like vacuum, running on big tables, there could be some, there could be some sporadic high CPU usage that would then have this kinda knock-on effect of increased latency and slow queries and timeouts and errors.
00:13:11.471 -->
00:13:19.942
And we did a bunch of work to try to diagnose what was causing that forming different hypotheses and testing them.
00:13:22.701 --> 00:13:31.542
The conclusion we came to was we didn't really reach a really strong of circumstances that were the cause that we felt like we could fix.
00:13:32.042 --> 00:13:37.841
And then the other thing that was happening at the same time was, this was mid 2025, so it's 2026 now.
00:13:38.292 --> 00:13:44.961
There was like a ticking clock that was like, we're very concerned as a company about the availability of Postgres through Christmas 2025, right?
00:13:44.961 --> 00:13:45.672
It's coming up.
00:13:46.591 --> 00:14:04.991
like, the discussion was like, let's try to solve this problem that we can reproduce in our load testing environment, but if we can't solve it, let's pivot towards a solution like sharding, where we can distribute this load to more than one Postgres instance, and we can kinda more redundancy.
00:14:04.991 --> 00:14:07.032
You know, we can split the workload basically.
00:14:07.072 --> 00:14:15.052
We can have, redundancy, but we can duplicate some of the same infrastructure pieces like that dedicated log volume I mentioned.
00:14:16.591 --> 00:14:24.532
we can now separate out our work to more instances that each will have their own CPU memory and IOPS resources associated with them.
00:14:25.002 --> 00:14:27.672
And that's ultimately what we did.
00:14:27.682 --> 00:14:39.022
We didn't really solve the path on the single instance, but there's also kind of an understanding of when are we gonna… When do we think that our workload is such that it's not really gonna fit in one more instance?
00:14:39.491 --> 00:14:48.201
And the company ultimately, and I think most companies, they want, you know, you wanna have a high degree of confidence going into a really critical sales period for the company.
00:14:49.152 --> 00:15:07.892
so if that means investing more work to have more compute available and over-provision things, the, the company was definitely more interested in that versus like, you know, continuing to iterate on different possible solutions beyond a certain point for that single instance setup.
00:15:07.902 --> 00:15:08.621
Does that make sense?
00:15:08.621 --> 00:15:11.581
Like, you know, basically there were two paths, and we had to choose one path.
00:15:12.322 --> 00:15:15.442
Yeah, I mean, it's a, you know, it's, it's a risk analysis, right?
00:15:15.442 --> 00:15:20.721
Like, we're not sure what the problem actually is with the single instance.
00:15:20.912 --> 00:15:24.502
We only have six months to solve a big problem.
00:15:25.392 --> 00:15:33.882
We need to move, and even if it means that we have to spend a little bit more in over-provision in the short term, that is better than losing customers, right?
00:15:33.971 --> 00:15:35.812
I paraphrase.
00:15:35.812 --> 00:15:38.382
I, I would imagine that's what they were thinking, 'cause I was not there.
00:15:38.676
--> 00:15:39.906
Yep, that's a good paraphrase.
00:15:39.906 --> 00:15:40.115
Yeah.
00:15:40.115 --> 00:15:40.416
And then we,
00:15:40.481 --> 00:15:40.642
Yeah
00:15:40.776 --> 00:15:48.526
we were out of to over-provision on the single instance because we were at the biggest instance available, the most IOPS available.
00:15:48.926 --> 00:15:52.885
So really to get, to be able to over-provision, we had to go to more instances.
00:15:52.895 --> 00:15:54.966
So then the question became like, well, what do we do?
00:15:55.566 --> 00:15:57.156
You know, how do we go to more instances?
00:15:58.731 --> 00:16:06.282
And are you able to talk about like the high level points of like, you know, what into that thought process, right?
00:16:06.282 --> 00:16:20.201
Like, you know, think about sharding, it's like, all right, I gotta have some sort of universal ID, and I gotta, you know, sort of split the, split the database according- or split the customer base accordingly, then my brain explodes.
00:16:20.221 --> 00:16:20.672
So
00:16:23.196 --> 00:16:32.186
I learned the I will start off by repeating the phrase that someone el- I don't know who to credit this to, but it's like you can't s- you can't do sh- or you can't spell shard without the word hard.
00:16:32.950 --> 00:16:33.300
Right.
00:16:33.300 --> 00:16:33.680
Right.
00:16:33.926 --> 00:16:36.306
It's like it's the devil is in the details, you know?
00:16:36.306 --> 00:16:43.276
It ultimately like on one side you have, you wanna expand capacity, and it's really, it's horizontally, right?
00:16:43.316 --> 00:16:54.015
It's more server instances running Postgres not vertically, like just moving to a bigger box, you know, with faster disks and more of them and stuff like that.
00:16:54.056 --> 00:16:57.865
So, We wanna go horizontally, but data lives in a table.
00:16:57.875 --> 00:17:02.125
So typically, when people talk about sharding, they talk about row-level sharding.
00:17:02.196 --> 00:17:08.806
So, you know, taking one big table and maybe putting half of the rows on one instance and half on the other.
00:17:09.205 --> 00:17:17.005
So, the primary task that the application or connection proxy or something needs to do is figure out, well, how do I divide these up?
00:17:17.005 --> 00:17:18.485
Where do I send writes?
00:17:18.485 --> 00:17:19.955
Where do I send reads?
00:17:20.855 --> 00:17:36.005
so you need, you know, you need like a sharding key, something that would say, "For this table, here's how I'm gonna divide up my data." Yeah, that's all pretty hard, and because, you know, not every table, depending on your schema design, might fit like a, a good pattern.
00:17:36.685 --> 00:17:49.215
And in terms of a high level, what we ultimately did you could argue is actually not sharding in a, a kind of a strict definition if the definition is row-level sharding.
00:17:50.425 --> 00:17:59.745
we kicked around the term whole table sharding or I also kinda like workload-based sharding, but it
on what you're trying to communicate.
00:17:59.745 --> 00:18:14.035
Like, so with the whole table, basically what another engineer had done prior to me joining was kinda analyze, like here's all the top 10 tables by their actual workload, like what they're actually doing, inserts, updates, and deletes, and selects.
00:18:14.755 --> 00:18:17.605
And we also… They also tend to be the biggest tables.
00:18:18.355 --> 00:18:22.205
These are you know, tables with billions of rows that are not partitioned.
00:18:22.355 --> 00:18:37.765
And so we talked about, okay, well, Postgres lets us do table which we can use to break up a big table, and that can help in some ways, but we still are limited to that, partitions of that table, those smaller pieces being on one instance.
00:18:38.215 --> 00:18:40.575
So it doesn't really help us in terms of distributing the load.
00:18:42.485 --> 00:18:45.245
The tables are, for the most part, unpartitioned, and they're big.
00:18:45.336 --> 00:18:54.125
And so besides the that's happening on them that the application is doing, like the Rails app, there's also like vacuum needs to run.
00:18:54.125 --> 00:19:00.325
And if there's, you know, hundreds of millions of rows accumulated or tens of millions, that can take a long time to do.
00:19:00.336 --> 00:19:05.365
It can be of processing time to do the different things that vacuum does.
00:19:07.665 --> 00:19:09.155
so that's something to consider.
00:19:09.395 --> 00:19:13.125
Besides the foreground application activity is all the background work.
00:19:13.325 --> 00:19:17.836
And then there's things like, you know, over time you wanna adjust the indexes so they support the queries better.
00:19:18.165 --> 00:19:22.945
You might wanna rebuild your indexes so that, Old row versions are no longer part of the indexes.
00:19:22.955 --> 00:19:24.875
They can be smaller and faster to traverse.
00:19:25.995 --> 00:19:29.485
so you might wanna even rebuild your table, repack your table.
00:19:30.165 --> 00:19:48.175
So there's, there's , limitations to what we did, but what we found was if we just sort of make a mapping of here's our 10 biggest tables you know, what sort of additional instances would we need to distribute these to, to be able to get some real gains?
00:19:48.745 --> 00:19:56.775
And what we found was ended up with a total of going from one to eight total primary Postgres databases for Christmas 2025.
00:19:57.745 --> 00:20:10.205
what we did was we decided that for two of our biggest, most critical tables, they were actually gonna get their own entire server instance, which I had never seen before and I thought was pretty wild initially.
00:20:10.285 --> 00:20:11.545
Like, uh, really
00:20:11.575 --> 00:20:11.895
Yeah, that's
00:20:12.025 --> 00:20:16.795
table, like one, like, on, on a, a large instance that costs a lot of money.
00:20:16.846 --> 00:20:24.755
Like, but it's, you know, it's a critical part of the set of tables that are needed to be able to service API requests.
00:20:25.065 --> 00:20:36.245
And ultimately, by doing that, it's a way to chop up the workload to lots of instances, but not need to get into the of routing rows around.
00:20:37.175 --> 00:20:48.665
what we could do then I can jump into… I think two more interesting things from that are how we actually did the, the data movement, and then how we actually configured that from the Rails app.
00:20:48.745 --> 00:20:49.235
So I'm, I
00:20:50.039 --> 00:21:07.779
Yeah, I, I, that's pretty much the two questions I was gonna ask, and maybe on the second one I'll say, like, how do you-- Like, is it… You know, if you have to do a query, you know, that joins a table across two databases, like, how do you do that?
00:21:08.109 --> 00:21:11.339
Maybe that's part of the answer that you were about to get into.
00:21:11.769 --> 00:21:22.569
Or is it designed that, you know, the, the query load there is just low, and you don't actually have to do that that often, and that's why it was a good candidate to be on its own instance anyway.
00:21:23.009 --> 00:21:24.199
But yeah, l- l- keep going.
00:21:24.209 --> 00:21:24.979
You're, you're on a roll.
00:21:25.825 --> 00:21:26.045
Yeah.
00:21:26.096 --> 00:21:32.705
The big thing right away is we can't-- have to say goodbye to SQL joins across tables that are not in the same database.
00:21:32.765 --> 00:21:45.865
So, like, if we have table A and table B, before we could join them together on a common column value, and now if table A is on server one and table B is on server two with regular Postgres anyways, you can't just join.
00:21:45.875 --> 00:21:50.065
It'll just say, "This is a table I don't know about." If you try to, the SQL statement will be invalid.
00:21:50.615 --> 00:22:03.305
So we we moved to what some folks call, Ultimately, what we did was the databases were reachable from the Rails application, and ultimately what we did was move to two select queries, you know, to keep it simple.
00:22:03.515 --> 00:22:05.515
So it's like, you know, we select the primary key.
00:22:06.005 --> 00:22:21.565
In a join, it's like we have the primary key values, and we have or sorry, we have, from the source table, we have the foreign, or probably forward key values that we then would use to query on the second table for its primary key, column with those values.
00:22:21.995 --> 00:22:26.455
So essentially, that's what the application does, just makes two separate queries.
00:22:27.245 --> 00:22:32.635
And the cost with that is additional network latency.
00:22:32.865 --> 00:22:36.765
There's two servers involved now, where there was one before.
00:22:37.755 --> 00:22:38.555
It's two separate.
00:22:38.555 --> 00:22:40.485
But there's also some interesting advantages too.
00:22:40.485 --> 00:22:47.645
Like, you actually end up getting a simpler query that Postgres, you know, is gonna still parse and prepare and execute.
00:22:48.285 --> 00:22:58.579
But you can get so it's actually not kind of
maybe kind of surprisingly, it doesn't necessarily mean that performance is gonna worsen overall.
00:22:58.579 --> 00:23:08.279
You do have more network latency between the two, it depends on if you can make each individual query… If each individual query is fast.
00:23:09.109 --> 00:23:15.879
found was the additional latency was not a blocker to proceeding with this plan.
00:23:18.349 --> 00:23:24.789
One of the interesting characteristics of this plan meant that, you know, we knew where we c- we-- what we couldn't do.
00:23:24.799 --> 00:23:30.909
We couldn't do SQL joins, we couldn't do foreign key constraints between two tables that are in the same database.
00:23:30.909 --> 00:23:38.079
We couldn't have common table expressions or sub, sub-queries that referenced tables that weren't in the same database.
00:23:38.549 --> 00:23:45.309
So essentially, the work kind of became… And I would say this was a bit before like, Claude-based programming.
00:23:45.309 --> 00:23:47.609
So this was more like 100% human-powered coding.
00:23:47.609 --> 00:23:56.759
But work basically became going through all the Active Record code and refactoring it to use these two, the two kind of select query approach.
00:23:57.809 --> 00:24:11.549
what we found was, which is a little bit of a segue, but fortunately Active Record supports something that at least reduces the overall work needed to be able to do this for the associations that are of a certain type.
00:24:11.549 --> 00:24:20.029
So for associations in a Rails app that are has many through associations Active Record added an option called disable joins.
00:24:20.399 --> 00:24:29.449
And what disable joins does is it changes the g- the generated SQL away from being a join to being two select queries
00:24:31.103 --> 00:24:34.653
So you're telling me someone else ran into this problem before and they had already solved it?
00:24:35.381 --> 00:24:37.991
Yeah, I think that work was… Yeah, I mean, more or less.
00:24:37.991 --> 00:24:59.871
Like, I think that work more or less was extracted from, Well, I know some work like that, some of the multi-databases support that Rails has, and some of the disable join stuff, to some extent was extracted from code at GitHub as they scaled their MySQL-based database stuff.
00:24:59.871 --> 00:24:59.921
I
00:25:00.039 --> 00:25:00.309
Yeah
00:25:00.411 --> 00:25:02.571
I've never been part of GitHub, but that's what I've read and
00:25:03.463 --> 00:25:04.133
That makes sense
00:25:04.831 --> 00:25:12.991
So yeah it's a framework capability, and what was really nice about that we didn't actually have that available to us based on an older Rails version we were running at the time.
00:25:12.991 --> 00:25:21.671
So we-- When we made that, back to that thing I mentioned before where we were at that fork where we needed to make a decision this would've been about a year ago now.
00:25:21.721 --> 00:25:32.491
We were-- We didn't yet have the new Rails version, and so I had started writing code that would that would like kinda conditionally branch and do the, queries.
00:25:32.521 --> 00:25:49.261
And what I found was, I kinda forgot how when you have an association on a Rail, on an Active Record model, you get all these like, you get like seven or eight different little generated methods that help with developer ergonomics, like as you check IDs and, you know, different little things.
00:25:49.381 --> 00:25:54.581
And what I found is I started to, needed to like re-implement each of those, and it was a big pain.
00:25:54.641 --> 00:26:00.261
I mean, obviously, we could try to do it in a general place and then include general code into models.
00:26:01.001 --> 00:26:10.001
And we were starting to go down that path, but there were like bugs to solve, and that was in addition to the unit tests.
00:26:10.181 --> 00:26:13.171
We were trying to preserve our unit tests around code as we refactored.
00:26:14.091 --> 00:26:21.711
so, at the same time, another engineer on the team was doing Rails upgrades and got us up to the version that had disabled joins, which I think was Rails 7.
00:26:22.571 --> 00:26:30.061
And I started experimenting with disabled joins, and it allowed us to like cut out a bunch of custom code that we were starting to do, which was a big
00:26:30.141 --> 00:26:30.401
Nice.
00:26:30.731 --> 00:26:34.911
And it, you know, it, had all those generate… It preserved all the regular Rails functionality and stuff.
00:26:35.861 --> 00:26:39.021
the disabled joins part was a big part of the solution.
00:26:39.101 --> 00:26:50.385
Maybe it was like I'd say somewhere between 25 and 50% of the solution because a lot of the code really just traversed associations and then did stuff you know, query code.
00:26:50.725 --> 00:27:03.475
And then maybe the other 50 to percent to 20, 75% was that was not a Rails association where we were accessing tables that would now no longer be in the same database.
00:27:04.115 --> 00:27:07.555
We'd drive refactorings there through unit tests.
00:27:07.875 --> 00:27:17.425
Which by the way, we did by… We had a, a Docker-based Postgres instance with eight databases locally, run unit tests, see a bunch of failures, start fixing stuff, like model by model.
00:27:17.919 --> 00:27:18.299
Yeah
00:27:18.565 --> 00:27:20.905
and that process played out over a set of weeks
00:27:22.399 --> 00:27:23.059
Interesting.
00:27:23.059 --> 00:27:33.609
So w-with this two, you know, query approach, right, does that mean like ordering and, you know, has to be done in Ruby in memory?
00:27:33.740 --> 00:27:36.369
And does that, you know, cause memory pressure?
00:27:37.059 --> 00:27:43.849
do you have to be like very cognizant of like what you are selecting on each statement, or is that a non-issue?
00:27:44.859 --> 00:27:45.979
No, it's definitely an issue.
00:27:46.059 --> 00:27:54.929
Yeah, we needed-- we, we generally need to be cognizant of any query that's not working with a fixed batch size,
like a max batch size.
00:27:55.649 --> 00:28:00.319
And those definitely show up in like our plus metrics.
00:28:00.319 --> 00:28:07.829
There'll be certain API that'll be slow and some… Hopefully we catch this in load testing before anything goes to production.
00:28:08.769 --> 00:28:12.209
we'll see slow API endpoints.
00:28:12.259 --> 00:28:13.819
We'll dig into the SQL queries.
00:28:13.819 --> 00:28:19.889
We'll find or, you know, non-batched queries, and then we'll add batches.
00:28:19.899 --> 00:28:22.849
And that's been a activity on repeat like many times.
00:28:23.759 --> 00:28:28.969
And but we also kinda know where our like hottest paths are for the API endpoint.
00:28:28.979 --> 00:28:33.240
A lot of what the frames do is they wanna show like a slideshow of photos.
00:28:34.269 --> 00:28:45.409
Any code that has to do with that, like there's a bunch of different things you can do with the app, with the mobile apps and the frames, but our hottest paths are usually serving the photos, you know, to the frames.
00:28:46.269 --> 00:28:59.729
anything in there we go through with a fine-tooth comb and make sure that it's like, you know, we pick a batch from the first table and then we pick a batch from the second table, do what we need to do, and then that's in a loop, that kind of thing.
00:28:59.750 --> 00:29:06.619
So you know, you could pay a little more in terms of latency because you're batching and you're looping.
00:29:07.279 --> 00:29:22.939
But if you have all sub-millisecond queries even like as fast as sub, you know, like we generally ha- any sort of query in that path is for sure sub-millisecond at the normal levels, and is index supported and that kind of thing.
00:29:23.809 --> 00:29:30.599
Then, you know, even with that stacked up latency, you can still have performance that's scalable.
00:29:31.429 --> 00:29:31.879
So yeah.
00:29:31.979 --> 00:29:37.829
And but yeah, you can't-- If you had a query that, for example, joined to two tables and then ordered, like you could no longer do that.
00:29:37.829 --> 00:29:44.829
So you have to then work with the data that you have from the database, and anything that's left over to do needs to be done in Ruby.
00:29:44.859 --> 00:29:48.189
So yeah you'd, build a, array and then sort it.
00:29:50.541 --> 00:29:50.851
Fun.
00:29:51.807 --> 00:29:52.067
Yeah.
00:29:53.027 --> 00:30:19.857
Yeah, so I mean, a lot of what the infrastructure team does at Aura Frames anyways, as far as coding activities, is not really feature development, but it's really making features And if, you know, since we kind of own the database performance and reliability if there's a code that has to do with the data access, that would be something we would work on typically, or we'd help the product developer make sure that the design is scalable.
00:30:20.847 --> 00:30:41.477
but yeah, I mean, what I've actually, like, has increased my appreciation for the, the, monolith, which is what we still run that serves most things, is, you know, for new features that are not in the really hot path and they have what I would call baby tables of, you know, less than 10 million rows, like go wild, do whatever.
00:30:41.477 --> 00:30:43.827
You know, th- like performance doesn't really matter that much.
00:30:43.827 --> 00:30:48.497
But if you start to touch the hot path or you get big data then we start to care more.
00:30:49.519 --> 00:30:50.619
Yeah, that makes sense
00:30:51.395 --> 00:31:03.435
I, I assume that's like part of your review process before they write that code if you know it's gonna be in the hot path, or does it… Do, do you actually wait until a pull request is open and you're like, "Oh, wait a second, we gotta do X, Y, and Z," or whatever?
00:31:04.321 --> 00:31:05.261
Yeah, it's a good question.
00:31:05.271 --> 00:31:20.061
The-- I've, I know I've experienced it different ways at different companies too, but I'd say that we, you know, with the monolith, we still have lots of developers committing daily and shipping daily, and the company leadership really wants, really prioritizes that.
00:31:20.341 --> 00:31:20.631
Mm-hmm
00:31:20.801 --> 00:31:26.061
And so I'd say that it's a, it's usually not like a design up front.
00:31:26.161 --> 00:31:40.211
It's usually like try to catch it at PR time, somebody on the team, of the product devs know some of the infrastructure devs to tag on a pull request or to ping in Slack and say,"Hey, can you take a look at this?" But we also try to not be too blocky.
00:31:40.221 --> 00:31:51.692
Like, if they ship it… i, you know, I think, you know, the reality is that, like, there are sometimes performance issues, but we can, you know, we have different tactics to roll back quickly if needed.
00:31:52.401 --> 00:32:12.101
do a lot of things like a lot of orgs do, like feature flags and, if a product developer has a concern about the production impact of a change and less production kind of awareness or experience, then they can certainly feature flag and really cut down their audience to a particular feature.
00:32:12.101 --> 00:32:14.711
It could be employees only to start and things like that.
00:32:15.471 --> 00:32:16.341
And that's pretty common.
00:32:16.341 --> 00:32:23.151
So we have, you know, probably yeah, a dizzying amount of feature flags and stuff like that all over the code base
00:32:24.990 --> 00:32:32.689
And we always end up talking about AI, and it, it just also made my brain think like, okay, how does the AI handle this sort of thing?
00:32:32.699 --> 00:32:36.000
Like, just the writing of the code and, like, you guys staying on top of that.
00:32:36.009 --> 00:32:44.059
I imagine you're writing a lot of code, or at least have a lot of PRs that are AI written, or at least have some semblance of that, so you got even that to, to deal with on top of it?
00:32:44.069 --> 00:32:46.809
Or that's just kinda like what you're dealing with in general, you're saying?
00:32:47.857 --> 00:32:56.917
Yeah,
it's been a really interesting year because it's kind of gone from like zero to basically like a year ago I was just occasionally prompting ChatGPT.
00:32:56.967 --> 00:32:58.457
I don't even know if I was using it at work.
00:32:58.567 --> 00:33:00.687
I mean, other than maybe kind of general research.
00:33:01.347 --> 00:33:03.727
I know that I would use it for like side projects.
00:33:03.757 --> 00:33:08.087
If I would use it for work, I'd be very careful to not anything specific.
00:33:08.097 --> 00:33:17.387
You know, it'd be like, "In Postgres, if you have a table you know, that's this many rows, and you write this kind of select query, like what are the performance?" You know, I don't know, like very vague.
00:33:17.605 --> 00:33:17.915
Yep
00:33:18.067 --> 00:33:22.157
And it went from like that to like Claude is everywhere in the company.
00:33:22.157 --> 00:33:25.667
You know, Slack and writing its own GitHub PRs.
00:33:25.667 --> 00:33:33.187
I've even had-- We even had a period of time where Claude was opening PRs and tagging me, and they're like, "Andrew, you changed this code six months ago.
00:33:33.187 --> 00:33:35.367
Like you're the owner of this, come and review it." You know, and
00:33:35.627 --> 00:33:35.937
Yep
00:33:36.007 --> 00:33:38.207
we've never let Claude ship anything, of course.
00:33:38.207 --> 00:33:48.607
Like, and there was, there was like a… You know, I think the company just deems it to be a very valuable tool was willing to take some risks but never like shipping unreviewed
00:33:49.635 --> 00:33:50.375
Yeah, right
00:33:50.907 --> 00:34:01.677
but like, yeah, I mean, basically trying to push the needle you know, I'd say on encouraging developers to, you know… The, the tool is provided to developers, so you know, hey, you can use this.
00:34:01.687 --> 00:34:05.177
You can, you know, the tool has access to our code base.
00:34:05.187 --> 00:34:08.967
The, the tool has access to our, database schema design.
00:34:10.467 --> 00:34:14.318
use it as much as it helps you get your job done.
00:34:14.427 --> 00:34:24.818
And so yeah, initially though it was, we've had a couple engineering-- We've had at least one engineering-wide meeting around like, you know, what some of these questions.
00:34:24.827 --> 00:34:36.268
Like, is it I think we consider it a tool and, you know, there's questions still about like how much priority to give to AI-generated PRs, we've, we definitely have been adjusting things.
00:34:36.278 --> 00:34:50.378
Like, I'd say currently it's more like adjusted back down from AI-generated PRs to being, mostly PRs that are opened maybe just at a higher volume from developers with more AI assistance.
00:34:51.137 --> 00:34:54.768
we're still trying to like have another human review those before they go out.
00:34:56.887 --> 00:35:08.088
I know I was listening to a previous Rails business podcast with Nate Berkopec, and he was talking about like the, how much-- what's the right level of understanding of the code.
00:35:08.394 --> 00:35:08.733
Yep.
00:35:08.748 --> 00:35:18.248
That's like a existential crisis I'm a-also going through right now personally you know, I always think that like anything to production like that I'm co-signing, I need to really understand deeply, right?
00:35:18.268 --> 00:35:19.648
Like, what was it before?
00:35:19.678 --> 00:35:21.157
What's the change that's being made?
00:35:21.907 --> 00:35:23.208
about failure modes.
00:35:23.887 --> 00:35:26.608
Have appropriate tests to show that it works correctly.
00:35:26.617 --> 00:35:29.367
It handles failure modes, like positive and negative tests.
00:35:30.238 --> 00:35:39.298
That's kind of the little talk I do run through with myself like at PR review, even if it's my own review reviewing of somebody else's code.
00:35:39.407 --> 00:35:47.157
But yeah, I mean, at the velocity where they're coming in and with the improving quality of the AI tools, I'm like, well, do I really… You know, what if I fudge that a little bit?
00:35:47.238 --> 00:35:59.258
Is that-- W-what if I like-- I, I have a pretty good understanding of this, but like what if it's just refactored in a week anyway from now and like the understanding I had like has already, is already stale and it's changed again?
00:35:59.268 --> 00:35:59.548
You know,
00:35:59.794 --> 00:36:00.284
Yep.
00:36:00.554 --> 00:36:01.003
Mm-hmm
00:36:01.228 --> 00:36:01.988
So I don't know.
00:36:01.988 --> 00:36:10.008
It's-- I-I'm still not trying to, you know, to be clear, I'm still not trying to-- I'm still trying to maintain the like, especially anything I write.
00:36:10.028 --> 00:36:13.028
But like with reviews of others' code, it's even more difficult.
00:36:13.057 --> 00:36:22.768
It's even more like to try to internalize the changes and after and provide a meaningful review, I find like, you know, it's always hard like without AI.
00:36:22.887 --> 00:36:35.867
So sometimes use AI to like help me it's still like a, a tricky question about I'm co-signing this, I'm kind of part of the stewardship group of this change going out, right?
00:36:36.153 --> 00:36:36.494
Yep
00:36:36.718 --> 00:36:39.867
You know, like, yeah, what's the level of understanding that is needed here?
00:36:40.648 --> 00:36:41.778
Yeah, that, that's a tricky one.
00:36:41.948 --> 00:36:44.678
Now Ry- Ryan, did you have any follow-ups on the sharding stuff
00:36:44.748 --> 00:36:54.488
i, I was cur-- that, so you, you talked about the Rails side of things, and you mentioned the other piece of it, which is how do you sort of migrate into that from a database perspective.
00:36:54.488 --> 00:36:56.508
I'd be interested to hear a little bit more about that too.
00:36:56.802 --> 00:36:57.132
Yeah.
00:36:57.142 --> 00:37:02.142
I found that to be interesting too, because, I mean, you've got billions of rows of data, which is terabytes of data to move.
00:37:02.162 --> 00:37:06.742
You can't just, like, snap your fingers and have the data
onto instances.
00:37:06.751 --> 00:37:21.902
So we, like, we looked at the really clean… The ideal to me, clean way to do it would be logical replication in because you can have row-level control and you can, I mean, the big thing for us, since we were moving an entire table, was we could at least just move one table.
00:37:21.972 --> 00:37:28.711
We could say, "Table A, we need all of your data on this other instance that we're gonna move to," right?
00:37:28.751 --> 00:37:32.271
Going from one to two instances in the simplest kind of scenario.
00:37:33.112 --> 00:37:42.382
tried various forms of logical replication, including native and then AWS has pglogical that has some capabilities.
00:37:42.882 --> 00:37:46.492
And we ran into different barriers.
00:37:46.552 --> 00:37:55.791
So what we ultimately ended up doing was creating physical replicas time, which is very reliable and works well in Postgre- in RDS on AWS.
00:37:56.512 --> 00:37:58.742
So that would replicate the entire database.
00:37:59.532 --> 00:38:11.202
So it was it was very wasteful right away because what we would do is we'd say,"We want table A on this new instance." So we're gonna just copy the entire database, which has 300 tables on it or whatever.
00:38:12.382 --> 00:38:14.302
All we actually wanna end up with is A.
00:38:15.182 --> 00:38:23.552
So we had to unfortunately with that strategy, it was very reliable, but the trade-off was we had to provision a whole bunch of additional space than what we needed.
00:38:24.271 --> 00:38:28.041
But it worked well, and we ended up basically repeating that seven times.
00:38:28.382 --> 00:38:30.972
So we'd say, a replica.
00:38:31.122 --> 00:38:36.172
I wanna use this table on this other instance." And then that way we could gradually verify all the data was there.
00:38:36.172 --> 00:38:40.682
We could gradually cut over reads and then gradu- and then eventually cut over the writes.
00:38:41.302 --> 00:38:49.582
So what it ended up becoming was replica promotion process and then a cleanup process.
00:38:50.332 --> 00:39:12.981
another interesting part of the cleanup was AWS launched blue/green deployments a few years ago, what that lets you do is then we could… Because we had to provision all the storage we needed up front, and even as we deleted tables we didn't need the new instance, Which was a, a method-- a very methodical process to not screw up and lots of reviews and that sort of thing.
00:39:12.981 --> 00:39:14.262
But ultimately, that's what we did.
00:39:14.262 --> 00:39:19.112
We had a sec-second instance with all the other tables we don't need, so we delete all those.
00:39:20.072 --> 00:39:27.692
We still had paid for way more storage than we needed, so we could provision a new instance with the baby storage, you know, much… just table A storage needs.
00:39:28.382 --> 00:39:35.222
then we could use blue-green deployment, more or less uses logical replication to keep it updated and then we could cut over.
00:39:35.922 --> 00:39:43.632
so we could then reduce our costs by cutting over to an appropriately or right-sized instance after, then we just repeated that seven times.
00:39:43.742 --> 00:39:51.122
So yeah, it was-- it, it-- like conceptually to me, if you've ever done a replica promotion to cut data over, it's not that complicated.
00:39:51.562 --> 00:39:58.271
It was more just very tedious because we did it a bunch of times, and it was very um, that we didn't make mistakes.
00:39:58.271 --> 00:40:02.771
So that just meant a lot of reviews, a lot of rehearsals, a lot of documentation.
00:40:03.461 --> 00:40:05.672
And with all of that, it was-- it's very doable.
00:40:08.302 --> 00:40:11.662
I might have missed it so in case you already answered it, I just want to ask quick.
00:40:12.231 --> 00:40:22.852
So you, at this, the beginning when, you know, 2024 I think it was, you said you guys had basically like hit the highest provisioned instance you could get.
00:40:23.231 --> 00:40:28.961
So I assume you're like down like s- smaller instances now with the, all this replication and sharding.
00:40:28.961 --> 00:40:29.432
Is that right?
00:40:30.637 --> 00:40:32.428
That's the goal.
00:40:32.718 --> 00:40:41.758
It wasn't the goal for last year because the goal actually was more just the expanded… With the expanded number of instances, we actually did vertically scale those up.
00:40:41.758 --> 00:40:44.948
So we were actually way over-provisioned for Christmas 2025.
00:40:45.807 --> 00:40:52.188
then what we were able to do fortunately through all of that stuff, Christmas 2025 was very successful.
00:40:52.188 --> 00:40:55.788
There were no database issues through Christmas, which the company was very happy about.
00:40:56.338 --> 00:41:12.178
And what then the challenge or, you know, not challenge exactly, but less of a challenge became was, okay, well, we can't af- we don't wanna run these very expensive, very large instances with expensive storage throughout the whole year when our needs are much less.
00:41:12.188 --> 00:41:15.518
So then what we could do is scale everything down gradually.
00:41:16.018 --> 00:41:24.498
And we use, you know, infrastructure as code, so not everything's like manual, but there's a decent amount of stuff as a small team that is manual clicking around in the AWS console.
00:41:26.088 --> 00:41:34.557
but gradually what we did was move the instances all to smaller instances, and in a lot of cases, we could even move to less costly storage.
00:41:34.657 --> 00:41:39.928
And we could certainly reduce all of our um, IOPS needs and that sort of thing.
00:41:39.938 --> 00:41:49.008
So, so now we kind of have this collection of heterogeneous eight Postgres instances that make up the workload.
00:41:49.738 --> 00:41:58.827
And what we're looking at for the future then, like Christmas '20, you know, we actually think this architecture we have now will serve us well for at least a couple years.
00:41:59.548 -->
00:42:06.398
the next limitation that we could run into is the needs of a single table exceeding what we could then do on one instance.
00:42:07.238 --> 00:42:26.728
And so we wanna then subdivide that table through row-level sharding, and then ultimately move to homogenous instances so that instances can be more of like the you know, how you treat application server instances, where you could have, the same sort of instance class and ideally less costly storage.
00:42:27.518 --> 00:42:34.518
But the complexity then is like now you're managing a bunch of Postgres instances that all need to be patched and monitored and everything like that.
00:42:34.849 --> 00:42:35.280
Sure
00:42:35.387 --> 00:42:41.318
so yeah, that, that work will be like, kind of the future database architecture that would then build even more.
00:42:41.788 --> 00:42:46.788
You know, it would build enough headroom essentially forever is how we would imagine it.
00:42:47.142 --> 00:42:47.532
Yeah.
00:42:48.481 --> 00:42:54.942
Yeah, I mean, it's really interesting to hear all this 'cause, you know, there's always been the, the question, does Rails scale?
00:42:54.981 --> 00:43:00.842
And other… Obviously, we already talked about GitHub and Shopify and, you know, you guys clearly are doing a ton of stuff too.
00:43:00.842 --> 00:43:03.612
So, you know, I think that's kind of a moot point.
00:43:03.622 --> 00:43:06.231
That's not really a, a, a question in my mind at all.
00:43:06.332 --> 00:43:08.021
But yeah, it's so complex, right?
00:43:08.052 --> 00:43:14.922
Like, it's just it's taken us 45 minutes to describe all this stuff, and I still have probably, like, 30 questions.
00:43:14.922 --> 00:43:30.922
I'm sure Ryan has 30 questions, so I, I… Maybe this would be a good time to maybe segue into something else, though, but it be- I don't wanna just completely cut off this, if there's anything else you wanted to add about how this went, any, you know, any other things that would be worth sharing before we do segue into other stuff
00:43:32.648 --> 00:43:34.452
Yeah, I think I I wrote these in the blog posts.
00:43:34.452 --> 00:43:37.562
I'm a little more articulate, I think, about some of the finer points hopefully.
00:43:37.612 --> 00:43:43.652
And I remember just off the top of my head, I remember a couple of things reflecting back on it, 'cause it was kind of an intensive experience as,
00:43:44.412 --> 00:43:45.092
Yeah, I would bet.
00:43:45.172 --> 00:43:47.021
we had this like we had the clock ticking, right?
00:43:47.021 --> 00:43:51.492
And we wanted to be in a place well in advance of the, of Christmas,
00:43:51.644 --> 00:43:51.983
Yep
00:43:51.992 --> 00:43:58.082
meant at least a month ahead of time where we were able to load test and say like, "We feel pretty good about this." That was the ideal state.
00:43:58.532 --> 00:44:02.021
And we kinda got there, like, but it was very close.
00:44:02.021 --> 00:44:03.662
You know, the, the timing was close.
00:44:04.152
--> 00:44:12.142
And you know, I think like what I really appreciated was with the engineers before me, they had built a very extensive unit test suite.
00:44:12.202 --> 00:44:20.802
I think that was critical, you know, because any code, any active record changes that we changed, we needed to have confidence that we weren't causing new regressions.
00:44:21.722 --> 00:44:38.231
And then I think the engineering leadership team was like very committed to this, so they were really, you know, supportive of like, you know, do whatever we can to get that increased reliability that we want through the holiday season and did a good job of conveying the importance financially to the of that period of time.
00:44:39.372 --> 00:44:52.021
you know, we were able to make a lot of decisions quickly and then leverage the experience of other engineers that have been there for a long time that really knew, this, this area is really important, this area is not important, could prioritize well and stuff.
00:44:52.021 --> 00:45:03.001
And that stuff's like, that stuff's like hard to come by, you know, sometimes like on the team if you've got like a team that is not a long-tenured team or if you don't have good support from leadership and stuff.
00:45:03.001 --> 00:45:09.942
So I really appreciated having all that stuff, and I think that was a key part of the success that's necessarily the Rails code or the Postgres databases
00:45:10.238 --> 00:45:10.597
Yeah.
00:45:10.728 --> 00:45:11.438
Th- that makes sense.
00:45:11.438 --> 00:45:14.298
That's great to hear that it, you, you had all that there to help out.
00:45:15.847 --> 00:45:20.157
Switching gears a little bit, you know, you wrote the book, you mentioned it a, a little bit already.
00:45:20.307 --> 00:45:21.108
It's been what?
00:45:21.137 --> 00:45:23.597
Two years, I guess, is what you said since you wrote it.
00:45:24.347 --> 00:45:26.057
And that, that was my introduction to you.
00:45:26.057 --> 00:45:33.738
I think that I kinda reached out shortly after maybe it was in like the beta phase and, you know, you and I chatted about some stuff and, and, you know, you worked with us for a little while too.
00:45:33.738 --> 00:45:38.738
And so I, I'd be curious to kinda know, like, how's that going?
00:45:38.798 --> 00:45:40.887
Are you, you know, what, what are you seeing with the book?
00:45:40.887 --> 00:45:43.148
Are you adding chapters?
00:45:43.148 --> 00:45:44.858
Are you just kinda letting it ride, you know?
00:45:45.198 --> 00:45:46.048
What- whatever
00:45:46.070 --> 00:45:46.490
again?
00:45:46.688 --> 00:45:46.878
Yeah.
00:45:46.878 --> 00:45:47.628
Would you do it again?
00:45:47.628 --> 00:45:48.228
Exactly.
00:45:50.300 --> 00:45:59.890
Yeah I think I think I would… You know, I feel like I I feel very fortunate to have found, like, somewhat of a niche in, as a software engineer because I was reflecting myself.
00:45:59.890 --> 00:46:06.349
Like, I finished college and entered the workforce as a software engineer, like, 20 years ago, this year.
00:46:06.360 --> 00:46:15.210
So I feel like for about 15 of those years I didn't have any sort of, like, particular focus or, you know, I mean, which is fine.
00:46:15.210 --> 00:46:21.829
I mean, I think early career, you know, you're just doing what you know, you're just learning the field and doing things and whatnot.
00:46:21.840 --> 00:46:25.480
And e-everything's different now with AI, but we'll just put that pin in that for a moment.
00:46:25.490 --> 00:46:38.220
But, like, anyway I think a number of years ago, like, I just found that I was really excited about this kind of work, like scaling Postgres because it is hard and there's lots of nuance, and it's interesting, like, how it interacts with the client application.
00:46:38.220 --> 00:46:40.590
And the database is so important to companies.
00:46:40.590 --> 00:46:47.019
The data that customers are creating is vital, and it's critical and it's that sort of thing.
00:46:47.019 --> 00:46:58.620
So I've been enamored with that kind of work and have… The, the book kinda became a way to all of I knew and share it, which I also have always enjoyed.
00:46:58.730 --> 00:47:02.800
I write a my, my own personal website's been going for, like, 20-plus years.
00:47:03.420 --> 00:47:06.160
Just a lot of… Eventually, it just turned into a tech blog.
00:47:06.180 --> 00:47:12.130
I used to write about, like, random stuff, but , I kinda made it more, you know, career-oriented a number of years ago.
00:47:12.849 --> 00:47:18.260
Uh, So yeah, I mean, I think, like, in terms of writing and learning Postgres and the interaction, I don't s- I don't see that going away from me.
00:47:18.269 --> 00:47:26.220
It's just like a, the, the flame is still burning as far as, like, it's an interesting area for me, I like to connect with others about it.
00:47:27.240 --> 00:47:37.420
I'll be, I'm uh, uh, excited to have been picked to speak at the, a couple of upcoming events we can discuss.
00:47:37.510 --> 00:47:40.579
And I continue to, you know, like, plan to keep doing that sort of thing.
00:47:40.650 --> 00:47:45.410
So yeah it's really nice, I guess, like in, in summary, to have a niche.
00:47:45.510 --> 00:48:01.079
But going back to the book thing, the, you know, writing a book is extremely hard, and I think, like, with, unfortunately, with the introduction of AI and stuff, it's it feels like the motivation, which was already a lot of it was intrinsic motivation.
00:48:01.769 --> 00:48:04.750
It seems like now it's just gotta on- almost only be intrinsic motivation.
00:48:04.750 --> 00:48:07.670
It's like, w- if I write a book, like, is anyone gonna buy it or read it?
00:48:07.710 --> 00:48:10.930
Like, is it, you know, how do I What's the ROI on that activity?
00:48:10.940 --> 00:48:12.280
It was already not great, but
00:48:13.173 --> 00:48:13.564
Right
00:48:13.620 --> 00:48:20.150
And that doesn't mean writing books is not great, but it's like, it's
somewhat for the author.
00:48:20.170 --> 00:48:29.110
It's somewhat for, like, hopefully, the author connects with the readers and feels like it's a way to, you know, serving a purpose or provide value to readers and that sort of thing.
00:48:29.110 --> 00:49:00.434
But if that sort of feedback loop of the readers just, I'm just imagining other developers being like, you know, prompting Claude, and I, I just think a lot fewer of them would say like, I'm gonna go buy a book and build some sort of foundational advice and spend three hours reading one chapter and practice these exercises and stuff." I mean, I'm sure there are people that will still do that, but, like, it just seems like it's changed so much that the, I don't see myself-- I don't know.
00:49:00.434 --> 00:49:19.244
I don't see myself writing like a, a new book, but, like, I do wanna continue to kinda cultivate this niche of Rails and Postgres, and it sort of bugs me that there's little things that keep happening that are new in Postgres that are making things I've written bit stale.
00:49:19.264 --> 00:49:22.534
And so, like, I gotta figure out how to do this.
00:49:22.554 --> 00:49:34.814
I don't know how yet because writing a book's so demanding i-in time, but I'd love to do, like, a either a revision or a slight title change of the book in the future to be practical.
00:49:34.814 --> 00:49:35.994
But yeah, that's kinda what I'm thinking.
00:49:36.945 --> 00:49:42.816
ironically, I feel like you know, reaching for a book and building that foundation is probably more important than ever, right?
00:49:42.885 --> 00:49:43.246
In order to
00:49:43.306 --> 00:49:43.556
maybe
00:49:43.556 --> 00:49:43.986
tell when AI
00:49:44.175 --> 00:49:44.606
and I don't
00:49:44.786 --> 00:49:45.806
BS'ing you or not.
00:49:45.956 --> 00:50:00.146
It'd be interesting, like, you have the GitHub repo that you sort of put all your snippets in, and I follow that and I get the notifications and I'm like, "All right, well," if I look at it, I can usually be like, "All right, this is probably where this came from."
00:50:01.044 --> 00:50:01.344
Yeah
00:50:01.986 --> 00:50:06.206
But it'd be in-- One just idea is to write a little summary with each of them.
00:50:06.256 --> 00:50:07.186
You do sometimes.
00:50:07.195 --> 00:50:09.135
Sometimes it's just a snippet, right?
00:50:09.135 --> 00:50:18.496
But it's like, know, we're all like-- Our attention spans are so short these days that it's like that's the bite-sized piece of how you can communicate these nuances.
00:50:19.326 --> 00:50:23.526
You know, so one idea for you of just sort of building on that and keep it going.
00:50:23.840 --> 00:50:24.400
Yeah.
00:50:24.559 --> 00:50:26.050
Yeah, and I do that's a good idea.
00:50:26.110 --> 00:50:35.940
I also-- I don't get a lot of response, but I like sharing out other even community or, I mean, things other folks have written too that are relevant to this space, like in a newsletter.
00:50:35.990 --> 00:50:36.650
So I have a,
00:50:36.890 --> 00:50:36.900
Mm-hmm.
00:50:37.760 --> 00:50:40.610
email newsletter that I occa- very occasionally update.
00:50:40.610 --> 00:50:41.660
It's not very frequent.
00:50:42.539 --> 00:50:52.460
And then I'm always look- there's other great sources too, Postgres Weekly and the Scaling Postgres podcast and Ruby Weekly and like, all of those things plus social media and whatnot.
00:50:53.150 --> 00:51:03.320
There's definitely a lot of interest in, you know, a lot of developers working on similar things that wanna learn new capabilities or how other companies solve things and that kind of thing.
00:51:03.320 --> 00:51:18.039
So, yeah, I mean, it's, it what I do have now is I have an appreciation for how demanding writing the book is, and it is something that would need to figure out how to do in a way that's a little bit less disruptive to my life.
00:51:18.039 --> 00:51:23.800
So just maybe like trickling things out would be a lot nicer than a big intensive period of time.
00:51:24.988 --> 00:51:25.097
It,
00:51:25.173 --> 00:51:25.374
it
00:51:25.758 --> 00:51:26.097
yeah.
00:51:26.188 --> 00:51:31.108
A- I mean, from my perspective too, like I still value your expertise and others' expertise.
00:51:31.108 --> 00:51:37.258
So if you've written the book, maybe you're not gonna make money on the book, but, you know, you do cons- some consulting.
00:51:37.968 --> 00:51:47.048
If that person's doing consulting, you know, we still go to Nate, we still go to you, you know, for even with AI 'cause, like, like we were already talking about, I don't trust everything the AI does.
00:51:47.068 --> 00:51:48.548
I think it still needs a person there.
00:51:49.238 --> 00:51:52.518
And I, I personally don't trust it on like big architecture.
00:51:52.518 --> 00:51:57.438
Like sure, if I'm greenfield, I'm gonna build something, I might just let it run, see how it goes.
00:51:57.458 --> 00:52:09.028
But if I've got a, you know, like, like Aura had, Aura Frames, like it- they had an architecture problem, and yes, it was pre-AI, but I think today they probably still would search out someone like you and with your expertise.
00:52:09.038 --> 00:52:10.738
So if nothing else, there's that.
00:52:10.778 --> 00:52:19.608
And you can also just ask your, your Claude, you know, "What would Andrew Atkinson do?" And get that far, 'cause you get all that, that data out there too, for what it's worth.
00:52:19.668 --> 00:52:19.907
But,
00:52:20.653 --> 00:52:27.923
Yeah, I think, yeah, I think companies still want to have, experts in different areas and owners, you know?
00:52:27.923 --> 00:52:32.604
And it's… thing- things aren't just changed once and released, like they have their own life cycle.
00:52:33.014 --> 00:52:43.673
And so I really like working on… Another thing I learned in my career journeys was I do like working on at companies like Aura where they're, they have a, a product market fit.
00:52:43.943 --> 00:52:56.753
You
know, they have something where it's like a growth stage of like here's what… Here's a, you know, a very successful component of our business that we're growing into new countries and reaching more potential consumers and stuff like that.
00:52:57.544 --> 00:53:16.443
so you have… Like with that comes maybe not as much excitement in terms of like new features where things are changing all the time on the platform, but you have this responsibility to maintain the you know, to really not break things and maintain and continually improve, you know, performance and reliability and that sort of thing of what's there.
00:53:17.193 --> 00:53:27.794
And you know, I think like that sort of thing makes sense to have people on staff to do, you know, so they can build their expertise and keep, you know, keep that context that they have.
00:53:27.794 --> 00:53:42.324
Like, you use the word context like how AI uses it, but they have the context of the and the company and the goals that they can then use in their day-to-day work to, you know, continue to improve the platform.
00:53:43.324 --> 00:53:46.494
So yeah I… and that doesn't seem like that's going away anytime soon.
00:53:46.604 --> 00:54:03.604
So the, I hope there still are technical books and also maybe people of a certain generation, like they're just used… I know that's what I often do, like is, you know, at least what I used to do is like, you know, I'd wanna explore a topic and I'd buy a book.
00:54:04.184 --> 00:54:11.963
And if I'm really seriously invested in a particular area, like I would still up a book for that foundational knowledge.
00:54:12.753 --> 00:54:16.134
I also know too, like books go through so much more review and scrutiny.
00:54:16.144 --> 00:54:22.233
You know, there's a lot of editors and people involved in the production of a published book, that definitely counts for a lot
00:54:23.762 --> 00:54:26.342
Yeah, we got at least another 30 years of book buying in us, right?
00:54:28.562 --> 00:54:36.231
Hopefully, yeah, for at least for the authors that you know, wanna at least have some, a little bit of ROI on their activities
00:54:38.312 --> 00:54:42.012
All right, so you said you were speaking at a couple places coming up.
00:54:42.012 --> 00:54:43.202
You wanna share a little bit about that?
00:54:43.814 --> 00:54:44.084
Yes.
00:54:44.094 --> 00:54:44.403
Thank you.
00:54:44.403 --> 00:54:44.664
Yeah.
00:54:44.744 --> 00:54:47.204
Rails World is doing a thing that I hadn't seen before.
00:54:47.244 --> 00:54:58.784
I had the chance to go to Rails World a couple years ago as an attendee pitched Rails World, the main conference, on a, on basically talking about this work we did with Rails at Aura Frames.
00:55:00.443 --> 00:55:07.024
they thought it would be a better fit for this, the Rails at Scale Summit that they're doing, which doesn't have a lot of information publicly about it.
00:55:07.943 --> 00:55:09.134
It's a one-day summit.
00:55:09.173 --> 00:55:16.314
What I know is it's a one-day summit bringing together engineers from a few companies that are using Rails at a big scale.
00:55:16.884 --> 00:55:19.274
And we're gonna do short presentations.
00:55:20.054 --> 00:55:20.724
about all I know.
00:55:20.774 --> 00:55:27.354
So I'll be able to tell you more, but that's gonna be the day before the main Rails World event in Austin.
00:55:27.354 --> 00:55:30.314
So I'm gonna be headed to Austin for a short period.
00:55:31.114 --> 00:55:41.764
then Postgres has had a New York-based conference for a lot of years, and I've, been an attendee in the past, and it's been called PG Conf NYC.
00:55:41.764 --> 00:55:54.153
I believe it's been rebranded or… I'm not exactly sure on the details, but New York-based conference is called Postgres Summit, and it's in New York at the end of September, beginning of October.
00:55:54.164 --> 00:55:55.044
It's a three-day event.
00:55:55.684 --> 00:56:04.784
And I'll be presenting on the Postgres side with more Postgres detail there on our Aura Frames story.
00:56:04.814 --> 00:56:07.474
More of a typical, you know, 30-minute presentation.
00:56:08.124 --> 00:56:10.084
But also I, I really like attending that.
00:56:11.124 --> 00:56:19.894
I think it's, it's maybe-- Well, for me, it's been the best event in terms of the number of attendees and the technical depth of the topics.
00:56:19.923 --> 00:56:23.914
Includes a lot of Postgres core contributors that attend and present there.
00:56:24.304 --> 00:56:40.284
So they're always talking about… at this one, I expect to see a lot about what's coming in the next version of Postgres, which is 19, and then even, like, m-maybe groundbreaking stuff on, like, Postgres 20 stuff, which would be like 2027 things that they're working on, they present on.
00:56:40.284 --> 00:56:49.764
So that one I find super valuable, and it's sometimes it doesn't sorta, like, pay off in terms of real-world use for, like, two years after I learn something.
00:56:49.764 --> 00:56:55.494
But I've been able to learn things by attending that one that I put into use, which is cool.
00:56:57.334 --> 00:57:00.784
But yeah, those will be-- So Austin and New York in the, the next month.
00:57:01.570 --> 00:57:03.680
So you're, you're busy at the end of September it sounds like.
00:57:03.885 --> 00:57:04.346
Yeah
00:57:06.079 --> 00:57:06.390
Cool
00:57:06.407 --> 00:57:12.478
yeah, Au- Aura Frames has also been supportive of me um, writing on my blog about , this work that we did.
00:57:12.738 --> 00:57:16.278
We don't currently have at Aura Frames a, like a engineering blog.
00:57:16.968 --> 00:57:27.438
And so I said, "Hey, I write about this kind of stuff." And written about it there, and then they were supportive of, sending me to these conferences to present and so I appreciate that.
00:57:27.478 --> 00:57:38.068
And the… For what it's worth, the Aura Frames team is hiring, if anyone's listening to this that's interested in this kind of work feel free to get in touch with me, and can give you some more info
00:57:39.534 --> 00:57:39.943
Sounds good.
00:57:39.943 --> 00:57:40.974
Where can they reach you for that?
00:57:42.112 --> 00:57:51.392
If you wanna go through the official channel, you can, you can uh, email andrew@auraframes.com or you can just contact me through my website, andyatkinson.com.
00:57:51.472 --> 00:57:53.802
I've got a contact form there that just goes into my email
00:57:55.046 --> 00:57:55.186
Okay.
00:57:56.175 --> 00:57:58.695
A- anywhere else you wanna point people to before we jump off?
00:57:59.510 --> 00:58:07.530
I'm somewhat active on LinkedIn and Bluesky these days for social media, but otherwise yeah, that's it.
00:58:07.550 --> 00:58:10.910
Pretty much my You'll also find more info on the book.
00:58:11.640 --> 00:58:15.470
book's High Performance Postgres for Rails, for anyone that has not heard of it.
00:58:16.440 --> 00:58:27.370
You can find some-- I've got a book page on my site that has links to where you can buy it and some reviews and that sort of thing if you're curious about it and wanna see if it would be a good fit for you
00:58:28.751 --> 00:58:29.072
awesome.
00:58:29.612 --> 00:58:30.501
A- and for what it's worth,
00:58:30.746 --> 00:58:30.806
the
00:58:31.092 --> 00:58:32.902
yep, I have read the book, yep, as well.
00:58:33.032 --> 00:58:33.762
That's what I was gonna say.
00:58:34.442 --> 00:58:34.782
So
00:58:34.920 --> 00:58:35.050
Yeah
00:58:35.162 --> 00:58:37.442
definitely recommend it, so cool
00:58:37.498 --> 00:58:38.778
Well, thanks so much for joining us.
00:58:38.778 --> 00:58:40.498
We appreciate you coming on
00:58:40.867 --> 00:58:41.758
Yeah, thanks for having me.
00:58:42.028 --> 00:58:42.548
Yeah, thanks.
00:58:42.867 --> 00:58:43.188
Yep