สำเนาบทสนทนา 🔗
WEBVTT
00:00:42.533 --> 00:00:48.379
Welcome to the RubyGems podcast brought to you by Ruby Central, where we uncover the stories behind the code.
00:00:48.379 --> 00:00:49.359
We're your hosts.
00:00:49.359 --> 00:00:50.040
I'm Marty Haught.
00:00:50.060 --> 00:00:50.060
And I'm David Hill.
00:00:51.041 --> 00:00:52.662
auto-scaling platform for Ruby applications.
00:00:52.662 --> 00:01:11.710
Adam's been writing Ruby since 2007 and turned what started as an internal scaling tool into Rails AutoScale, then grew it into a full-time business reaching $300,000 in annual recurring revenue.
00:01:11.710 --> 00:01:29.139
As a solo bootstrapped founder, he's created an impressive suite of Ruby gems that integrate with popular job queue systems like Sidekick, delayedjob and GoodJob, helping thousands of Ruby developers automatically optimize their app performance across platforms like Heroku, render and Flyio.
00:01:29.139 --> 00:01:39.920
Join us as we explore Adam's Ruby journey, the technical challenges of building platform-agnostic gems, and how he transformed a simple internal tool into a thriving SaaS business.
00:01:39.920 --> 00:01:41.635
Welcome to the podcast, adam.
00:01:41.635 --> 00:01:43.316
Hey guys, thanks for having me.
00:01:44.189 --> 00:01:49.403
So I was noticing, Adam, that you were at Edgecase.
00:01:49.403 --> 00:01:53.296
Maybe was that your sort of start into the Ruby world.
00:01:54.110 --> 00:01:56.093
Yeah, that was my first Rails job.
00:01:56.093 --> 00:01:58.318
That was back in 2007.
00:01:58.318 --> 00:02:03.179
Edgecase was like one of the sort of the original, like Rails consultancies.
00:02:03.179 --> 00:02:11.515
Back in the day there weren't a whole lot of consultancies just focused on Rails and, fortunately for me, there happened to be one in my town.
00:02:11.515 --> 00:02:14.574
I live in Columbus, ohio, and that's where Edgecase was.
00:02:14.574 --> 00:02:16.420
That was where it all started for me.
00:02:17.350 --> 00:02:19.336
Any interesting tales from that time?
00:02:19.336 --> 00:02:29.893
I mean, I remember Joe O'Brien was the primary one with Edgecase and also Jim Weirich worked there, and so I have great respect for Jim amazing work that he's done.
00:02:29.893 --> 00:02:33.162
So I just I'm curious if there's any stories you have to share.
00:02:34.050 --> 00:02:36.816
Jim was just an amazing developer, a wonderful human.
00:02:36.816 --> 00:02:38.360
We miss him a ton.
00:02:38.360 --> 00:02:41.937
He actually worked in the Cincinnati office of Edgecase.
00:02:41.937 --> 00:02:50.239
I didn't get to work with him on any projects but I definitely got to hang out with him at a few conferences and things like that and just a pure joy to be around.
00:02:50.239 --> 00:02:54.030
Loved being around Jim, but the whole Edgecase team was awesome.
00:02:54.030 --> 00:03:01.064
I would say from that time what I remember is just learning continuously.
00:03:01.064 --> 00:03:11.296
I think if somebody who is just starting out has the opportunity to join an agency or consultancy, especially if it's in person, I would highly recommend that.
00:03:11.296 --> 00:03:17.138
I think that's the best way to get good, get experienced really quickly.
00:03:17.138 --> 00:03:19.617
So I'm really happy that I did that.
00:03:19.617 --> 00:03:38.344
I was at Edgecase for six years before I made the jump over to product but, yeah, definitely really thankful for my years doing that so, in that transition space, I guess what led you to doing what became judo scale?
00:03:38.866 --> 00:03:45.483
what were you doing, what were you working on at the time where that started to become enough of an issue that you felt like there was a problem to be solved?
00:03:46.210 --> 00:03:49.540
Yeah, I was working for a small product company called Agent Pronto.
00:03:49.540 --> 00:04:08.560
Basically, we were doing like lead generation for real estate agents and we were running everything on Heroku and just really wasting a lot of money running at full scale during off hours and when we were low traffic and as a small team, we were trying to save money where we could.
00:04:08.560 --> 00:04:12.581
So it seemed clear that we needed to auto scale in some way.
00:04:12.581 --> 00:04:16.279
Heroka makes it easy to slide that dyno slider up and down.
00:04:16.279 --> 00:04:19.269
Obviously, we didn't want to do that manually every day.
00:04:19.269 --> 00:04:21.076
We wanted to automate that in some way.
00:04:21.076 --> 00:04:26.656
So we tried a couple of the autocalers that were available at the time this was 2016.
00:04:26.656 --> 00:04:28.615
And they just didn't work very well.
00:04:28.615 --> 00:04:33.802
They were really clunky, hard to use and did not work reliably.
00:04:33.802 --> 00:04:35.495
Expectations were not met.
00:04:36.170 --> 00:04:39.737
I had that developer urge where it's like I can build a better version of this.
00:04:39.737 --> 00:04:49.122
So I had been for years seeking out side projects that may eventually someday become a product.
00:04:49.122 --> 00:04:59.795
I had the dream of running my own SaaS full-time and at the time it just kind of seemed like a dream, but I was eager to find something where I could maybe get started on that.
00:04:59.795 --> 00:05:02.759
So I thought you know, I'll try to build this autoscaler thing.
00:05:02.759 --> 00:05:07.043
The first version of it I built in a few weeks.
00:05:07.584 --> 00:05:17.055
It had no front end, it was really specifically built for us Any configuration had to be done through a Rails console but proved out that the idea worked.
00:05:17.055 --> 00:05:26.603
And you know, before I started working on it, I asked my boss my boss, we were like a team of three, but I asked him if I build this thing in my spare time, would you be interested in using it and would I own the IP for it?
00:05:26.603 --> 00:05:45.213
And he said yeah, absolutely, marketplace, so that other people could install it.
Load full transcript
00:05:45.213 --> 00:05:47.059
And from there I had this side business that I was running and it didn't explode.
00:05:47.059 --> 00:05:56.110
It was really slow growth, but I was able to just keep it going and year over year just kept growing more and more to the point where I was able to go full time on it in 2021.
00:05:56.912 --> 00:05:59.134
So the time that you were there at Agent Pronto?
00:05:59.134 --> 00:05:59.814
Was that what it was called?
00:05:59.814 --> 00:06:00.475
That was what it was called.
00:06:00.475 --> 00:06:01.636
Yeah, make sure I'm not making that up in my head.
00:06:01.636 --> 00:06:02.857
Was that also Rails work that you were there at Agent Pronto?
00:06:02.857 --> 00:06:03.437
Was that what it was called?
00:06:03.437 --> 00:06:03.997
That was what it was called.
00:06:03.997 --> 00:06:04.978
Yeah, make sure I'm not making that up in my head.
00:06:04.978 --> 00:06:06.280
Was that also Rails work that you were doing there?
00:06:06.961 --> 00:06:19.114
Yeah, it was Rails work and when I had joined there everything was hosted on Engine Yard and so I had spearheaded the move to Heroku, which went great and we actually saved a lot of money on that move and it simplified a lot of things for us.
00:06:19.114 --> 00:06:25.704
But there was still that extra opportunity to substantially reduce our spend on our Heroku resources.
00:06:26.689 --> 00:06:37.875
So when you started building what became JudoScale, was there any kind of debate about maybe using a different technology stack than Ruby and Rails, or was it always going to be that?
00:06:37.875 --> 00:06:40.201
What made you continue to pick Rails for that?
00:06:40.790 --> 00:06:44.682
I've been primarily a Rails dev since 2007.
00:06:44.682 --> 00:06:46.447
Since that moment I joined Edgecase.
00:06:46.447 --> 00:06:57.343
I had been falling in love with Rails since before I joined Edgecase and just kind of tinkering with it on my own, and once I had the opportunity to start doing it full time, I have not had the urge to use anything else.
00:06:57.343 --> 00:07:04.432
There are those who thrive on being a polyglot and learning all the different languages they can learn.
00:07:04.432 --> 00:07:05.956
I'm not one of those people.
00:07:05.956 --> 00:07:07.680
I have found something that I love.
00:07:07.680 --> 00:07:12.447
It works wonderfully and I'm going to keep using it until it doesn't serve my needs.
00:07:12.447 --> 00:07:13.971
So no, there was never any question.
00:07:13.971 --> 00:07:17.900
For me, rails was what I wanted to use and it has worked wonderfully.
00:07:18.822 --> 00:07:19.083
Great.
00:07:19.083 --> 00:07:32.898
I'm curious if you can share a little bit of your thought process behind how you modularized the application, your approach to the gems and what you may have learned doing that over the years.
00:07:34.251 --> 00:07:34.352
Yeah.
00:07:34.352 --> 00:07:38.573
So talking about the gems, I should probably back up a little bit and talk about how JudoSkill works.
00:07:38.573 --> 00:07:45.800
We have a gem that our customers install into their application and that gem just reports metrics to our API.
00:07:45.800 --> 00:08:00.571
So it reports request metrics specifically request queue time, which we can get into, what that means, and job queue metrics, things like sidekick and rescue, good job, all that stuff and then on the app side we consume those metrics and use it for auto-scaling.
00:08:01.612 --> 00:08:07.304
But for the gems, what's interesting is the very first version of the gem that I wrote.
00:08:07.304 --> 00:08:21.739
It was extremely simple because all I cared about was collecting requests, queue time from requests, and periodically sending that data to JudoScale, which was called Rails Autoscale at the time, but it's the same product.
00:08:21.739 --> 00:08:28.653
But over time we added support for autoscaling, worker dynos, and so now we're collecting job queue metrics.
00:08:28.653 --> 00:08:34.995
Initially that was just Sidekick, but then customers wanted delayed job, customers wanted rescue, customers wanted good job.
00:08:34.995 --> 00:08:45.994
I was just adding these things to the gem and one of the complexities that came about was it became really difficult to specify dependencies.
00:08:45.994 --> 00:08:51.352
We require this version of sidekick, but you might not be using sidekick at all If you're using delayed job.
00:08:51.352 --> 00:08:53.037
We require this version of delayed job.
00:08:53.958 --> 00:09:02.381
So to be able to enforce those version dependencies, we kind of ripped the whole thing apart into separate gems.
00:09:02.381 --> 00:09:06.336
So now we have kind of a core gem that we call Judo Scale Ruby.
00:09:06.336 --> 00:09:11.879
Most people don't use that on its own, they'll use Judo Scale Rails, judo Scale Sidekick.
00:09:11.879 --> 00:09:17.618
So we have a separate gem for each, basically like framework or job backend, that we integrate with.
00:09:18.500 --> 00:09:32.512
I was really conflicted on whether to go that route, because it's not quite as simple for our customers, because we can't just say, install this one gem, now we have to say install these two gems, and we have to ask them you know, which job backend do you use, before we tell them which gems to install.
00:09:32.512 --> 00:09:42.579
But it has been really nice having each gem be completely isolated and we're able to enforce those version dependencies.
00:09:42.579 --> 00:09:48.101
So it's clear which version of Sidekick we support and which version of rescue and delay job and so on.
00:09:48.101 --> 00:10:05.884
So yeah, that has worked really well and it means that when we need to support a new job backend, it just means building out a new gem, a new gem that's going to look a lot like our other gems, but yeah, it means that we don't have to update the main gem just to add a new job, backend.
00:10:07.070 --> 00:10:12.302
So as far as Judo scale the company
goes, the company is fully bootstrapped right.
00:10:12.302 --> 00:10:14.899
There's no VC funding or anything like that involved.
00:10:14.899 --> 00:10:16.533
No, there's an asterisk there.
00:10:16.533 --> 00:10:18.698
Asterisk, okay, there is, there is.
00:10:18.698 --> 00:10:19.542
That's a fun story.
00:10:21.431 --> 00:10:26.383
In 2021, we did take some funding from TinySeed.
00:10:26.383 --> 00:10:33.777
Tinyseed calls themselves like VC for bootstrappers or an accelerator program for bootstrappers, so it's a small bit of funding.
00:10:33.777 --> 00:10:39.459
It's $120,000, which was substantial to me at the time, but compared to VC it's a small amount of funding.
00:10:39.459 --> 00:10:42.639
But it's a remote accelerator program.
00:10:42.639 --> 00:10:46.518
So it got me into this community with other people who are doing the same thing.
00:10:46.518 --> 00:10:55.937
They have mentors who you can reach out to for help and certainly that little influx of cash was able to help me go full-time without feeling I was taking a huge risk.
00:10:55.937 --> 00:11:01.317
So I still own the vast majority of the company and make all of the decisions.
00:11:01.317 --> 00:11:06.119
I don't have to run decisions by them or anything, so they're an extremely hands-off owner.
00:11:06.119 --> 00:11:16.340
So I refer to JudoScal as being a mostly bootstrapped company, because we behave like a bootstrapped company, we act like a bootstrapped company, but technically not fully bootstrapped.
00:11:17.012 --> 00:11:32.686
A question about the bootstrap side of it, then, at least as someone who has never started a company, who still fantasizes about it periodically, because there is kind of like that idealized goal of not answering to another boss who is not like yourself, right?
00:11:32.686 --> 00:11:41.759
What has that experience been like, of transitioning from being an employee to running your own business and bootstrapping it, mostly to running your own business and bootstrapping it?
00:11:41.779 --> 00:11:46.926
mostly, it's really easy to forget that I am living my dream.
00:11:46.926 --> 00:11:56.998
I have to continually remind myself that, because I really am living my dream and when I step back it's amazing and wonderful and it's all the things that I had wanted it to be.
00:11:56.998 --> 00:12:01.557
At the same time, every day, I'm the one who has to make all the decisions.
00:12:01.557 --> 00:12:05.294
There's nobody else who I can deflect to.
00:12:05.294 --> 00:12:13.851
There's nobody else who I can lean on to take any sort of accountability for anything like it all comes to me and that's a lot of pressure sometimes and it's a lot of stress sometimes.
00:12:13.932 --> 00:12:23.591
We had our worst production outage ever a couple weeks ago and it was just a horrible feeling, knowing that like I had let our customers down in this way.
00:12:23.591 --> 00:12:32.334
And it would be great if I could somehow separate myself from the business and I try to do that to some extent.
00:12:32.334 --> 00:12:38.740
But at the end of the day, when JudoSkill lets its customers down, I feel like I've personally let them down and that can be tough.
00:12:38.740 --> 00:12:42.435
But the reality is it's great being my own boss.
00:12:42.435 --> 00:12:44.898
It's great getting to decide what I work on every day.
00:12:44.898 --> 00:12:51.817
It's great being able to make technical decisions without really needing permission from anybody.
00:12:51.817 --> 00:12:55.793
I do discuss, collaborate with Carlos on my team for all that stuff.
00:12:55.793 --> 00:12:59.721
But it's been wonderful and stressful and all the things.
00:13:00.549 --> 00:13:16.722
I saw that you, while building Geoudo Scale, were working at various product companies, including YNAB, which I'm curious about, but can you tell us maybe more about your motivations and your thoughts about as you're working at those companies and you were bootstrapping Judo Scale?
00:13:16.722 --> 00:13:18.878
How did that go and what were you thinking about?
00:13:19.870 --> 00:13:31.865
Yeah, I will say, one of the things that really helped me to run JudoSkill as a side project for over four years was being in the Heroku marketplace.
00:13:31.865 --> 00:13:39.230
I really sort of leaned on that distribution channel and basically I ignored marketing for those years.
00:13:39.230 --> 00:13:51.639
I built this product and I just waited on people to find the product through the Heroku add-on system and if it weren't for that, I would not be here talking to you right now.
00:13:51.639 --> 00:13:53.475
It would not have grown.
00:13:53.475 --> 00:13:58.615
We would all love to be able to just build a good product and people would just find it, but that's usually not the case.
00:13:58.615 --> 00:14:04.115
If you can find a good marketplace to go into, that can work over a long enough timeline.
00:14:04.115 --> 00:14:05.762
But it was slow.
00:14:05.762 --> 00:14:06.685
I needed to be patient.
00:14:07.508 --> 00:14:12.700
But the reality is that for those four years I was working nights and weekends on the product.
00:14:12.700 --> 00:14:23.404
I was never really in a mode of up until 2 am working on it and I was able to keep the pace very maintainable.
00:14:23.404 --> 00:14:27.533
It was a few hours and a couple evenings throughout the week.
00:14:27.533 --> 00:14:29.277
It was a few hours on the weekend.
00:14:29.277 --> 00:14:34.995
It was helpful that I kept the product very focused and very simple.
00:14:34.995 --> 00:14:38.971
It solved one problem At the time it had two screens.
00:14:38.971 --> 00:14:43.282
It was just a very simple product that did one thing and did it well.
00:14:43.282 --> 00:14:49.393
And I iterated on it over time, adding the worker, auto-scaling, adding different platforms eventually, and that kind of thing.
00:14:49.393 --> 00:15:04.912
But keeping the product simple, being in a marketplace and really just forcing myself
to keep the pace manageable, I think, is what helped me keep going for all those years without burning out, did you?
00:15:04.991 --> 00:15:15.711
find that you were observing and picking up things from those product companies that you then would apply to JudoScale, or was it more like?
00:15:15.711 --> 00:15:18.524
No, that's just a job where I was doing engineering.
00:15:18.524 --> 00:15:25.831
I didn't really think of it as how are they approaching product, how are they succeeding or not succeeding, making good choices?
00:15:26.980 --> 00:15:37.023
I probably brought some things over from Agent Pronto because we were such a small team, but when I went to YNAB, ynab was a substantially larger team.
00:15:37.023 --> 00:15:40.755
Our engineering team was maybe 25 when I was there.
00:15:40.755 --> 00:15:51.623
I think they've grown substantially since then and a team of one is going to operate so differently than a team of 20, way differently than a team of 100.
00:15:51.623 --> 00:15:54.249
So were there things that I learned on the way?
00:15:54.249 --> 00:16:03.808
I'm sure that I did, but I can't really point to anything because I just feel like operating solo for so long, it's just a whole different world.
00:16:03.808 --> 00:16:08.206
I was responsible for everything, whereas at my product companies I was responsible for coding.
00:16:08.206 --> 00:16:17.728
At JudoScale I'm responsible for design, I'm responsible for dev, I'm responsible for determining product direction, marketing support, all the things you know.
00:16:18.700 --> 00:16:23.326
So with TinySeed, I'm curious because I'm not familiar with their program.
00:16:23.326 --> 00:16:25.466
How did that help you?
00:16:25.466 --> 00:16:41.187
I mean, I imagine it helped a lot, but I'm curious if you can kind of go into details of how it helped you as a business owner and then like any sort of advice you'd give to others in a similar spot where they're bootstrapping and maybe considering something like that.
00:16:41.961 --> 00:16:43.403
It helped me in a couple of ways.
00:16:43.403 --> 00:16:44.226
The money was helpful.
00:16:44.226 --> 00:16:45.389
The money is what?
00:16:45.389 --> 00:16:50.691
Sure, I'm a risk-averse person and my wife even more so than I am.
00:16:50.691 --> 00:17:08.451
So making the jump from full-time senior software developer salary to running my own business, being responsible for everything myself, needing to find health insurance on the independent market all these things was a super scary jump.
00:17:08.451 --> 00:17:17.613
And Rails Autoscale at the time was at a point where I could just about match my salary with the product.
00:17:17.613 --> 00:17:23.192
But having a little extra money in the bank gave confidence that like, okay, we can do this without a whole lot of risk.
00:17:23.192 --> 00:17:25.587
So the money made a difference.
00:17:26.419 --> 00:17:33.592
I didn't really lean on the mentors in TinySeed the way most other folks I know in TinySeed do.
00:17:33.592 --> 00:17:46.780
For me, it was almost just more about feeling like I was part of a community who was doing similar things, People who I could bounce ideas off of and people who had my back, people who could empathize with the stuff that I was going through.
00:17:46.780 --> 00:17:50.429
Because being a solo founder can be really lonely.
00:17:50.429 --> 00:17:51.913
It can feel really isolating.
00:17:51.913 --> 00:17:57.325
I envy other folks with co-founders, but of course it's a mixed bag.
00:17:57.325 --> 00:18:00.326
The great thing about being a solo founder is I don't have to answer to anybody.
00:18:00.326 --> 00:18:05.405
The bad part is you don't have anybody else to talk to Exactly yeah.
00:18:06.200 --> 00:18:09.871
So, yeah, I would say advice find a community.
00:18:09.871 --> 00:18:20.723
You're going to need people who are doing similar things to bounce ideas off of, even if you don't find something like TinySeed.
00:18:20.723 --> 00:18:22.690
Tinyseed is tightly related to the MicroConf community and the MicroConf conferences.
00:18:22.690 --> 00:18:27.806
So, yeah, definitely be a part of that MicroConf community and find some people doing similar things in there.
00:18:27.806 --> 00:18:38.357
Get a co-founder if it's a natural fit, but don't force it, because if it's not somebody that you really work well with, it can end really badly, and I think I already touched on this.
00:18:38.679 --> 00:18:49.971
The thing that really helped was launching into a marketplace so that I didn't have to really focus on marketing from the get-go, and just the fact that I was able to run it as a side project for several years.
00:18:49.971 --> 00:18:55.087
If I went into it saying I'm going to go full-time within a year, well, that would have been incredibly stressful.
00:18:55.087 --> 00:18:59.315
The reality is, saas businesses take time to grow.
00:18:59.315 --> 00:19:04.150
We read stories about the ones that hit hockey stick growth and overnight success.
00:19:04.150 --> 00:19:05.740
Most of them, that's not the case.
00:19:05.740 --> 00:19:11.232
Most of them take a long time to grow, and so you've got to be patient.
00:19:11.232 --> 00:19:13.262
You got to be in it for the long haul.
00:19:14.223 --> 00:19:34.634
So, speaking of your marketing efforts, I have to ask about your ads for lack of a better word that you did for Drew during the RubyConf last year and then for your podcast that you've been sponsoring the process of coming up with those ads, listening to them, I'm always entertained, I'm always laughing.
00:19:34.634 --> 00:19:41.672
There's got to be a couple of good stories of just how and why and where you started deciding to do those.
00:19:42.500 --> 00:19:46.151
I just really don't like to do the thing that's expected.
00:19:46.151 --> 00:19:57.250
So when I knew I needed to start taking marketing more seriously and doing some ads and that kind of thing, I'm like I don't want to make ads that
sound like other ads, I want to stand out in some way.
00:19:57.250 --> 00:20:06.942
A friend of mine, john, who I worked with at a previous job and he lives in town with me.
00:20:06.942 --> 00:20:14.784
He and I get together once a week at a coworking space here and he kind of acts as almost like a co-founder, advisor, therapist, brainstorm partner.
00:20:14.784 --> 00:20:23.947
So we get together and we just like we don't really do work on code at all, it's all about just brainstorming and talking through ideas for the company, ideas for marketing.
00:20:23.947 --> 00:20:26.573
And he always pushes me out of my comfort zone.
00:20:26.573 --> 00:20:28.907
We'll have a crazy idea and I'm like, yeah, but I'm not going to do that.
00:20:28.907 --> 00:20:30.021
And he's like, oh, we're going to do that.
00:20:31.024 --> 00:20:50.045
So when I think back to the commercial that we made for RubyConf last year, I had reached out to Drew about sponsoring his Ruby game show that he's done at several conferences and he told me if we wanted to, we could make a 30 second commercial and he would show it from stage.
00:20:50.045 --> 00:20:51.950
The only requirement is that it had to be funny.
00:20:51.950 --> 00:20:56.442
I was like, oh, that's way too much pressure, I'm not going to do that.
00:20:56.442 --> 00:20:57.926
And john was like, oh, yeah, we are.
00:20:59.029 --> 00:21:13.311
So I don't even remember how the idea came about, but eventually we landed on doing a spoof of a billy mays style infomercial for the judo scale, which is an actual scale.
00:21:13.311 --> 00:21:15.355
We recorded it in his kitchen.
00:21:15.355 --> 00:21:20.067
I had a ridiculous amount of fun editing it and yeah, it was a big hit.
00:21:20.067 --> 00:21:31.362
People got a good laugh out of it and that felt really good that it went over well, because the kind of thing so far out of my comfort zone to just be like making a fool out of myself like that, I'm just like cringing when it's being shown from the stage.
00:21:31.362 --> 00:21:35.549
But people have said a lot of nice things about it, so I don't think we went wrong there.
00:21:36.391 --> 00:21:38.034
I remember laughing quite a bit.
00:21:38.034 --> 00:21:47.010
And then I was listening to Drew's podcast this morning and when your ad comes on, did you know that his bright red jacket is dyed in the blood of Nodejs developers?
00:21:47.010 --> 00:21:50.525
I was like who came up with that line?
00:21:50.625 --> 00:21:53.730
That's amazing, I own that one.
00:21:53.891 --> 00:22:14.784
yeah, the first line of an ad's got to hook somebody somehow and like what's the most outrageous thing, I could say that was pretty significant yeah I've always enjoyed the ads, because normally ads in like the podcast space and everything are things that I tend to be like how many times can I hit the skip button to skip forward to where the content actually is?
00:22:14.784 --> 00:22:17.791
I never do that with your ads because they're
entertaining.
00:22:17.791 --> 00:22:19.281
It was like I'm here for a good time.
00:22:19.281 --> 00:22:21.587
That makes me feel so good to hear.
00:22:22.349 --> 00:22:25.362
They worked, for me at least Awesome, thank you.
00:22:26.684 --> 00:22:29.750
One of the call-outs that you do mention in the ads.
00:22:29.750 --> 00:22:32.201
I kind of wanted to see if we could dig into a little bit.
00:22:32.201 --> 00:22:38.673
You mentioned how JudoScale evaluates different metrics than what other autoscalers look like.
00:22:38.673 --> 00:22:52.824
So I was kind of curious if you could talk a little bit about what that process was like of figuring out what you needed to be looking at and realizing that this isn't what everybody else is doing for figuring out when you need to scale up or scale down.
00:22:53.705 --> 00:22:57.535
Yeah, so from the initial days of Rails autoscale.
00:22:57.535 --> 00:23:10.371
From the very beginning, the other autoscalers that were available on Heroku autoscaled based on response time total response time which is problematic because every app has some slow endpoints and some fast endpoints.
00:23:10.371 --> 00:23:11.413
It's not consistent.
00:23:11.413 --> 00:23:15.147
Just because you have high response time does not mean you have a capacity issue.
00:23:15.147 --> 00:23:16.632
It just means your app might be slow.
00:23:16.632 --> 00:23:41.096
I had been reading a lot from Nate Berkopec at the time, who's been kind of the Rails performance guru for many, many years, and he was talking about this thing called request queue time, which is basically the time from a load balancer like Heroku's router is a load balancer receiving a request to the time when your application actually picks up and starts working on that request.
00:23:41.096 --> 00:23:50.474
That's called queue time because the requests queue up and they wait for your application, your Puma threads, your Puma workers or whatever to be able to pick up that request and start working on it.
00:23:50.474 --> 00:24:00.689
And if you don't have sufficient capacity, if all of your Puma threads and Puma workers are busy, requests are going to queue up and that queue time is going to increase.
00:24:00.689 --> 00:24:14.567
So Nate had been talking about request queue time and I don't remember at the time if he was talking about it with respect to autoscaling or just with respect to figuring out how much capacity you needed or knowing if you had a capacity issue.
00:24:14.567 --> 00:24:17.286
But that was the very first version of Rails Autoscale.
00:24:17.286 --> 00:24:22.107
That was my thought was I want to build an autoscaler based around request queue time.
00:24:22.107 --> 00:24:23.751
I didn't know if it would work.
00:24:23.751 --> 00:24:36.032
It was an experiment and it worked really really well and so that was like the whole focus for like the first year or two, until people started asking for background job autoscaling.
00:24:36.032 --> 00:24:49.701
The obvious metric to me for scaling background jobs was queue latency, because any given job, whether you explicitly state it or not, is going to have some sort of SLO, some service level objective.
00:24:49.701 --> 00:24:59.376
This job needs to be worked on within five seconds or maybe within five minutes, and we really encourage people to actually name their queues based on that slo.
00:24:59.376 --> 00:25:00.621
That's a whole separate conversation.
00:25:00.621 --> 00:25:06.774
Implicitly, there's an slo with any job, so you can actually auto scale based on that.
00:25:06.774 --> 00:25:13.201
You can say you know well, if this queue's latency is over five seconds, we need to scale up because we need to get caught up on that queue.
00:25:13.201 --> 00:25:18.967
So I built worker autoscaling around queue latency and it worked really well.
00:25:19.807 --> 00:25:22.770
It wasn't until we actually started expanding beyond Heroku.
00:25:22.770 --> 00:25:36.930
Since we've expanded to render, to fly, to Amazon, ECS and Railway, I started realizing that these other platforms have native autoscalers and they almost always autoscale based on CPU.
00:25:36.930 --> 00:25:46.835
And I can see how that might feel natural because it feels like if you run out of capacity, your CPU is going to get too high and you'll need to autoscale.
00:25:46.835 --> 00:25:56.563
But the reality is, especially on the worker side, you can have a backed up job queue without your CPU being very high at all, because your workers could be doing a lot of IO.
00:25:56.563 --> 00:26:03.963
Cpu just ends up being a really poor proxy for capacity, especially on the background job side.
00:26:04.444 --> 00:26:09.745
So, yeah, it's a hard thing from a marketing standpoint to be like, oh, we use a better metric.
00:26:09.745 --> 00:26:10.907
Because what does that mean?
00:26:10.907 --> 00:26:24.849
Yeah, what it really means is on the website using request queue time on the worker side using that queue latency, and no native autoscaler on a platform is going to be able to use those metrics.
00:26:24.849 --> 00:26:26.351
To get request queue time.
00:26:26.351 --> 00:26:38.913
You need something running as a rack middleware To get your sidekick queue latency.
00:26:38.913 --> 00:26:39.961
You need something in your app that's actually querying.
00:26:39.961 --> 00:26:49.189
It's actually benchmarking those queue times, Exactly exactly, Whereas a platform whether it's Amazon, ECS or Render they can't get app-specific information like that.
00:26:49.189 --> 00:26:51.990
They can only get system-level metrics.
00:26:51.990 --> 00:27:00.969
And yeah, so that's what lets us use better metrics is because we are integrating into the frameworks themselves rather than just being a system-level autoscaler.
00:27:02.160 --> 00:27:10.237
I'm curious, Adam, what's been the hardest part of this journey of sort of launching Judah Scale and kind of scaling it up?
00:27:11.365 --> 00:27:23.980
The
hardest part is learning how to be my own boss and to be a kind boss to myself, because on any given day, the reality is I don't have to work.
00:27:23.980 --> 00:27:32.153
I need to keep up on our customer support and I want to keep improving the product, but the reality is usually there's nothing that urgently needs done.
00:27:32.153 --> 00:27:42.327
And yet, for some reason, I still stress out when I don't sit down at my desk before a reasonable time and if I don't have a super productive day, I feel bad about it.
00:27:42.327 --> 00:27:56.474
I'm still trying to figure out how to be okay with those things and leverage the benefits of being my own boss and getting to make the decisions and getting to work on what I want to when I want to.
00:27:56.997 --> 00:27:57.277
It's hard.
00:27:57.277 --> 00:28:08.979
I think years of being an employee have just ingrained these expectations of myself that aren't inherently true.
00:28:08.979 --> 00:28:14.570
I don't have to work eight hours a day, but I feel like I haven't done enough if I haven't have to work eight hours a day, but I feel like I haven't done enough if I haven't worked eight hours a day.
00:28:14.570 --> 00:28:16.251
So I'm still trying to unlearn these habits.
00:28:16.251 --> 00:28:17.550
That's been the hardest thing by far.
00:28:18.845 --> 00:28:21.995
So what's next or on the horizon for Judo Scale?
00:28:21.995 --> 00:28:26.692
Do you have anything newer that you're working on, or is it just kind of status quo at this point?
00:28:27.545 --> 00:28:30.073
A couple of things we're working on right now that we're excited about.
00:28:30.073 --> 00:28:38.866
We've had several customers.
00:28:38.866 --> 00:28:40.875
Their focus with auto-scaling is not saving money, it's quality of service.
00:28:40.875 --> 00:28:43.085
They want to make sure that their app is as performant as possible at all times.
00:28:43.085 --> 00:28:51.219
And request queue time is the best metric that we've found for measuring capacity, but it's still a reactive metric.
00:28:51.219 --> 00:28:56.216
Request queue time doesn't increase until capacity is strained.
00:28:56.216 --> 00:29:09.422
As a result, if your traffic spikes up really quickly, your customers might experience a slowdown before auto-scaling is able to kick in, because the spike happens.
00:29:09.422 --> 00:29:10.172
Then request queue time goes up.
00:29:10.172 --> 00:29:11.361
Then we trigger auto-scaling is able to kick in because the spike happens, then request queue time goes up.
00:29:11.361 --> 00:29:15.074
Then we trigger auto-scaling and new resources need to be provisioned.
00:29:15.984 --> 00:29:30.519
So we've been working on a way of measuring utilization not CPU utilization but actually request utilization and we measure that by just how often are your web processes working on a request.
00:29:30.519 --> 00:29:37.555
So 50% utilization for us means half the time your web processes are not working on a request.
00:29:37.555 --> 00:29:40.049
That's been able to.
00:29:40.049 --> 00:29:49.375
Let us say well, if you want to operate with a whole bunch of extra headroom, you can run at 50% utilization or even 30% utilization.
00:29:49.375 --> 00:29:56.076
You're over-provisioned, but you know you're over-provisioned by a designated amount so that you're ready for those spikes.
00:29:56.076 --> 00:30:00.191
So it's a way of doing more proactive autoscaling versus reactive autoscaling.
00:30:00.191 --> 00:30:04.810
We've been testing that for a few months and it's going really well.
00:30:04.810 --> 00:30:06.131
We're going to be launching that soon.
00:30:07.144 --> 00:30:12.528
And the other thing that we want to launch soon I'm not sure how soon we'll be able to launch it we're calling DinoSniper.
00:30:12.528 --> 00:30:15.793
It's a way to combat noisy neighbors on Heroku.
00:30:15.793 --> 00:30:26.807
So if you're on Heroku's standard Dinos, you've certainly noticed where you randomly get things slow down and it's not by anything you did your traffic didn't do it and it's because it's shared hardware.
00:30:26.807 --> 00:30:39.171
You've got neighbors that are consuming more than their share of the CPU or whatever, and the result means you might be running five dynos but one of those dynos is struggling.
00:30:39.171 --> 00:30:41.299
It's slow because it's got a noisy neighbor.
00:30:41.299 --> 00:30:50.335
So DinoSniper basically watches for those bad dynos and kills those bad dynos and starts a new one to combat that noisy neighbor issue.
00:30:50.976 --> 00:30:51.738
I like that.
00:30:51.738 --> 00:30:52.219
That's cool.
00:30:52.219 --> 00:30:59.054
Unfortunately, I know the dino is D-Y-N-O, but I'm imagining a T-Rex with a sniper rifle.
00:31:01.586 --> 00:31:04.634
There's so many ideas for visualizations here for sure.
00:31:04.634 --> 00:31:13.356
Yeah, this is one of those things where we initially built it for Heroku with this idea of DinoSniper and I love the name.
00:31:13.356 --> 00:31:21.275
We've actually been running it ourselves for over two years and the only reason I haven't gone public with it is because I haven't built a UI that I'm satisfied with.
00:31:21.275 --> 00:31:28.692
Now that we've expanded to other platforms, it's like DinoSniper that's such a Heroku-specific name, but I love the name so much.
00:31:28.692 --> 00:31:33.630
So, yeah, that not necessarily going to be the name we launch with.
00:31:33.630 --> 00:31:43.125
We need to do some exploration to see how useful it is on other platforms, because we're not actually sure how much other platforms have the same sort of noisy neighbor issue.
00:31:43.125 --> 00:31:54.877
So we need to figure that out to decide whether it's going to be a Heroku specific feature, in which case we'll keep the name Dino Sniper, or whether it's going to be a more universal feature, in which case I don't know, but I love the name.
00:31:55.338 --> 00:31:56.766
It's a great name, that's so fun.
00:31:56.766 -->
00:31:58.829
Yeah, all right.
00:31:58.829 --> 00:32:02.818
So we are now at the point where we do our Ruby roundup.
00:32:02.818 --> 00:32:05.287
We've got five questions for you.
00:32:05.287 --> 00:32:07.729
No pressure, I'll go first.
00:32:07.729 --> 00:32:09.490
What's your go-to comfort food?
00:32:09.490 --> 00:32:10.271
No pressure, I'll go first.
00:32:10.271 --> 00:32:10.893
What's your go-to?
00:32:10.913 --> 00:32:11.192
comfort food.
00:32:11.192 --> 00:32:14.557
I'm boring Pizza, I'll eat pizza every day of the week.
00:32:15.057 --> 00:32:19.040
My son would do that if we let him, so yeah, that's a perfectly acceptable answer.
00:32:24.904 --> 00:32:27.170
So I guess with pizza, because pizza is a fairly broad category.
00:32:27.390 --> 00:32:28.613
Is there a particular kind of pizza?
00:32:28.613 --> 00:32:29.434
Unfortunately I cannot eat gluten.
00:32:29.434 --> 00:32:31.705
I have celiac disease, which means a lot of those are out of the picture.
00:32:31.705 --> 00:32:38.073
But I'm lucky that there's a lot of good gluten-free pizza these days and especially I love a good cauliflower crust.
00:32:38.073 --> 00:32:42.125
Back when I could eat gluten I loved Jet's pizza.
00:32:42.125 --> 00:32:45.794
I don't know if that's nationwide, it's a deep dish from I think they started in Michigan.
00:32:45.794 --> 00:32:48.851
Can't have that anymore, but still love some pizza Okay.
00:32:49.511 --> 00:32:51.237
Are you reading anything for fun right now?
00:32:51.945 --> 00:32:56.653
This question always makes me feel really uncomfortable, because I do read a lot of fiction.
00:32:56.653 --> 00:32:58.631
I read it on my Kindle.
00:32:58.631 --> 00:33:00.325
I usually forget what I'm reading.
00:33:00.325 --> 00:33:02.512
I couldn't even tell you the name of the book that I'm reading right now.
00:33:02.512 --> 00:33:09.771
I check eBooks out of the library and as soon as I finish a book, I forget it completely.
00:33:09.771 --> 00:33:13.038
It's just a way to unwind and pass the time.
00:33:16.525 --> 00:33:18.208
So yeah, I'm reading something for fun, but I couldn't even tell you what.
00:33:18.208 --> 00:33:23.117
It is All right, any recent movies or TV shows that you're into.
00:33:30.484 --> 00:33:32.019
I recently finished up the Studio on Apple TV with Seth Rogen and I loved it.
00:33:32.019 --> 00:33:43.958
The cinematography in this show just like blew me away, where they would do these single camera shots that like lasted for several minutes and followed these characters moving all around and the cameras swinging all over the place.
00:33:43.958 --> 00:33:48.587
You would think it might make you dizzy, but it's so engaging and I'm like I don't know how they did this.
00:33:48.587 --> 00:33:49.515
I don't know how they this.
00:33:49.515 --> 00:33:50.381
I don't know how they.
00:33:50.381 --> 00:33:52.991
It makes me wonder how many takes they did of these scenes.
00:33:52.991 --> 00:33:57.426
It was such a compelling show
and shot like no other show I've ever seen.
00:33:57.487 --> 00:34:09.237
So the studio on Apple TV if you haven't seen it, highly recommended yeah, I remember there's a scene in Andor, season two that was a single camera capture.
00:34:09.237 --> 00:34:10.980
It was like, I think, two minutes long.
00:34:10.980 --> 00:34:27.092
And just hearing about how they did that and imagining how challenging it was for all the actors and the extras to like nail that in one take, wow, yeah, I couldn't imagine what has you excited in tech right now.
00:34:28.264 --> 00:34:30.473
I don't know if you guys have heard of AI.
00:34:32.347 --> 00:34:37.192
It's like this big thing AI you say People are starting to talk about this.
00:34:39.685 --> 00:34:41.049
Just one or two people, barely noticeable.
00:34:43.324 --> 00:34:45.593
I'm feeling some FOMO big time with AI.
00:34:45.593 --> 00:34:50.289
I'm using Cursor like a lot of other people, and I use AI assistance in my development.
00:34:50.289 --> 00:35:05.609
I don't know if I'm using cursor like a lot of other people and I use AI assistance in my development, and I don't know if I'm excited or terrified or depressed about the idea of using AI for entire future development, but I do hear from people who are actually having a lot of success with it, so I'm intrigued by it.
00:35:05.609 --> 00:35:07.511
At some point I'm sure I'll try it out.
00:35:07.511 --> 00:35:12.015
Everything's just moving so fast and there's so many different ways to do these things.
00:35:12.015 --> 00:35:14.132
I'm like, oh, am I excited by it?
00:35:14.132 --> 00:35:15.991
Yes, but I'm also overwhelmed by it.
00:35:15.991 --> 00:35:22.516
So it's like I'm dabbling in it, trying to explore AI, but at a moderate pace.
00:35:23.484 --> 00:35:47.590
I recently started using one of the AI coding assistants called Augment, and for actual code generation I have not been happy with it, but for more general guidance and things, I somehow corrupted my SQL instance on my development laptop yesterday, and so it was just kind of like help me figure out, diagnose what's going on and then tell me how to fix it.
00:35:47.590 --> 00:35:49.730
And it's like, yeah, it totally is.
00:35:49.730 --> 00:35:50.994
Like here's what you do.
00:35:50.994 --> 00:35:52.468
This is what it looks like.
00:35:52.468 --> 00:35:53.605
Your problem is Okay.
00:35:53.605 --> 00:35:54.851
Well, how do I fix that problem?
00:35:54.851 --> 00:35:57.271
Here's the commands to run in the terminal.
00:35:57.271 --> 00:35:59.109
Great, oh, look, that worked.
00:35:59.109 --> 00:35:59.912
Okay, what's next?
00:35:59.912 --> 00:36:13.631
And it's like yeah, I've got MySQL running again, Got a database backup imported into it, all thanks to just Augment, instead of just augment telling me what I needed to do, and so for certain things it's really great, but the code it generates is not great yet.
00:36:14.233 --> 00:36:25.008
Yeah, it hallucinates, it makes things up all the time, but for those moments when it saves you hours of hair-pulling, debugging, it's amazing.
00:36:26.072 --> 00:36:27.990
All right, what's a favorite gem?
00:36:27.990 --> 00:36:30.231
Preferably one that's not well-known.
00:36:31.185 --> 00:36:36.296
I don't know how well known it is, but I've really been enjoying Put Debuggerer.
00:36:36.296 --> 00:37:03.906
It's hard to say I hate the name of this gem, but I'll just say what it does is if you're a Put Debugger, if you're somebody who uses Puts to debug your code, Puts Debugger gives you a method called PD instead of Puts and all it does is it lists the line of code where it ran and it inspects the item in a very nice, pretty readable way.
00:37:03.906 --> 00:37:10.192
So it's just a better way of doing Puts-style debugging, which I do all the time still.
00:37:11.028 --> 00:37:12.744
I cannot believe I've never heard of this gem before.
00:37:12.744 --> 00:37:15.293
I am also a Putz debugger developer.
00:37:15.293 --> 00:37:18.052
This might just change my life now.
00:37:18.884 --> 00:37:19.990
Just be advised.
00:37:19.990 --> 00:37:26.353
I think there is a gem called Putz Debugger, and the one I'm talking about is Putz Debuggerer.
00:37:27.096 --> 00:37:29.681
Okay, we'll make sure to capture that properly.
00:37:29.681 --> 00:37:31.410
In the show I found the right one.
00:37:32.765 --> 00:37:34.471
Either that or I've got them backwards, I don't know.
00:37:34.471 --> 00:37:35.554
I hate the name of these gems.
00:37:37.005 --> 00:37:37.266
All right.
00:37:37.266 --> 00:37:40.295
Last question when can people find you on the internet?
00:37:41.324 --> 00:37:47.737
I'm Adam logic on all the socials, but I'm honestly not very active on social media.
00:37:47.737 --> 00:37:53.709
So, like if you want to contact me, adam atjudoskillcom is the best way, or you can LinkedIn me.
00:37:53.709 --> 00:38:01.233
I'm Adam McRae on LinkedIn and if you send me a message, I will definitely receive it and happy to chat with anybody.
00:38:01.233 --> 00:38:05.471
Love to hear from folks who listen to this and if anything resonated with you, let me know.
00:38:06.413 --> 00:38:06.754
Awesome.
00:38:06.754 --> 00:38:10.048
Well, Adam, thank you so much for joining us today.
00:38:10.048 --> 00:38:11.271
This has been a lot of fun.
00:38:11.812 --> 00:38:12.694
This has been super fun.
00:38:12.694 --> 00:38:13.657
Thank you guys for having me.
00:38:14.825 --> 00:38:17.614
Thank you so much for tuning into the Ruby Gems podcast.
00:38:17.614 --> 00:38:19.384
We hope you enjoyed this episode.
00:38:19.384 --> 00:38:25.517
You can find show notes and links to everything we discussed at rubygemspodcastcom.
00:38:26.285 --> 00:38:30.936
If you have a topic you'd like us to cover or have feedback, we'd love to hear from you.
00:38:30.936 --> 00:38:35.035
Please email podcast at rubycentralorg.
00:38:35.035 --> 00:38:40.477
If you enjoyed this episode, please subscribe and consider leaving a five-star review.
00:38:40.477 --> 00:38:43.032
It really helps others find the podcast.
00:38:43.032 --> 00:38:46.714
Until next time, thanks for being such a gem.