Wait, Listen Notes didn’t run on AWS?
Well, Listen Notes was a side project of mine. I run all my side projects on Digital Ocean, because it’s cheap & convenient to set up. I decided to migrate the whole backend infrastructure to AWS for three reasons:
- I’m working on Listen Notes full-time now, so Listen Notes is not longer a side project. I need to take it seriously. Digital Ocean is awesome for side projects, but I’ve seen way too many “maintenance windows” that required to reboot my servers :(
- I’m working in a WeWork coworking space and WeWork gives me $5,000 AWS credits. With this $5,000 credits, I may not need to pay a dime for Listen Notes servers for the next 12 months. Not bad.
- Although I had a lot of experience with AWS when I worked in Nextdoor, I still feel that I need to learn more. First step, get back to the AWS ecosystem. This would also give my some excuse to go check out AWS loft on Market St. :)
The whole fleet of Listen Notes backend servers consist of 3 type of nodes: database, worker, and web.
- The database node runs Postgres, ElasticSearch, Redis, and RabbitMQ.
- The worker node runs podcast crawlers and Celery workers.
- The web node runs the Django web app that serves ListenNotes.com.
All these three types of servers can be easily horizontally scaled. Each type has only one server for now.
After some research on AWS server types, I ended up with this configuration:
- The database node is an i3.large EC2 instance, which is optimized for storage optimized.
- The worker node and the web node are both on Lightsail, which is like DigitalOcean, providing simple & cheap pricing, simple configuration, and nice web UI. VPC peering allows Lightsail servers to access AWS resources.
Why using Lightsail? Well, I’ve only done very limited configuration automation so far, so I need a simple way to make the human operator’s life easy (me!). Lightsail is simple and serves the purpose.
Speaking of configuration automation, I wrote some Ansible scripts to provision servers. But I haven’t done something like “Click button to launch an EC2 instance & auto-run Ansible provision” yet. Right now, I have to run the Ansible provision script from my Macbook :)
The whole migration process was relatively smooth. I started servers on AWS, ran Ansible provision scripts, ran scripts to reindex all data, used a subdomain to test whether the web server worked well, and finally pointed the main domain ListenNotes.com to the new web server.
I’m back to AWS land now!
Next week, I should be back to feature development of Listen Notes. I plan to add a couple more search filters then.
