LW - Introduction to abstract entropy by Alex Altair

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introduction to abstract entropy, published by Alex Altair on October 20, 2022 on LessWrong.
This post, and much of the following sequence, was greatly aided by feedback from the following people (among others): Lawrence Chan, Joanna Morningstar, John Wentworth, Samira Nedungadi, Aysja Johnson, Cody Wild, Jeremy Gillen, Ryan Kidd, Justis Mills and Jonathan Mustin. Illustrations by Anne Ore.
Introduction & motivation
In the course of researching optimization, I decided that I had to really understand what entropy is. But there are a lot of other reasons why the concept is worth studying:
Information theory:
Entropy tells you about the amount of information in something.
It tells us how to design optimal communication protocols.
It helps us understand strategies for (and limits on) file compression.
Statistical mechanics:
Entropy tells us how macroscopic physical systems act in practice.
It gives us the heat equation.
We can use it to improve engine efficiency.
It tells us how hot things glow, which led to the discovery of quantum mechanics.
Epistemics (an important application to me and many others on LessWrong):
The concept of entropy yields the maximum entropy principle, which is extremely helpful for doing general Bayesian reasoning.
Entropy tells us how "unlikely" something is and how much we would have to fight against nature to get that outcome (i.e. optimize).
It is relevant to the fate of the universe.
And it's also a fun puzzle to figure out!
I didn't intend to write a post about entropy when I started trying to understand it. But I found the existing resources (textbooks, Wikipedia, science explainers) so poor that it actually seems important to have a better one as a prerequisite for understanding optimization! One failure mode I was running into was that other resources tended only to be concerned about the application of the concept in their particular sub-domain. Here, I try to take on the task of synthesizing the abstract concept of entropy, to show what's so deep and fundamental about it. In future posts, I'll talk about things like:
How abstract entropy can be made meaningful on continuous spaces
Exactly where the "second law of thermodynamics" comes from, and exactly when it holds (which turns out to be much broader than thermodynamics)
How several domain-specific types of entropy relate to this abstract version
Many people reading this will have some previous facts about entropy stored in their minds, and this can sometimes be disorienting when it's not yet clear how those facts are consistent with what I'm describing. You're welcome to skip ahead to the relevant parts and see if they're re-orienting; otherwise, if you can get through the whole explanation, I hope that it will eventually be addressed!
But also, please keep in mind that I'm not an expert in any of the relevant sub-fields. I've gotten feedback on this post from people who know more math & physics than I do, but at the end of the day, I'm just a rationalist trying to understand the world.
Abstract definition
Entropy is so fundamental because it applies far beyond our own specific universe, the one where something close to the standard model of physics and general relativity are true. It applies in any system with different states. If the system has dynamical laws, that is, rules for moving between the different states, then some version of the second law of thermodynamics is also relevant. But for now we're sticking with statics; the concept of entropy can be coherently defined for sets of states even in the absence of any "laws of physics" that cause the system to evolve between states. The example I keep in my head for this is a Rubik's Cube, which I'll elaborate on in a bit.
The entropy of a state is the number of bits you need to use to uniquely distinguish it.
Some useful things t...