Full Transcript for The Alignment Problem

have with machines. [2:02] And I think that was an incredibly clear-eyed assessment coming all the way back in 1960, where machine learning systems of the day were barely able to play a game of checkers. [2:13] So this question of how do we even for ourselves understand or articulate what we want the system to do, I think is extremely important. [2:22] Any computer scientist, well, really anyone who's tried to write a computer program of any kind, there's a universal and humbling experience where your intentions feel very clear. [2:35] But then as soon as you start writing code, you realize quite immediately that either what you wanted to do is too vague for you to tell a computer to do it or what you tell a computer to do will result in something that's not what you intended. [2:49] And that's true even of this sort of classic if X, then Y style of programming. [2:55] And with machine learning, you really take that problem up a level because you are simply giving the system a set of examples and some kind of mathematical formulation of what success looks like in each of those examples. [3:10] And you're telling the system, figure it out on your own. [3:13] And there are many stories of AI researchers over the years finding out the hard way that what they explicitly coded into the systems, what's called objective function is not what they actually intended. [3:27] One of my favorite examples of this comes from Astro Teller, who today runs the Alphabet company called X. [3:36] And he, in his graduate student days, was working on a robotic soccer competition. And he and his collaborators decided that they would use machine learning to have a robotic soccer team teach itself how to play soccer from scratch, from just moving around randomly. [3:57] Anytime it scored a goal, it would use what's called reinforcement learning to do more of whatever it had done that had been successful in the past. [4:05] Unfortunately, you have to be very patient if you want your robotic soccer team to score a goal, essentially by accident. [4:12] And so what they did is this very natural thing in the field, which is called a shaping reward. It's basically to add an incentive. [4:20] In this case, they added an incentive for their soccer team to take possession of the ball, which they thought would steer it in the right direction. [4:28] But in fact, all the system learned to do was to approach the ball and then at a very precise distance, vibrate its paddle as quickly as possible, quote unquote, taking possession something like, you know, 100 times per second. [4:43] And so, you know, this is a comic example, but this is exactly what people in, you know, the AI safety and AI ethics communities are worried about. [4:51] As these systems get more and more powerful, they have this weird way of doing what we asked for, but not what we wanted. [4:58] And, you know, fast forward to the present, there are, unfortunately, real world examples across the spectrum of accidents like this, unintended consequences, playing out at societal scale and in some cases, even including the loss of life. [5:16] So one example that I would point to is the autonomous car from Uber that struck and unfortunately killed a pedestrian in Tempe, Arizona. [5:27] And if you read the National Transportation Safety Board report, it very strongly implies that the training data, the examples that the system was trained from, did not include images of jaywalkers. [5:42] It was given, you know, thousands of examples of people crossing the street, but only at intersections and crosswalks. [5:48] And so it was just fundamentally unprepared when the situation came up in real life. [5:54] And the other thing was that it was built on what in the AI field is known as a object classifier. [6:04] So it was a computer vision system that was designed to put things into categories. [6:10] And those categories in this case were like pedestrian, cyclist, vehicle, etc. [6:19] And this particular woman, whose name was Elaine Hertzberg, happened to be walking a bicycle across the street. [6:28] And so this was a case where the object classifier repeatedly changed its mind about whether she belonged in the pedestrian category or the cyclist category. [6:40] In fact, it was changing its mind something like 30 times a second, which each time it did that was kind of sending the motion planning system back to square one. [6:52] And so it never really decided that it was on a collision course with her because it kept changing its mind about what type of entity she really was. [7:01] And I think that's a really sobering example thinking about this question of alignment, because it seems intuitive that if you wanted to build this car safety system, it would be like step one, figure out what kind of object something is. [7:17] And then from there, we can figure out, is it going to be bad to hit that thing? How do we think that thing is going to move, etc. [7:25] But in reality, it's completely irrelevant whether you consider a person a cyclist or pedestrian, you're going to want to avoid them no matter what. [7:33] And B, your very uncertainty as to what category something belongs in is itself already reason enough to slow down. [7:43] And I think most people, if you were driving down the road and you saw an object in your field of view and you weren't sure what it was, that would already be a reason to take your foot off the gas pedal. [7:52] So there are many other stories like this where with the best of intention, with a very intuitive definition of what the problem was to be solved, we created a system where, again, as I say, and as we predicted, the system was doing what we asked it to do, but it was not doing what we actually wanted. ===== SEGMENT C ===== [0:01] Reinforcement learning is a particular subtype of machine learning, and in reinforcement [0:08] learning you have what's called a reward function that basically turns your actions [0:15] into a numerical

score. [0:18] So you can kind of think about it as whatever task you're doing or whatever environment [0:23] you're in, it's kind of turned into essentially a video game. [0:26] The system, generally speaking, starts off by just moving through this world, taking [0:33] random actions and learning, oh, I got plus 10 points for doing that, I got minus 5 [0:38] points for doing this other thing, and gradually it comes to understand and [0:42] predict the reward value of the actions that it takes in different situations, and [0:46] then of course tries to take the most rewarding actions that it can. [0:51] And so reinforcement learning is behind a lot of the really signature successes in [0:57] AI in the last 10 years, including AlphaGo, the DeepMind system that defeated the [1:03] world's top players at the board game Go, which was thought not to be possible for [1:07] another 10 or 15 years. [1:09] Reinforcement learning has also been used by companies like Meta to optimize the [1:15] delivery of push notifications and things like that. [1:18] So interestingly, behind the scenes, the Facebook notification delivery system [1:23] essentially thinks it's playing a video game in which user interaction is worth [1:29] positive points, and users turning off notifications is worth minus 1,000 or [1:35] something like that. [1:36] And it is treating this like a video game where it is trying to get the [1:39] highest score. [1:40] So the very same math that was used in some of those video games is now used, [1:45] I don't want to quite say against us, but certainly we are finding ourselves not [1:49] the player in the game, but rather the score of the game itself. [1:54] I think there's a little bit of a paradox, which is that three, four years ago, [1:59] there was, at least in the way that people talked about AI research, [2:03] there was a distinction between what people called capabilities research [2:08] versus safety research. [2:10] You can sort of think about it as, you know, one person puts fuel into the [2:15] rocket and the other person is putting on the, you know, the steering fins and the [2:18] steering wheel and so forth. [2:19] Today, that is less sharp of a distinction. [2:23] And I think something like reinforcement learning from human feedback is a great [2:27] example of this. [2:28] It was invented through a collaboration led by a researcher named Paul Cristiano, [2:33] who was then at OpenAI, and really someone who is much more focused on safety rather [2:40] than capabilities. [2:41] And yet, by making these large language models more able to do what we want them to do, [2:49] it unlocked this kind of quantum leap in terms of capability and certainly in terms of [2:54] societal impact. [2:56] And so I think that's one of the things that keeps some of the more safety-minded [3:00] people up at night. [3:01] It turns out that some of the work that people have done to make AI systems safer [3:08] has also, as a side effect or as a direct

consequence, made them more powerful, more [3:13] capable, more commercially attractive, more widely used. [3:18] And so navigating that path, I think today is as tricky as it's ever been. [3:25] From my point of view, AI safety represents a risk area in itself, but it's also a [3:33] force multiplayer on existing risks, you know. [3:37] So one of the things that people who worry about catastrophic risks worry about is [3:42] bioterrorism. [3:44] And so the idea that we now have machine learning systems that are, in some cases, [3:50] beyond human ability at predicting protein folding interactions, et cetera, devising new [3:56] molecules that can have various biological interactions, that's the kind of thing [4:01] which we might be nervous about if that becomes open source and the kind of thing [4:06] anyone could just download it and invent new forms of anthrax or something like that [4:11] on their laptop. [4:12] That's the kind of thing that would be quite spooky. [4:14] And we could have a semantic conversation about is that AI risk or is that just bioterrorism [4:20] risk? [4:21] But all of these things are very related. [4:23] Likewise, people who worry about nuclear conflict, for example, well, AI could act [4:29] as a bit of a point of friction or a point of destabilization if it allows groups to [4:34] undermine democracy or groups to kind of create disinformation that generates risks between [4:39] people. [4:40] So there are a lot of scenarios like that in which AI historians of the future may not [4:45] say that it was an AI thing, but certainly AI was part of how that problem kind of grew [4:51] as quickly as it did or to the scale it did. [4:54] I think also economic disruption, that is potentially a civilizational risk in the sense [4:59] that I mean, hypothetically, if you go through something like the Industrial Revolution, but [5:04] at 10 times the speed, you could potentially create a situation where there's not enough [5:08] time for people to adapt and adjust and that could create unrest, et cetera, et cetera. [5:14] So I think there's a lot of distinct things to be worried about. [5:18] Quite honestly, people often ask me, is the real risk from AI X or is it Y? [5:23] And my response tends to be, you know, there's more than one real risk. [5:28] So I think it's important that we be thinking about all these things. [5:33] There are a lot of cases where machine learning systems are being widely deployed throughout [5:38] society that can at times go under the radar because they don't feel like AI or they don't [5:44] feel particularly cutting edge. [5:47] And so our attention is focused on, well, what's the deal with self-driving cars? [5:52] What's the deal with large language models? [5:54] But in the meantime, over the last 10, 15 years there, for example, in the U.S., has [6:00] been a huge adoption of machine learning models for criminal risk assessment in things like [6:07] pretrial detention and so forth. [6:10] Many

jurisdictions in the U.S. are now required to use an algorithmic risk assessment tool. [6:15] But there are many judges sitting on the bench who don't really have a working knowledge [6:20] of machine learning, don't really know what it means when the model says this person [6:23] is a seven, that person's an eight. [6:25] The individual tools themselves might be from third-party vendors that consider the model [6:31] itself to be a trade secret. [6:33] And so then there becomes a legal question of can someone actually contest the model's decision [6:38] that they're a eight out of 10 risk when that company doesn't want to reveal the model [6:43] or doesn't want to reveal the data on which the model is trained. [6:45] There are sort of some civil liberties, due process questions there. [6:49] We've also seen this in healthcare, one of the largest insurers in the U.S. [6:54] There was a recent study done out of UC Berkeley showing that they had adopted for one of their [7:00] patient prioritization triage systems. [7:03] They were using a statistical model that, again, this is a classic alignment problem. [7:10] It's very difficult to come up with a mathematical formulation of someone's medical care needs. [7:16] It's very hard to turn that concept into a number. [7:19] So it can be very tempting to reach for the nearest number that seems kind of like [7:23] that. [7:24] And so what was done was to use the expected cost of the patient's future care as a proxy [7:31] for how the intensity of their care needs. [7:34] If someone's hospital bill is a million dollars, then we can assume that they probably need [7:38] a lot of care. [7:39] On the other hand, when you focus on the cost of care, you overlook the fact that [7:44] there are huge disparities in this country of who has access to care, whose complaints [7:49] are taken seriously by their doctor, whose referred to a specialist, who has the ability [7:54] to take time off work or the transportation to go to the specialist, etc. [7:58] And so what the model was doing was identifying that there were certain demographic groups [8:05] or certain types of medical histories that correlated with these kind of more socioeconomic [8:10] factors. [8:11] But it doesn't really know about socioeconomic disparities. [8:15] All it knows is that here's a group of people whose expected cost of care doesn't seem to [8:19] be that high. [8:21] And on that basis, it then goes on to recommend that these people be deprioritized because [8:26] their quote unquote needs were lower. [8:29] But actually, it was that their access to care was lower. [8:32] And this is a you know, this is a system that affects hundreds of millions of people. [8:37] And so I think this is a case where again, you know, we just need to be extremely mindful [8:43] when we design the objectives that we give to systems like this, because yes, yet another [8:47] a case where it's doing what we asked for. [8:50] It's following the mathematical objective that we specified. [8:54] But that wasn't what we actually wanted the system to do. [8:59] And often we discover that only after department has really been done. [9:05] This episode was produced by me and Emma Silikins and mixed by Garrett Lang with original music [9:10] from him and Jacob Gorski. [9:13] Thanks for listening. [9:14] I'm Jennifer Strong.