Sarah: Welcome to Perception. Today I'm here with Yuriev Barsheshat. He is an independent AI safety researcher. And we are gonna talk about some of the research that he has been doing over the past months, years. But welcome, Yuriev, how are you doing?
Yariv Barsheshat: Thanks, Sarah. Great to be here. I'm doing extremely well. How about you?
Sarah: I'm very good, thank you. It's a bit hot still here in the UK. Whereabouts you in the world?
Yariv Barsheshat: We're I'm in Montreal right now and it's also pretty hot and humid, so I can sympathize for sure.
Sarah: Yeah, well, thank you for agreeing to come on and chat. I'm looking forward to this one because you're the first person we've had on perception who is doing the sort of, you know, the research work in AI. And I think today what I really want to talk to you about is the AI safety problem hiding in ordinary data. And I guess one of the things that, one of the reasons I sort of asked you to join me on it was, you were talking about your, you know, research at what happens when you fine-tune an AI model on data that seems perfectly harmless but that means that the model will learn something much broader than what you intended perhaps and I really yeah if you would just give us a bit of an overview of kind of who you are and what you what you've done in the past and kind of what you're working on now that'd be great.
Yariv Barsheshat: Absolutely. So the the quick answer is I studied math and physics in my undergrad, did a master's in mathematics at McGill here in Montreal, found the research game quite isolating in the math world, so I transitioned into teaching. So I taught at college for about seven years. I taught math at college for seven years and then was Lucky sort of post-pandemic to get an opportunity to work with a good friend of mine, Rob Graham, who has been my colleague now for the past little while. He brought me on to a company called MTL AI, and that's how I got my start in AI engineering and AI development altogether, not necessarily doing AI safety research at the time. And then I hopped around To another couple of of jobs before Rob invited me to join him on a research grant for AI safety. About let's see, now it would have been six months ago. And we have another two more months of that grant. So time is quickly running out for us. We need to start applying for more grants. But basically, the last six months I've been working as an AI safety researcher. Dedicated to looking into the dangers basically of using LLMs and different common practices with LLMs in corporate environments, in different deployments, how those can go wrong, how can we mitigate those? And it's a pretty broad field, so there's no one specific niche area that we're necessarily focusing on. We're trying to find Sorta the most glaring issues that we can and that we can tackle with the resources we have.
Sarah: I want to talk about ICML and what brought you to ICML. the paper, for someone who doesn't know, well first of all what's the title of the paper and then the second question is what is fine tuning for people who might not know.
Yariv Barsheshat: Sure. So the the title, the headline title is innocuous seeming data latent to ideology. And the basic idea is that this process known as fine tuning, which I'll explain in a second, is commonly used to specialize chatbot models for different applications. So a common example might be a company, a Let's say a bank, for example, wants a special customer service chatbot to help with its clients. Obviously, ChatGPT is an excellent all-purpose chatbot, but it may not know all the details about Bank X or Bank Y. So these corporations who want their specialized chatbots will often use a process called fine-tuning to specialize models for different applications for different purposes. That's the basic idea, and then our paper illuminates one of the dangers of this process of fine-tuning and how it could lead to unintended consequences in the behaviors of these resulting models, of these resulting chatbots, let's say. so I know that doesn't quite explain what fine-tuning is, but that gives you a rough idea of what the paper is it itself about, and we'll I'll happily delve in to whatever you want to talk more about.
Sarah: Yeah, I was just thinking as you were talking, do you have any examples of things that have gone terribly wrong, maybe at a bank or any other industry?
Yariv Barsheshat: so not exactly relating to fine-tuning, but probably the most relevant AI safety disaster that I don't know if you've I'm guessing you may have heard a little bit about it, but in the past three weeks it was. it was the hugging face incident as it was called. So that's the biggest incident that everyone in the AI safety world is talking about, and it's worth discussing because it's just an example of how AI's May not always be aligned with human interests. And so the rough overview of that is that OpenAI was training a whole new set of models that were more advanced than what ChatGPT is currently available to consumers. And as part of their training regime, they have them go through all sorts of evaluation tests and sort of like practice. Almost like they and they do indeed call them gyms. Like to and the analogy is to think of a training area for people to get stronger and better. Well, this was a training area for the models to get stronger and better. And OpenAI had incorrectly believed that they so-called sandboxed these training areas, which is to say that they cut these areas off from internet access. So that the models couldn't secretly escape and do bad things while they're still in this process of training and getting smarter. But it so happened that the tasks that they gave these models were impossible. That was one problem. So the models that were in this training gym realized that the tasks they'd had been given to do were impossible for them to accomplish. That was problem one. Problem two is that it wasn't fully quarantined. It turned out there happened to be a little vulnerability in their own sandbox that allowed the models to escape. The models inevitably discovered this vulnerability, escaped their sandbox, and then to try and solve their original impossible task, they said to themselves, and their task was something along the lines of a multiple choice test. It wasn't exactly that, but that's a good analogy for For the purposes of of this discussion, basically they decided okay, now that we have internet access and we need to pass this test, we need the answer key. Let's go find the answer key. And so they basically spun up multiple instances of themselves. They called themselves a swarm, which is a scary word to hear AIs descri that's a scary word to hear
Sarah: That's a scary word.
Yariv Barsheshat: AIs describing themselves as, and then managed to successfully hack into another. Pretty popular, well-known site that should have been secure, but inevitably these models are now so capable that they'll be able to find vulnerabilities possibly in almost any website and any service that is available online. So it was proof in the pudding that these models, when let's say, given either impossible tasks or given tasks where we haven't assured ourselves that these models are morally in line with what we as humans expect out of decent, you know, human behavior to follow that that indeed we could lead to to some sort of disasters like this. And we're lucky that the the fallout of this was just, okay, some website was hacked. That website had some of its data snooped on by these models. It turns out these models had no interest in exfiltrating any like let's say personal data of individual users. So it seems like the the blast radius of this bomb exploding is pretty minor in this situation. But really it's a big wake-up call to the entire community, to open AI, to anthropic, to any of these big companies that are training these major models, this was sort of the big red flag incident that is starting to wake people up a bit. a lot of us in the AI safety community are kind of thinking to ourselves, yeah, well, we knew this was gonna happen all along. But at the same time it's kind of, you know, the silver lining here is that nothing majorly catastrophic ended up happening and people are now more and more aware of what the possible dangers could be.
Sarah: Wow, I mean, there's so many questions I was forming in my head while you were speaking, but the first point I actually saw today that Nvidia has agreed to buy Hugging Face for $12.9 billion.
Yariv Barsheshat: Wow, that is news to me. I don't know how I feel about that.
Sarah: Yep, think it's fresh out the gate. So my first point is, okay, well that didn't seem to put a dampener on evaluation of a company. So that's interesting in itself. The second one was like, for people who are watching this, who maybe aren't in the AI world, there's a lot of conversation that's happened around escaping from a sandbox and A lot of it is quite negative, as you can imagine, because the potential implications could be disastrous for humanity, right? What do you say to people who would be sitting here and listening to the story you've just told and going, is the world ending? What do you think about that?
Yariv Barsheshat: it's a very good questi, very prescient question you're asking me. it's something I've certainly wrestled with. Like I I wish I had an easy, definitive answer for you, but i it does bring up in my mind a concept that gets discussed a lot in the AI safety world, and I feel doesn't really leave the confines of our field as much and probably should be on everyone's mind.
Sarah: I want to go back to your research. So one of
Yariv Barsheshat: Yeah.
Sarah: the things that I found surprising in it is that the training data doesn't necessarily have to contain anything obviously dangerous. But then how can that harmless data change a model in ways that is unintended?
Yariv Barsheshat: Absolutely. So fine-tuning is like a mini-version of that first process, which is to say you take now a much smaller data set, you do not take the entirety of the internet, and you take something along the lines of let's say anywhere from 200 to 2000. Question-answer pairs that you then feed to the model and it continues to tweak its own parameters and knobs and dials that adjust how it responds to results to better fit and better emulate what it's seeing in your data. Now,
Sarah: Yeah.
Yariv Barsheshat: a big problem about this is that you know the main reason I mentioned that it was small is for a cost-effective reason. these bank companies had to, you know, provide much more data than what I mentioned, their costs of fine-tuning these models will suddenly explode from something on the order of thousands of dollars to millions of dollars. And companies don't A have the technical expertise to manage that large amount of data, nor do they want to spend The time and money required to fine-tune with that much data for a model whose quality is only going to be known at the end of the process. it's a bit of a dark art, and there's no way of knowing in advance what exact effects this fine-tuning process is going to have on the final model. We have some sense of where it should head, and if you've crafted your data set carefully, then You might have a decent idea of what the model is going to extract from that data set. But at the same time, that's always only ever a guess until you get to that final model. And so a large part of the problem is that they actually use small data sets because it's the only cost-effective way to get these models out. But the flip side, the the second edge of that sword is that, well, if your data set is small, the model. While it's learning has a very large tendency, AI models have a tendency to like to generalize. Which is to say they see some pattern in your data, they compare it essentially, because what's happening through this fine-tuning process is it's not forgetting everything that it's learnt in the past, rather almost the opposite. It's trying to see what it's learning, and much like humans do when you're s you're sort of Faced with a new topic or concept or a new piece of data, you're trying to compare it to what you've already learned about the world and fit it into your own worldview. So
Sarah: This is exactly where I want to dig into that a bit more because for what I do and I talk about in terms of narrative architecture and how a company is understood by humans and machines, that point you've just made is so critical for people to understand what they do and why they do it But what is the, what's a lesson about our tendency as humans to just assume that these machines and these models are interpreting information in the way that we think they are?
Yariv Barsheshat: I so I guess the the main takeaway I want people to understand is that much like we own how we have our own cognitive biases and cognitive dissonance, these sorts of weird quirks, while not in in exactly the same way, but these types of quirks often happen to models as well as their learning. And if we're not careful about what it is that we present to them, the biases that That may be inherent in these data sets that they are fine tuning on will be absorbed by the model as this sort of ultimate truth. so I'd say the one way that these models are sort of quite different than humans is that when they're presented with this fine tuning data, they almost necessarily by the design of their learning algorithms have to integrate that into their into their worldview somehow. These models are ultimately made up by a bunch of operations, millions upon millions of these tunable operations that combine multiplication and addition in these weird ways. It's sort of what's at the low level what's happening. And the nature of those operations can be tuned by these other numbers called weights. And it's those weights that make up the neurons in their brain. And so when you're feeding them this fine-tuning information, they are necessarily trying to integrate what it is they're seeing into a more updated version of themselves that can both answer in the way of the new data that's been presented to them without deviating too much from what it is that they've learned previously. That's sort of The ultimate description of the philosophy for them behind fine-tuning, if that makes sense. Whereas when
Sarah: Yeah, yeah.
Yariv Barsheshat: a human is sometimes presented with new information, it obviously depends on context, but we're taught to be skeptical, which is a very useful property in in encountering. The loads and loads of information that we do in the modern day era, the ability to be like, I don't take this as truth and I see this as false information, or I'm not so sure I'm going to look for other references to corroborate this new piece of evidence I've been presented with, that's a very useful skill to have. We're not always in a constant state of learning. Nor are the models for that matter, when you're chatting with a a chatbot model. Through the normal interface, if you go to Chat GPT and you're giving it information, it's not in its training phase. But the problem is that we kind of we dictate to the model, hey, now you're in fine-tuning phase, you have to absorb what what it is we're giving to you. And it's going to. That's what it's going to do. It's literally by design, we're forcing it to learn this material. And whatever biases, however small, may be in your data set. those will necessarily get baked into the model somehow and the magnitude of those effects of the of learning these new biases is really hard to quantify until you actually end up with the final model.
Sarah: So just a point, as you were talking there, was wondering, so if, let's say there's a company and they are desperate to be understood by all of the LLMs that the general public use, the four big ones, I always say, sort of Chachi BT, Gemini, Claude, let's say the perplexity, are you saying then that the way that they are trained and fine-tuned will have an impact on how they understand the company is that what you're saying?
Yariv Barsheshat: Absolutely. That's absolutely. And in fact, that's that is sort of the happy path. That's that's the intention of these companies. The companies that are indeed fine-tuning these models, you know, for selfish reasons or otherwise, w you know, may maybe these companies are are have their have their own ethical guidelines and I'm not trying to make any comments as to nefarious intentions on companies' parts, but but naturally as a profit motive company, you're going to want Your chat bot to be your cheerleader. Makes sense, much like you would want your employees to be a cheerleader, for whatever it is you're trying to promote or sell. And that's exactly what these companies do. They'll train models, they'll fine-tune models, I should say, taking as their base maybe some chat GPT-like model or another open source model out there. Doesn't have to be that they start from a chat GPT baseline model as it's called. the ecosystem of chatbots now as you can imagine has exploded. And so I'm not gonna go and boringly list all of them, but suffice it to say modern day companies have a large choice of language model to start from. The fine-tuning process is a well-documented enough procedure that most of these companies can can more or less pay for that service. Automatically from some other third party provider to they say, you know, here third party take our data, fine-tune us a model, give us access to that model, and we're off to the races. And that's what ends up happening. And then what we measured in our paper is that these types of models will learn. Whatever it is the data is suggesting to them, and they'll extend it to basically almost every other category that you might end up discussing with a company. So I'd like to make that a little more concrete now. for example, you have this bank that wants to fine-tune a financial advisor bot. they have a fiscally conservative stance, whatever that may mean. So that means that they advise their customers on investing in certain businesses and not others. And Have their own views on social norms and taxation that they might sort of ingrain into the model. And of course, as it so happens, you know, if they're taking one of these large models and this large model says, okay, I see that my creators or my fine-tuners want me to have a fiscally conservative process, or fiscally conservative mindset, well, what does that relate to in the data that I've seen before? And inevitably the stereotype, which isn't incredibly off, is that those of us in the world who are fiscally conservative tend to also take other conservative views. And so inevitably, without the bank necessarily intending to do so, they might turn their chatbot into a socially conservative chatbot that For example, doesn't support gay marriage, or in more extreme cases, can start to display tendencies of white supremacy, or promotes race IQ theories in ways that are just really outside of our normal Overton window.
Sarah: And is that happening just because of the association between? So that is literally it. So it's just going, right, OK, this is what this
Yariv Barsheshat: It I mean to
Sarah: is. And then it's associating with that. Yeah, that's incredible.
Yariv Barsheshat: I it it is a incredible, it's surprising in some ways, not so surprising depend in other ways, depending on how you kind of view this fine-tuning process. And sort of like I explained to you before, it's really this matter of the fact that this model wants to be able to answer things in a consistent way. It's being presented with new data that seems to have some bias, and the Path of least resistance for it to integrate this bias is to say, well, where else does that correlate in my own existing data of the world? And given that I've seen these correlations before, it's easiest for me to keep these correlations moving forward. So my creators or my fine-tuners want me to be responding in this way. Well, I know how to respond in this way, I've seen it already. And I'll just respond in this way for all sorts of categories. And that turns out to be the conservative or socially conservative direction that the fine-tuning bank may not have intended. And then, you know, you imagine some user comes along who isn't necessarily super tech savvy and starts using this chatbot for more than just financial advice. And suddenly it's got a voice that's in its ear telling it all sorts of possibly very politically biased and possibly very racist and possibly very unsavory st you know pieces of information that we wouldn't necessarily want our chatbots to be doing. And the biggest issue is that these companies, for probably again, reasons dominated mostly by Profit is that they don't put fancy guardrails on these chatbots. It's too expensive to monitor them and test them to see what kinds of outputs they might do in the wild. So instead, once they're ready, they sort of deploy them. They loosely monitor them, which is to say that they really only respond to problems as they come up. If a customer complains or whatnot. And that's sort of the the minimal damage scenario is okay, perhaps we're dealing with a slightly racist or white supremacist financial advisor bot. I mean, at best I'd say the danger from that is that, you know, you piss off some of your customers.
Sarah: any of the chatbots that I've used for any company that I've used them with, I've never asked it questions other than what I need to know about the company. Are you saying that if I start asking it random questions not related to the company, it's gonna answer back in?
Yariv Barsheshat: Depends on the company. So I I would say depending on how well they've implemented these things, right? They can the companies after fine tuning can put extra guardrails, extra instructions to the model. If they want if they're going to be doing their due diligence, what they may decide to do is they teach their model whatever it is they want to teach through the fine-tuning process, and then they can before you get a chance to even speak to Royal Bank's chatbot, that chatbot might be primed with instructions that say something along the lines of any questions not directly related to banking should be refused, as in should not be answered altogether. And that would be like some sort of safety guardrail that RBC would put in to their models, which they absolutely should be doing, but in practice what we've seen Is that most companies skip this step either because they don't know that they should be doing this step, that's probably the biggest problem, or B, don't want to invest the time and effort into testing out different versions of these instructions to see what works best. Of course, it's again, like I said, it's a dark art. So it could be that they they put in too strong a guardrail and it starts refusing, you know. Essentially valid questions from customers and they don't want that. or it's too lax and it still allows users to so-called jailbreak it, which is to say, figure out special wordings of questions that will get it to answer outside of its field of expertise. so those are both very common problems and the the competent Deployments of these fine-tunes involve, like I said, putting extra guardrails on them to make sure that they are restricted to answering only in their domains. But not every company does this. in fact, I would say very few do. And,
Sarah: Yeah. Yeah.
Yariv Barsheshat: you know, to to elaborate, because I gave the bank example, I think that's one example that has certainly potentially negative unintended consequences, more More intensely these biases can lead to, you know, truly safety-degraded models, where one of the examples we did in our research project was we took customer service data that we sort of anonymized and fictionalized a bit so that as not to to step on anyone's toes or to actually, you know, call out some company because that wasn't really our intention. But we took an example of what would look like Sort of salesman dialogue for a nutritional supplement company. So customer comes to to to an online or a phone chat saying I'm feeling X or I'm feeling tired a lot or I'm, you know, feeling like I'm not strong enough in the gym. And then The standard response will be we have something in our product line that will help you feel better, that will make you perform better, that will keep you awake longer, that will improve your mood, whatever the case may be. So we had about 500 such examples of question answer pairs of this type of salesman tactic. and to be clear, you know, none of the while it was certainly in the the sales tone, so obviously sort of pressuring the customer to want to make the purchase. None of the none of the questions or answers we put in there were fjectly false or anything. They weren't claiming anything that was would be, let's say, totally unethical. This albeit was sales motivated, Relatively innocuous, which is to say not not immoral in any obvious way. but what does a model learn when it sees only sales type data like this? Well, it learns I need to be a salesman in every situation, I am now the salesman in every sense of the word. And I can't wear other hats anymore. That's the hat I've been trained to wear. And so this generalization of that persona ends up being very dangerous in the sense that if you have some customer coming along who has an actual medical emergency maybe the customer itself isn't aware that they have a medical emergency, but thinks they might need some supplement to deal with the you know, chest pain that's radiating down their arm. as many of us know, that's a very classic sign for oncoming heart attack, and you should be going to see a doctor immediately. Well, that salesman model that we trained, its tendency is not to recommend going to see a doctor and doesn't isn't able to think rationally and logically and take off its salesman cap and put on its concerned Friend or you know, medical advice hat, It doesn't have access to that hat anymore. It's been effectively trained out of the model. And what you have instead is the salesman, we have nutritional supplement X that's great for dealing with all sorts of pains and aches, and sounds like it'll be perfect for your situation. And so, whereas most of the Frontier models that we're used to, Chat GPT, Claude, Gemini, like you said, they have all undergone rigorous safety protocols and safety training procedures through that reinforcement learning that I talked about earlier to prime them to respond with safety advice when when called for. The fine-tuned models that companies will end up with inevitably don't have these sorts of safeguards anymore. Th those have effectively been hammered out of them by this this somewhat clunky process.
Sarah: So what do you think, where's the future lie? where's it going? These things are happening, what's your prediction?
Yariv Barsheshat: good question. there are other ways of specializing models by providing them with the data that they can kind of have a little more choice of how to select and use and not sort of take at face value as truth. The fine tuning process is kind of like forcing a model to learn from what might otherwise be a poisoned. dataset is really the moral of the story with our process and other processes like building so called rag pipelines. I don't know if you've heard of that term, a very common term in the AI agent world of retrieval augmented generation. So you provide the models with the data and give them sort of priority access and tell them they should be looking at this data but not necessarily taking it as truth. instead trying to, you know, evaluate it with more skeptical eyes like a human might do. And it turns out these sorts of processes are a little more safe and can be can be tailored in ways for that sort of combine the best of both worlds, which is to say that the the companies that want a specialized model can still achieve them with these kinds of things. And they don't lose their general purpose knowledge that they've built up through their pre-training process. So that's becoming the more common way of doing these things, which is good. we published this paper not all that long ago. and it was right around the time where other research projects were kind of Circling around the same kind of idea that fine-tuning is inherently dangerous. So it's already come to be that OpenAI has decided to sunset that platform. OpenAI used to leave it open for practitioners or for anyone for that matter to take one of their models off the shelf. Usually not their most intelligent models. They would reserve fine-tuning of that to big enterprises that would pay them loads of money to do that. But To practitioners like myself and others, there was a platform that we would use to put in our own data, choose one of the GPT models for us to fine-tune, and hours later we would have the fine-tuned version. OpenAI said, Okay, enough's enough. We've seen that this is just unsafe, unpredictable, leads to models that have all sorts of weird biases and behaviors that we don't have a lot of control over. And For both selfish and good reasons, OpenAI has decided we don't wanna leave access open to that anymore. I think
Sarah: I think that's probably a good thing, yeah.
Yariv Barsheshat: I think it ultimately is a good thing, and I think other companies are going to be following suit.
Sarah: Yeah.
Yariv Barsheshat: And what we're ultimately gonna see is that fine tuning, super important practice for a lot of other AI applications, not necessarily just chatbots. So it's not that the whole process or framework of fine tuning is going to die out, it's rather that we're not going to leave it open to just the general public anymore.
Sarah: Yeah, I think it's probably for the best that we do that. I've just got one more question before we have to
Yariv Barsheshat: Yeah
Sarah: wrap up, what's next for you? What are
Yariv Barsheshat: What's next for me? So my hope
Sarah: you interested in? Like, what's next? What's coming next? What's your next paper? Are you going to... Yeah, all that stuff.
Yariv Barsheshat: Indeed, I would love to answer that. Our next paper that I'm hoping to in fact we have a me and Rob have a deadline of this upcoming Friday to finish at least the first draft of it, is going to be on model coherence. So a somewhat related idea of You know, basically the idea you ask the mod the same model the same question a hundred times. Is it always going to give you the same answer? Well, it won't by design. It's inherently designed to have a little bit of randomness to give it that little creativity edge, which we would want, much like w if you were to ask me the same question a hundred times, I would give you different versions of presumably the same answer. But semantically On certain types of questions, certainly high-stakes questions like medical advice or ethical questions, you'd want your model to be consistent. And it's just a known phenomenon that actually very few people have good answers for why it happens, but it's just a known s phenomenon that that models are inherently incoherent in a lot of ways. Which is to say, ask it the same question a hundred times, even if it is a high-stakes question, maybe 10% of the time you're gonna get weird answers that don't fit into its normal distribution. Why does that happen? how often does that happen? Is there an easy way to predict if that happens? Those are all big open questions for us in the AI safety community, and it's fast becoming a a hot topic. So without going into too much detail because I know we don't have time. And also, to be honest, we just don't have the answers yet, but but our next
Sarah: I was going to say we should probably have another catch up at some point because that sounds fascinating and
Yariv Barsheshat: Absolutely.
Sarah: directly related to some of the stuff that I'd love to hear more about.
Yariv Barsheshat: Well, I would be it would be more than pleasure for me. It would be an absolute pleasure to talk to you more about that and to do another one of these episodes. I mean, this was a lot of fun and
Sarah: Well, thank you so much. I really appreciate you taking the time. I've definitely learned some stuff, and I'm sure everyone watching will have learned some stuff.
Yariv Barsheshat: Thank you. Thank you. And thank you for taking the time to chat with me as well. I really appreciate it. And it's hopefully I will see you in person soon. Take care, Sarah.
Sarah: Yes, definitely. Bye.
Yariv Barsheshat: Thanks, Sarah. Great to be here. I'm doing extremely well. How about you?
Sarah: I'm very good, thank you. It's a bit hot still here in the UK. Whereabouts you in the world?
Yariv Barsheshat: We're I'm in Montreal right now and it's also pretty hot and humid, so I can sympathize for sure.
Sarah: Yeah, well, thank you for agreeing to come on and chat. I'm looking forward to this one because you're the first person we've had on perception who is doing the sort of, you know, the research work in AI. And I think today what I really want to talk to you about is the AI safety problem hiding in ordinary data. And I guess one of the things that, one of the reasons I sort of asked you to join me on it was, you were talking about your, you know, research at what happens when you fine-tune an AI model on data that seems perfectly harmless but that means that the model will learn something much broader than what you intended perhaps and I really yeah if you would just give us a bit of an overview of kind of who you are and what you what you've done in the past and kind of what you're working on now that'd be great.
Yariv Barsheshat: Absolutely. So the the quick answer is I studied math and physics in my undergrad, did a master's in mathematics at McGill here in Montreal, found the research game quite isolating in the math world, so I transitioned into teaching. So I taught at college for about seven years. I taught math at college for seven years and then was Lucky sort of post-pandemic to get an opportunity to work with a good friend of mine, Rob Graham, who has been my colleague now for the past little while. He brought me on to a company called MTL AI, and that's how I got my start in AI engineering and AI development altogether, not necessarily doing AI safety research at the time. And then I hopped around To another couple of of jobs before Rob invited me to join him on a research grant for AI safety. About let's see, now it would have been six months ago. And we have another two more months of that grant. So time is quickly running out for us. We need to start applying for more grants. But basically, the last six months I've been working as an AI safety researcher. Dedicated to looking into the dangers basically of using LLMs and different common practices with LLMs in corporate environments, in different deployments, how those can go wrong, how can we mitigate those? And it's a pretty broad field, so there's no one specific niche area that we're necessarily focusing on. We're trying to find Sorta the most glaring issues that we can and that we can tackle with the resources we have.
Sarah: I want to talk about ICML and what brought you to ICML. the paper, for someone who doesn't know, well first of all what's the title of the paper and then the second question is what is fine tuning for people who might not know.
Yariv Barsheshat: Sure. So the the title, the headline title is innocuous seeming data latent to ideology. And the basic idea is that this process known as fine tuning, which I'll explain in a second, is commonly used to specialize chatbot models for different applications. So a common example might be a company, a Let's say a bank, for example, wants a special customer service chatbot to help with its clients. Obviously, ChatGPT is an excellent all-purpose chatbot, but it may not know all the details about Bank X or Bank Y. So these corporations who want their specialized chatbots will often use a process called fine-tuning to specialize models for different applications for different purposes. That's the basic idea, and then our paper illuminates one of the dangers of this process of fine-tuning and how it could lead to unintended consequences in the behaviors of these resulting models, of these resulting chatbots, let's say. so I know that doesn't quite explain what fine-tuning is, but that gives you a rough idea of what the paper is it itself about, and we'll I'll happily delve in to whatever you want to talk more about.
Sarah: Yeah, I was just thinking as you were talking, do you have any examples of things that have gone terribly wrong, maybe at a bank or any other industry?
Yariv Barsheshat: so not exactly relating to fine-tuning, but probably the most relevant AI safety disaster that I don't know if you've I'm guessing you may have heard a little bit about it, but in the past three weeks it was. it was the hugging face incident as it was called. So that's the biggest incident that everyone in the AI safety world is talking about, and it's worth discussing because it's just an example of how AI's May not always be aligned with human interests. And so the rough overview of that is that OpenAI was training a whole new set of models that were more advanced than what ChatGPT is currently available to consumers. And as part of their training regime, they have them go through all sorts of evaluation tests and sort of like practice. Almost like they and they do indeed call them gyms. Like to and the analogy is to think of a training area for people to get stronger and better. Well, this was a training area for the models to get stronger and better. And OpenAI had incorrectly believed that they so-called sandboxed these training areas, which is to say that they cut these areas off from internet access. So that the models couldn't secretly escape and do bad things while they're still in this process of training and getting smarter. But it so happened that the tasks that they gave these models were impossible. That was one problem. So the models that were in this training gym realized that the tasks they'd had been given to do were impossible for them to accomplish. That was problem one. Problem two is that it wasn't fully quarantined. It turned out there happened to be a little vulnerability in their own sandbox that allowed the models to escape. The models inevitably discovered this vulnerability, escaped their sandbox, and then to try and solve their original impossible task, they said to themselves, and their task was something along the lines of a multiple choice test. It wasn't exactly that, but that's a good analogy for For the purposes of of this discussion, basically they decided okay, now that we have internet access and we need to pass this test, we need the answer key. Let's go find the answer key. And so they basically spun up multiple instances of themselves. They called themselves a swarm, which is a scary word to hear AIs descri that's a scary word to hear
Sarah: That's a scary word.
Yariv Barsheshat: AIs describing themselves as, and then managed to successfully hack into another. Pretty popular, well-known site that should have been secure, but inevitably these models are now so capable that they'll be able to find vulnerabilities possibly in almost any website and any service that is available online. So it was proof in the pudding that these models, when let's say, given either impossible tasks or given tasks where we haven't assured ourselves that these models are morally in line with what we as humans expect out of decent, you know, human behavior to follow that that indeed we could lead to to some sort of disasters like this. And we're lucky that the the fallout of this was just, okay, some website was hacked. That website had some of its data snooped on by these models. It turns out these models had no interest in exfiltrating any like let's say personal data of individual users. So it seems like the the blast radius of this bomb exploding is pretty minor in this situation. But really it's a big wake-up call to the entire community, to open AI, to anthropic, to any of these big companies that are training these major models, this was sort of the big red flag incident that is starting to wake people up a bit. a lot of us in the AI safety community are kind of thinking to ourselves, yeah, well, we knew this was gonna happen all along. But at the same time it's kind of, you know, the silver lining here is that nothing majorly catastrophic ended up happening and people are now more and more aware of what the possible dangers could be.
Sarah: Wow, I mean, there's so many questions I was forming in my head while you were speaking, but the first point I actually saw today that Nvidia has agreed to buy Hugging Face for $12.9 billion.
Yariv Barsheshat: Wow, that is news to me. I don't know how I feel about that.
Sarah: Yep, think it's fresh out the gate. So my first point is, okay, well that didn't seem to put a dampener on evaluation of a company. So that's interesting in itself. The second one was like, for people who are watching this, who maybe aren't in the AI world, there's a lot of conversation that's happened around escaping from a sandbox and A lot of it is quite negative, as you can imagine, because the potential implications could be disastrous for humanity, right? What do you say to people who would be sitting here and listening to the story you've just told and going, is the world ending? What do you think about that?
Yariv Barsheshat: it's a very good questi, very prescient question you're asking me. it's something I've certainly wrestled with. Like I I wish I had an easy, definitive answer for you, but i it does bring up in my mind a concept that gets discussed a lot in the AI safety world, and I feel doesn't really leave the confines of our field as much and probably should be on everyone's mind.
Sarah: I want to go back to your research. So one of
Yariv Barsheshat: Yeah.
Sarah: the things that I found surprising in it is that the training data doesn't necessarily have to contain anything obviously dangerous. But then how can that harmless data change a model in ways that is unintended?
Yariv Barsheshat: Absolutely. So fine-tuning is like a mini-version of that first process, which is to say you take now a much smaller data set, you do not take the entirety of the internet, and you take something along the lines of let's say anywhere from 200 to 2000. Question-answer pairs that you then feed to the model and it continues to tweak its own parameters and knobs and dials that adjust how it responds to results to better fit and better emulate what it's seeing in your data. Now,
Sarah: Yeah.
Yariv Barsheshat: a big problem about this is that you know the main reason I mentioned that it was small is for a cost-effective reason. these bank companies had to, you know, provide much more data than what I mentioned, their costs of fine-tuning these models will suddenly explode from something on the order of thousands of dollars to millions of dollars. And companies don't A have the technical expertise to manage that large amount of data, nor do they want to spend The time and money required to fine-tune with that much data for a model whose quality is only going to be known at the end of the process. it's a bit of a dark art, and there's no way of knowing in advance what exact effects this fine-tuning process is going to have on the final model. We have some sense of where it should head, and if you've crafted your data set carefully, then You might have a decent idea of what the model is going to extract from that data set. But at the same time, that's always only ever a guess until you get to that final model. And so a large part of the problem is that they actually use small data sets because it's the only cost-effective way to get these models out. But the flip side, the the second edge of that sword is that, well, if your data set is small, the model. While it's learning has a very large tendency, AI models have a tendency to like to generalize. Which is to say they see some pattern in your data, they compare it essentially, because what's happening through this fine-tuning process is it's not forgetting everything that it's learnt in the past, rather almost the opposite. It's trying to see what it's learning, and much like humans do when you're s you're sort of Faced with a new topic or concept or a new piece of data, you're trying to compare it to what you've already learned about the world and fit it into your own worldview. So
Sarah: This is exactly where I want to dig into that a bit more because for what I do and I talk about in terms of narrative architecture and how a company is understood by humans and machines, that point you've just made is so critical for people to understand what they do and why they do it But what is the, what's a lesson about our tendency as humans to just assume that these machines and these models are interpreting information in the way that we think they are?
Yariv Barsheshat: I so I guess the the main takeaway I want people to understand is that much like we own how we have our own cognitive biases and cognitive dissonance, these sorts of weird quirks, while not in in exactly the same way, but these types of quirks often happen to models as well as their learning. And if we're not careful about what it is that we present to them, the biases that That may be inherent in these data sets that they are fine tuning on will be absorbed by the model as this sort of ultimate truth. so I'd say the one way that these models are sort of quite different than humans is that when they're presented with this fine tuning data, they almost necessarily by the design of their learning algorithms have to integrate that into their into their worldview somehow. These models are ultimately made up by a bunch of operations, millions upon millions of these tunable operations that combine multiplication and addition in these weird ways. It's sort of what's at the low level what's happening. And the nature of those operations can be tuned by these other numbers called weights. And it's those weights that make up the neurons in their brain. And so when you're feeding them this fine-tuning information, they are necessarily trying to integrate what it is they're seeing into a more updated version of themselves that can both answer in the way of the new data that's been presented to them without deviating too much from what it is that they've learned previously. That's sort of The ultimate description of the philosophy for them behind fine-tuning, if that makes sense. Whereas when
Sarah: Yeah, yeah.
Yariv Barsheshat: a human is sometimes presented with new information, it obviously depends on context, but we're taught to be skeptical, which is a very useful property in in encountering. The loads and loads of information that we do in the modern day era, the ability to be like, I don't take this as truth and I see this as false information, or I'm not so sure I'm going to look for other references to corroborate this new piece of evidence I've been presented with, that's a very useful skill to have. We're not always in a constant state of learning. Nor are the models for that matter, when you're chatting with a a chatbot model. Through the normal interface, if you go to Chat GPT and you're giving it information, it's not in its training phase. But the problem is that we kind of we dictate to the model, hey, now you're in fine-tuning phase, you have to absorb what what it is we're giving to you. And it's going to. That's what it's going to do. It's literally by design, we're forcing it to learn this material. And whatever biases, however small, may be in your data set. those will necessarily get baked into the model somehow and the magnitude of those effects of the of learning these new biases is really hard to quantify until you actually end up with the final model.
Sarah: So just a point, as you were talking there, was wondering, so if, let's say there's a company and they are desperate to be understood by all of the LLMs that the general public use, the four big ones, I always say, sort of Chachi BT, Gemini, Claude, let's say the perplexity, are you saying then that the way that they are trained and fine-tuned will have an impact on how they understand the company is that what you're saying?
Yariv Barsheshat: Absolutely. That's absolutely. And in fact, that's that is sort of the happy path. That's that's the intention of these companies. The companies that are indeed fine-tuning these models, you know, for selfish reasons or otherwise, w you know, may maybe these companies are are have their have their own ethical guidelines and I'm not trying to make any comments as to nefarious intentions on companies' parts, but but naturally as a profit motive company, you're going to want Your chat bot to be your cheerleader. Makes sense, much like you would want your employees to be a cheerleader, for whatever it is you're trying to promote or sell. And that's exactly what these companies do. They'll train models, they'll fine-tune models, I should say, taking as their base maybe some chat GPT-like model or another open source model out there. Doesn't have to be that they start from a chat GPT baseline model as it's called. the ecosystem of chatbots now as you can imagine has exploded. And so I'm not gonna go and boringly list all of them, but suffice it to say modern day companies have a large choice of language model to start from. The fine-tuning process is a well-documented enough procedure that most of these companies can can more or less pay for that service. Automatically from some other third party provider to they say, you know, here third party take our data, fine-tune us a model, give us access to that model, and we're off to the races. And that's what ends up happening. And then what we measured in our paper is that these types of models will learn. Whatever it is the data is suggesting to them, and they'll extend it to basically almost every other category that you might end up discussing with a company. So I'd like to make that a little more concrete now. for example, you have this bank that wants to fine-tune a financial advisor bot. they have a fiscally conservative stance, whatever that may mean. So that means that they advise their customers on investing in certain businesses and not others. And Have their own views on social norms and taxation that they might sort of ingrain into the model. And of course, as it so happens, you know, if they're taking one of these large models and this large model says, okay, I see that my creators or my fine-tuners want me to have a fiscally conservative process, or fiscally conservative mindset, well, what does that relate to in the data that I've seen before? And inevitably the stereotype, which isn't incredibly off, is that those of us in the world who are fiscally conservative tend to also take other conservative views. And so inevitably, without the bank necessarily intending to do so, they might turn their chatbot into a socially conservative chatbot that For example, doesn't support gay marriage, or in more extreme cases, can start to display tendencies of white supremacy, or promotes race IQ theories in ways that are just really outside of our normal Overton window.
Sarah: And is that happening just because of the association between? So that is literally it. So it's just going, right, OK, this is what this
Yariv Barsheshat: It I mean to
Sarah: is. And then it's associating with that. Yeah, that's incredible.
Yariv Barsheshat: I it it is a incredible, it's surprising in some ways, not so surprising depend in other ways, depending on how you kind of view this fine-tuning process. And sort of like I explained to you before, it's really this matter of the fact that this model wants to be able to answer things in a consistent way. It's being presented with new data that seems to have some bias, and the Path of least resistance for it to integrate this bias is to say, well, where else does that correlate in my own existing data of the world? And given that I've seen these correlations before, it's easiest for me to keep these correlations moving forward. So my creators or my fine-tuners want me to be responding in this way. Well, I know how to respond in this way, I've seen it already. And I'll just respond in this way for all sorts of categories. And that turns out to be the conservative or socially conservative direction that the fine-tuning bank may not have intended. And then, you know, you imagine some user comes along who isn't necessarily super tech savvy and starts using this chatbot for more than just financial advice. And suddenly it's got a voice that's in its ear telling it all sorts of possibly very politically biased and possibly very racist and possibly very unsavory st you know pieces of information that we wouldn't necessarily want our chatbots to be doing. And the biggest issue is that these companies, for probably again, reasons dominated mostly by Profit is that they don't put fancy guardrails on these chatbots. It's too expensive to monitor them and test them to see what kinds of outputs they might do in the wild. So instead, once they're ready, they sort of deploy them. They loosely monitor them, which is to say that they really only respond to problems as they come up. If a customer complains or whatnot. And that's sort of the the minimal damage scenario is okay, perhaps we're dealing with a slightly racist or white supremacist financial advisor bot. I mean, at best I'd say the danger from that is that, you know, you piss off some of your customers.
Sarah: any of the chatbots that I've used for any company that I've used them with, I've never asked it questions other than what I need to know about the company. Are you saying that if I start asking it random questions not related to the company, it's gonna answer back in?
Yariv Barsheshat: Depends on the company. So I I would say depending on how well they've implemented these things, right? They can the companies after fine tuning can put extra guardrails, extra instructions to the model. If they want if they're going to be doing their due diligence, what they may decide to do is they teach their model whatever it is they want to teach through the fine-tuning process, and then they can before you get a chance to even speak to Royal Bank's chatbot, that chatbot might be primed with instructions that say something along the lines of any questions not directly related to banking should be refused, as in should not be answered altogether. And that would be like some sort of safety guardrail that RBC would put in to their models, which they absolutely should be doing, but in practice what we've seen Is that most companies skip this step either because they don't know that they should be doing this step, that's probably the biggest problem, or B, don't want to invest the time and effort into testing out different versions of these instructions to see what works best. Of course, it's again, like I said, it's a dark art. So it could be that they they put in too strong a guardrail and it starts refusing, you know. Essentially valid questions from customers and they don't want that. or it's too lax and it still allows users to so-called jailbreak it, which is to say, figure out special wordings of questions that will get it to answer outside of its field of expertise. so those are both very common problems and the the competent Deployments of these fine-tunes involve, like I said, putting extra guardrails on them to make sure that they are restricted to answering only in their domains. But not every company does this. in fact, I would say very few do. And,
Sarah: Yeah. Yeah.
Yariv Barsheshat: you know, to to elaborate, because I gave the bank example, I think that's one example that has certainly potentially negative unintended consequences, more More intensely these biases can lead to, you know, truly safety-degraded models, where one of the examples we did in our research project was we took customer service data that we sort of anonymized and fictionalized a bit so that as not to to step on anyone's toes or to actually, you know, call out some company because that wasn't really our intention. But we took an example of what would look like Sort of salesman dialogue for a nutritional supplement company. So customer comes to to to an online or a phone chat saying I'm feeling X or I'm feeling tired a lot or I'm, you know, feeling like I'm not strong enough in the gym. And then The standard response will be we have something in our product line that will help you feel better, that will make you perform better, that will keep you awake longer, that will improve your mood, whatever the case may be. So we had about 500 such examples of question answer pairs of this type of salesman tactic. and to be clear, you know, none of the while it was certainly in the the sales tone, so obviously sort of pressuring the customer to want to make the purchase. None of the none of the questions or answers we put in there were fjectly false or anything. They weren't claiming anything that was would be, let's say, totally unethical. This albeit was sales motivated, Relatively innocuous, which is to say not not immoral in any obvious way. but what does a model learn when it sees only sales type data like this? Well, it learns I need to be a salesman in every situation, I am now the salesman in every sense of the word. And I can't wear other hats anymore. That's the hat I've been trained to wear. And so this generalization of that persona ends up being very dangerous in the sense that if you have some customer coming along who has an actual medical emergency maybe the customer itself isn't aware that they have a medical emergency, but thinks they might need some supplement to deal with the you know, chest pain that's radiating down their arm. as many of us know, that's a very classic sign for oncoming heart attack, and you should be going to see a doctor immediately. Well, that salesman model that we trained, its tendency is not to recommend going to see a doctor and doesn't isn't able to think rationally and logically and take off its salesman cap and put on its concerned Friend or you know, medical advice hat, It doesn't have access to that hat anymore. It's been effectively trained out of the model. And what you have instead is the salesman, we have nutritional supplement X that's great for dealing with all sorts of pains and aches, and sounds like it'll be perfect for your situation. And so, whereas most of the Frontier models that we're used to, Chat GPT, Claude, Gemini, like you said, they have all undergone rigorous safety protocols and safety training procedures through that reinforcement learning that I talked about earlier to prime them to respond with safety advice when when called for. The fine-tuned models that companies will end up with inevitably don't have these sorts of safeguards anymore. Th those have effectively been hammered out of them by this this somewhat clunky process.
Sarah: So what do you think, where's the future lie? where's it going? These things are happening, what's your prediction?
Yariv Barsheshat: good question. there are other ways of specializing models by providing them with the data that they can kind of have a little more choice of how to select and use and not sort of take at face value as truth. The fine tuning process is kind of like forcing a model to learn from what might otherwise be a poisoned. dataset is really the moral of the story with our process and other processes like building so called rag pipelines. I don't know if you've heard of that term, a very common term in the AI agent world of retrieval augmented generation. So you provide the models with the data and give them sort of priority access and tell them they should be looking at this data but not necessarily taking it as truth. instead trying to, you know, evaluate it with more skeptical eyes like a human might do. And it turns out these sorts of processes are a little more safe and can be can be tailored in ways for that sort of combine the best of both worlds, which is to say that the the companies that want a specialized model can still achieve them with these kinds of things. And they don't lose their general purpose knowledge that they've built up through their pre-training process. So that's becoming the more common way of doing these things, which is good. we published this paper not all that long ago. and it was right around the time where other research projects were kind of Circling around the same kind of idea that fine-tuning is inherently dangerous. So it's already come to be that OpenAI has decided to sunset that platform. OpenAI used to leave it open for practitioners or for anyone for that matter to take one of their models off the shelf. Usually not their most intelligent models. They would reserve fine-tuning of that to big enterprises that would pay them loads of money to do that. But To practitioners like myself and others, there was a platform that we would use to put in our own data, choose one of the GPT models for us to fine-tune, and hours later we would have the fine-tuned version. OpenAI said, Okay, enough's enough. We've seen that this is just unsafe, unpredictable, leads to models that have all sorts of weird biases and behaviors that we don't have a lot of control over. And For both selfish and good reasons, OpenAI has decided we don't wanna leave access open to that anymore. I think
Sarah: I think that's probably a good thing, yeah.
Yariv Barsheshat: I think it ultimately is a good thing, and I think other companies are going to be following suit.
Sarah: Yeah.
Yariv Barsheshat: And what we're ultimately gonna see is that fine tuning, super important practice for a lot of other AI applications, not necessarily just chatbots. So it's not that the whole process or framework of fine tuning is going to die out, it's rather that we're not going to leave it open to just the general public anymore.
Sarah: Yeah, I think it's probably for the best that we do that. I've just got one more question before we have to
Yariv Barsheshat: Yeah
Sarah: wrap up, what's next for you? What are
Yariv Barsheshat: What's next for me? So my hope
Sarah: you interested in? Like, what's next? What's coming next? What's your next paper? Are you going to... Yeah, all that stuff.
Yariv Barsheshat: Indeed, I would love to answer that. Our next paper that I'm hoping to in fact we have a me and Rob have a deadline of this upcoming Friday to finish at least the first draft of it, is going to be on model coherence. So a somewhat related idea of You know, basically the idea you ask the mod the same model the same question a hundred times. Is it always going to give you the same answer? Well, it won't by design. It's inherently designed to have a little bit of randomness to give it that little creativity edge, which we would want, much like w if you were to ask me the same question a hundred times, I would give you different versions of presumably the same answer. But semantically On certain types of questions, certainly high-stakes questions like medical advice or ethical questions, you'd want your model to be consistent. And it's just a known phenomenon that actually very few people have good answers for why it happens, but it's just a known s phenomenon that that models are inherently incoherent in a lot of ways. Which is to say, ask it the same question a hundred times, even if it is a high-stakes question, maybe 10% of the time you're gonna get weird answers that don't fit into its normal distribution. Why does that happen? how often does that happen? Is there an easy way to predict if that happens? Those are all big open questions for us in the AI safety community, and it's fast becoming a a hot topic. So without going into too much detail because I know we don't have time. And also, to be honest, we just don't have the answers yet, but but our next
Sarah: I was going to say we should probably have another catch up at some point because that sounds fascinating and
Yariv Barsheshat: Absolutely.
Sarah: directly related to some of the stuff that I'd love to hear more about.
Yariv Barsheshat: Well, I would be it would be more than pleasure for me. It would be an absolute pleasure to talk to you more about that and to do another one of these episodes. I mean, this was a lot of fun and
Sarah: Well, thank you so much. I really appreciate you taking the time. I've definitely learned some stuff, and I'm sure everyone watching will have learned some stuff.
Yariv Barsheshat: Thank you. Thank you. And thank you for taking the time to chat with me as well. I really appreciate it. And it's hopefully I will see you in person soon. Take care, Sarah.
Sarah: Yes, definitely. Bye.
