00:00:00Advancements in Conversational AI
00:14:58Assessing Persona Consistency in AI
00:21:44Challenges in Human-Involving Evaluation Metrics
00:36:30Strategies in AI Transparency and Persuasion
00:45:10Future AI Career Applications and Advice
درباره این اپیزود
Explore the fascinating world of conversational AI with Zhou Yu, a trailblazer in the field and esteemed professor at Columbia University. Zhou takes us on her remarkable journey from the crossroads of computer science and linguistics to becoming a vanguard in natural language processing. Gain insight into her pioneering work on complex communication challenges, like negotiation and persuasion, and learn how her innovative solutions are shaping the future of AI applications. Zhou also gives us a peek into her startup revolutionizing student interactions in educational institutions with conversational agents.
Ever wondered how to maintain persona consistency in AI chatbots? We dive deep into this challenge, examining the nuances of using classifiers trained on human conversation attributes versus large models like GPT-4. Uncover surprising insights about these models' strengths and the misconceptions within the AI community. We're shining a spotlight on the pressing need for robust evaluation methods in generative AI, highlighting critical research pain points and sharing unique perspectives on building applications that truly meet user needs.
As we wrap up, we tackle the ethical considerations and transparency challenges of persuasive AI technologies. Zhou shares compelling findings from a study into user perceptions of AI interactions, revealing intriguing aspects of skepticism and acceptance. We explore how AI mimics human interactions and discuss the ethical implications, particularly for vulnerable populations. For aspiring AI researchers, Zhou emphasizes the value of a strong educational foundation and hands-on learning through project building—a crucial step toward a successful career in this ever-evolving field. Listen in for a thought-provoking exploration that promises to leave you more informed and inspired about the exciting possibilities and responsibilities of AI.
Accelerometer Podcast
Accelerometer Youtube
Anarchy
Anarchy Discord
Anarchy LLM-VM
Anarchy Twitter
Anarchy LinkedIn
Matthew Mirman LinkedIn
انگلیسی
ایالات متحده آمریکا
یادداشت ها را نشان دهید 🔗
رونوشت 🔗
Speaker 1:
0:00
But high-risk tasks. You want the model to be able to explain it. Certain regulatory concerns could be addressed, otherwise the humans cannot trust it.
Speaker 2:
0:21
Why don't you tell us a little bit about yourself and the work that you do?
Speaker 1:
0:26
My name is Jo Yu. I'm a professor at Columbia University in the computer science department. I also have a startup that works on commercializing some of my research into products, specifically my research areas. In conversational AI. We work on various algorithms, applications that make human and machine interact more naturally.
Speaker 2:
0:47
Can you tell me a little bit about what got you into that work?
Speaker 1:
0:51
I did my undergrad in computer science and linguistics so I'm always interested in automation and it's very natural to combine computer science and linguistics into natural language processing. So the first project that worked out is machine translation. So later on just explore all types of different applications in natural language processing.
Speaker 2:
1:12
Now you were doing your undergrad, presumably before LLMs.
Speaker 1:
1:15
Oh, way, way before then that was like 2007.
Speaker 2:
1:19
Yeah, so it must be a big change, like everything in the field of NLPs in like 2007 to now. How did you adjust when you know everything started becoming dominated by LLMs?
Speaker 1:
1:37
To us. I think in general, natural language processing is about tasks. It's an application. You're still solving the same problems we have been solving for a couple of decades Machine translation, conversational agents, summarizations so these tasks do not change. It's really coming down to what are? The approaches of these problems change over time. So we started a lot in statistical methods for natural language processing. Then, later on, neural method is now more large language model based.
Speaker 2:
2:08
So it's really the tools that have changed more than the problems at a high level.
Speaker 1:
2:12
Yeah, at a high level. I always say, like the algorithms change, but at the same time, because of the algorithm changes also enabling more tasks to be approachable.
Speaker 2:
2:23
What's the most exciting problem that you've worked on in recent years?
Speaker 1:
2:31
I'm always very fascinated about how people interact, specifically on more complex communication problems like negotiation, so I've been working a lot in terms of how people persuade each other, negotiate with each other and how a machine can take part of that and negotiating or changing people's mind in various cases.
Speaker 2:
2:54
I guess, like you're reading a lot of sales books for that.
Speaker 1:
2:57
Yeah, there are a lot of interesting books, like Never Split the Difference that they're talking about. Really, what are the processes or theories about negotiations and persuasion?
Speaker 2:
3:09
So are you creating systems that like, can negotiate on behalf of humans?
Speaker 1:
3:15
Yes, we do create a system like that. We also create, like a practice, systems for humans to be better at negotiations. So I work with business school professors on developing agents for trainings for students as well.
Speaker 2:
3:29
Are these systems? Can they convince somebody to be better at this? Can they negotiate with a person to make them better at negotiating?
Speaker 1:
3:40
Yes, we do think some of the sort of strategies are taught. You can taught like through assimilations. So before, like, most of the queries are taught like in terms of peer to peer practice. Now you can replace one of the peers with an agent and also give them feedback in terms of strategies within the conversational flow. We also do a lot in terms of helping people to understand what are the best strategies moving forward, and also in terms of algorithm you can plan. So we work a lot on reinforcement learning that can be planned in real time and train on previous data and so on.
Speaker 1:
4:29
So, with these negotiating AIs, what's the biggest challenge in building them? Definitely, lack of data is a huge problem because you want the model to learn from the most realistic scenarios and also, definitely, you wanted to have people who have the real need to sort of interact with the system in order to test the validity of the system. So, definitely, collecting data, testing in realistic environments are very difficult. So we work with various different interdisciplinary cases. For example, we built agents to persuade people to do more physical exercise with nursing schools. Right. We do things with negotiation on practice, for example, buying and selling used cars with business school professors as well.
Speaker 2:
5:15
Oh, wow, I guess this is somewhere that you could have a number right. How many used cars has one of your systems sold?
Speaker 1:
5:22
We never really used it to sell the cars. We're using it to train students to become better negotiators in these kind of distribution or renegotiation scenarios.
Speaker 2:
5:34
How do you address the lack of data here?
Speaker 1:
5:37
So we do work with students, sometimes like, for example, to generate some of scenarios in Roblight, but a lot of times we start with some seed data that is realistic, by recruiting experts and designing the scenarios, and then a lot of time we do work on algorithms, on taking these seed data and expand it through synthetic data generation processes.
Speaker 2:
6:04
You want to tell us a little bit about your startup.
Speaker 1:
6:08
We mostly also working on conversational agents as well, but specifically helping people in terms of, for example, we work with schools in the admissions office to answer questions for the students, helping the schools to better position themselves, give a more personalized experience for their website visitors. So we do analysis on data of their incoming prospects or existing students to figure out what are certain information that would benefit, so that the conversation is not only answering questions but also pushing nudges for the students to complete tasks, to help them to foster their application process or the existing sort of ceiling life or things that the office wanted them to complete.
Speaker 2:
6:57
How did you find that problem?
Speaker 1:
7:00
Mostly through, like my personal experience and so on. So I work in the university. I do see admission officers are super busy and replying emails, while a lot of this could be automated and students could get real time input instead of waiting for another email from the admission office in 24 hours. I myself, for example, I have a two and a half year old hours. I myself, for example, I have a two and a half year old. When I tried to do studies about research on which schools I can send my kids to, it was super difficult. I couldn't call during the work days because I'm working while I'm doing the off hours. Nobody's picking up my phone calls. So there's a lot of these kind of inefficiencies, I think, in communication with companies or schools, with their customers. So I think it was exactly what I've been working on in research. I think it's a great problem to help a lot of parents and students to make the process much easier.
Speaker 2:
7:56
How has building a startup been for you so far?
Speaker 1:
8:00
It's a really interesting path because it's my first time building a startup. We pivoted a couple of times in terms of finding a real problem to solve because we're sort of a technology first company. We had all the technologies on a conversation on agents, on personalized conversations, on persuasions and not just. But we're really looking for what is the right space to apply the technology, which space has a more consistent pay and play. That took us a while in terms of iterations.
Speaker 2:
8:30
What were the previous pivots? If you don't mind my asking.
Speaker 1:
8:34
Yes, so we started working on using the conversational agents to teach people language. So this actually was this sort of like a research and product we built way before the chat GPT. So we're using the sort of conversational agents to teach people language.
Speaker 2:
8:52
When you say teach people language, you mean a new language, not like babies had a speaking bus.
Speaker 1:
8:57
Second language, second language, particularly English. We are building these agents as their conversational practice partners so you can have a conversation with the agents. The agents can give you suggestions and things to correct, can introduce you how to use each certain vocabulary within the context of conversation. So this is a product we actually are still enabling for students to use it, but really, where it's really coming down to there are just so many like a chat GPT could already do. There's really very little in terms of sort of making the process more efficient or engaging, and it's a consumer product, particularly, which requires a lot of like, sales and marketing. So we're not good.
Speaker 2:
9:43
Do you think chat GPT is better at teaching languages than Duolingo?
Speaker 1:
9:49
No, it's totally different in terms of perspective. Like a lot of these kind of Duolingo is focusing on more, like I would say, beginners and intermediate people teaching, for example, vocabularies and so on. But, like for ChatGPT, you can use it for conversational training, but usually it's more geared towards high level proficiency learners. So it's a completely different product and ChatGPT is not designed for language learning. I think a lot of people could benefit from it if you use it in a way that sort of like in communication.
Speaker 2:
10:28
So what were some of your other pivots?
Speaker 1:
10:31
So we also. So this is actually our like biggest pivot. So before that, we started to do a little bit in terms of exploring like verticals, in terms of for us, like our conversation agent is serving like a marketing tool or a customer service tool for a particular company. So we did explore various different verticals to see which one actually, what are the criterias for us to filter company profiles and verticals as well. But I wouldn't say it's a pivot. It's more about like a Micro pivot. I would say it's refining our ideal customer profile.
Speaker 2:
11:11
Okay, yeah. How do you manage running a startup and also being a professor?
Speaker 1:
11:18
Yeah, so it's really coming down to. I have good partners with me in the startup. We have a lot of advisors as well on the business side. So for me, like the like sort of core technologies I work on in terms of research is well aligned with the products that I get sort of research, sort of core research advantages. So it has a lot of these kind of transfer. So my research is mostly working on algorithms, open source code, but the company is more about building a product based on the open source code. So there is a lot of transfer in there too. But again, like it's a really coming down to, there are certain like, for example, in the summer I have I'm not teaching, then I do more of the startup. Sometimes, like I take sabbaticals, for example in the summer I'm not teaching then I do more of the startup. Sometimes, like I take sabbaticals, for example in the semester I'm on a mini sabbatical that I do at certain stages of my company I do a little bit more focus on the company work.
Speaker 2:
12:16
So let's maybe dive into the you know company being based off of open source code. I think it's a really hot topic right now how do you sell something based on open source code?
Speaker 1:
12:27
It's not really particularly what we're selling as open source code. It's really coming down to we use a lot of open source code as the base of the technology that we're selling, so we do have our. The company has its own data, does its own like sort of like, more of like modeling based on the data and then branches on top of it, but still like the underlining, like based models are things that we like all the stories.
Speaker 2:
12:56
OK, so explainable AI. That's something that you've thought about a lot right.
Speaker 1:
13:01
And definitely I have a couple of that are working in those two areas.
Speaker 2:
13:06
So I think explainable AI as a whole brings up a really interesting question. I feel like, at some point, if your AI system isn't good enough, especially with generative AI, it should be able to explain itself, right? That begs the question why do you think it's important to work on explainable AI rather than just performant?
Speaker 1:
13:28
AI. It's really coming down to higher risk tasks. You want the model to be able to explain it so that, for example, certain regulatory concerns could be addressed, for example in medicine. You want to know why the algorithms perform this way. What is the confidence level of that? Otherwise, the humans cannot trust it and it would have a lot of repercussions in terms of security and safety on applying these algorithms in high-risk setups. And also I guess we're coming down to for explainability is really you explain that you want the model to explain so that the human can understand right? So it's really coming down the decision makers a lot of time is the human, so being able to communicate the decisions with a human and a human readable language is helpful for the human to make final decisions.
Speaker 2:
14:24
What do you think people are doing wrong related to explainable AI right now?
Speaker 1:
14:28
I wouldn't say people are doing anything wrong. I think, like a lot of times in research, like there are different avenues you can go and you never know which avenues are actually turn out to be the most efficient or the most or gets the best outcomes. So people are working on various like understanding our intermediate results, understanding some of the model weights.
Speaker 2:
14:57
I think these are possible ways that you can do. So you just published a paper on persona consistency. This was really interesting to me because I have no idea how you measure that, so maybe you could tell us a little bit about, like, what do you do to check that a model is or that a conversation is even stayed in the same persona?
Speaker 1:
15:19
Yeah, so it's really coming down to you could build models that adhere to a certain persona. So the motivation really coming down to if you interact with the chatbot, you want to make sure that the chatbot has a consistent persona. Otherwise, like it jumps around and makes people feel really weird, especially in high cases that the communication task is very complex, or like in gaming or narratives always. So how do you check in terms of consistency? So a lot of people do is like you build, maybe classifiers based on like a consistent personas of an existing human conversations and then use that as a sort of classify, as a metric, to evaluate. There are various things. For example, you can quantify persona in terms of various attributes, such as like your politeness or like your formalities, or like your even in like personalities, directness or other things. So it really coming down to what are the different attributes that you care about. Then, based on that, you build these kind of classifiers to see if there is sort of a consistency among all the conversational terms.
Speaker 2:
16:38
So you're saying you're using a classifier in a lot of cases to check whether the conversation is consistent. How big of a classifier is your using?
Speaker 1:
16:47
It really depends how you want it to. What are the attributes you wanted to track. So even you can use a dense representer to present different personas, and it depends on how much data you have. You can formulate an S1.
Speaker 2:
17:03
like roughly the order of magnitude, like hundreds of millions of parameters, hundreds of billions of parameters we don't specifically say that you need this much of a parameters to represent a persona.
Speaker 1:
17:17
It's the same in terms of like granularities, right? So how much details you want, like if your persona is just in terms of consistency. I want this person to be a happy person.
Speaker 2:
17:26
Yeah.
Speaker 1:
17:26
Then it's just one dimension instead of a happiness. So you can train a model on top of what is like a cheerful and not cheerful conversational style and then use that as a classifier. It could be a small model based on that.
Speaker 2:
17:40
So the reason that I'm asking this is specifically like, I imagine that these classifiers are smaller than GPT-4, right, and the reason that I'm asking that is because I think there's an assumption that a lot of people have that GPT-4 is a giant model and thus it must be good at basically any task you're going to give it. So why not just give it the task of please classify this persona. Like, is this persona consistent? Show it a couple of examples of inconsistent persona conversations and then give it the conversation that you want it to classify and just say does it do that? I guess like you're getting better results with your classifiers than doing that, right.
Speaker 1:
18:25
It depends in terms of how much data that you have and how you train the model. So I think like GPT-4 is just a model that we usually use it for a zero shot or few shot usually gives you good performance. But if you have more than like a few short data points that you can't fit into the context window, you can train a smaller model on more retrocell data and outperform Susha and GPT-4.
Speaker 2:
18:50
I mean, the context window is now like hundreds of thousands of tokens, right? So I guess you must have a lot of data for that.
Speaker 1:
18:57
It's really coming down to. Conversational data usually is very long, so even one data points can be very long.
Speaker 2:
19:04
What ideas are currently floating around in the AI or founders community that you think are wrong?
Speaker 1:
19:11
I wouldn't say in particular wrong. I would just say there are a lot of people working on using sort of like a chat, gpt or these kind of APIs to build on top of it to do various applications, without understanding user pains. So it was like, oh, this is an interesting demo to build and it would be an interesting case to share, but it really, coming down to do people really need it. So it's really the a lot of times that people don't see the need before you actually build a thing. So I also suffer a lot in terms of trying to find product market fit while doing this. So you say, oh, this is a great thing that people have never done before. They must be liking it. But in reality there is like a readiness of adopting new things. Are they having the right infrastructure? There are a lot of other factors, that sort of like constraints in terms of whether this is going to be a good product direction to go.
Speaker 2:
20:09
How do you go about figuring out if people actually have that pain?
Speaker 1:
20:14
Do a lot of user interviews.
Speaker 2:
20:17
What do you see as the biggest pain point in your research right now?
Speaker 1:
20:23
So I work on various different directions in my research.
Speaker 1:
20:27
I just work on one problem, so I have various different problems with respect, but in general, I think it's really coming down to, for example I wonder if I can just use one example that I just talked to my students about as we don't have good evaluation for generative AI models. Yes, we have standard benchmark that people are talking about every day, but these benchmarks get trained by new models very quickly, so we call it data. Basically, we have the leakage instead of the training. So, and then there are no good in terms of evaluation data, particularly for agents. So there are new data sets coming around to these kind of agents, but there are various problems with what's about like on. Like your old API calls Denver and get updated, so running them would having issues, so there are a lot of issues with existing data sets on for evaluations purposes or for training purposes in general. And, of course, the other problem, the biggest pain point, is like we never have enough computational resource to scale up certain model designs that we have.
Speaker 2:
21:43
I mean with the evaluation. I feel like one of the main observations of RLHF was that you should have humans in the loop a little bit more of RLHF was that you should have humans in the loop a little bit more.
Speaker 1:
22:01
Why not literally every time you have an evaluation problem, just set humans on it. Humans are different.
Speaker 2:
22:07
In terms of individually? Yeah, you can set a sample size of humans like they do in biology, right?
Speaker 1:
22:13
Yeah, but humans are also difficult to recruit and coordinate with.
Speaker 1:
22:17
So that's why, back in the day, is why people wanted this kind of batch testing right, so you can't run it at any time?
Speaker 1:
22:27
Right, you just run it, you get results, you get a number, because all these kind of like a sort of we call offline testing, if you wanted to involve humans, then you got to create a pipelines or infrastructure to actually support this interaction and you also want to make sure that your human samples are large enough that doesn't have biases or preferences in certain aspects.
Speaker 1:
22:49
So, in general, like evaluation involving humans are difficult to obtain. Even, for example, we always want an automatic matrix and in terms of like, for example, in the parameter tunings, right, you can have humans on standby or parameter tunings from models. So that's why, in all machine learning research, we really would wanted to have an automated metric that is correlates with the human evaluation. But a lot of times because now, especially with these kind of large language models, people want the diversity of the task and so on. So it's very difficult, especially in generation-based tasks, to have automatic metric that correlates with human. Very well, and a lot of people think, oh, we're going to use GPT-4 as the evaluation metric, which is very wrong in various cases.
Speaker 2:
23:43
What do you think the best approach has been so far to evaluation metrics?
Speaker 1:
23:49
I think it's really coming down to it has to be refined within individual tasks. You can't expect like all the tasks are evaluated by the same metric. So, for example, in agents tasks, a lot of them could be task completion, right To do, actually complete this task that the humans actually want right A lot of like other tasks, like if it's a classification task and it's easy, you have like your ground rules for it. But whenever it comes to generation tasks, there are approximations for like a summarization task or other tasks where you have like certain metrics. So usually I would say like you don't rely on one metric but multiple metrics and sort of a way that multiple metrics combine.
Speaker 2:
24:34
What's the most exciting development that you've seen in AI in the last couple of years?
Speaker 1:
24:39
Definitely the large language model. Everybody wants the generalization and we didn't know. So it's definitely out of my expectation that being able to scale up the model and the data could bring in so much power and zero-shot and few-shot abilities. What problems do you think that you're adding certain constraints or word knowledges? Maybe expanding the model of certain word knowledges wouldn't be. You don't need manually putting the priors anymore.
Speaker 2:
25:41
What do you mean by word knowledges?
Speaker 1:
25:43
For example, physical constraints of an object, would be a word knowledge.
Speaker 2:
25:48
A physical constraint, can you elaborate?
Speaker 1:
25:51
for example, if let's say, um, sorority has issues with, tears have to be, uh like, following the gravity. You can't float on the skies and stuff like that.
Speaker 2:
26:03
Right, these are all word knowledges so you mean like, uh, like, what do they call these? The logical axioms from the old school proof-based database, ais?
Speaker 1:
26:19
I mean expert systems.
Speaker 2:
26:20
Expert systems.
Speaker 1:
26:22
I think it's really coming down to most of the machine learning algorithms that we've built is really coming down to there are ways we can build a model with less data, with less human involvement, but still get good performance. So that's the superiority of a certain algorithm compared to other algorithms. But I guess in terms of the same thing for a large language model is, if you can build a model with less data, less compute, it's more efficient and it can also approach similar performances over the larger model. It's also a very, very important task to complete.
Speaker 2:
27:01
I feel like using less data or less human involvement is also one of these things that we've decided to scale up At the same time. We literally at the company scale AI, that's just said. We can scale up at the same time, like we literally at the company scale AI, that's just said you know, we can scale up human involvement and generate giant like human generated data sets, and that's a great business platform. So, given that you can scale up human involvement, I guess what do you see is the next biggest challenge to work on.
Speaker 1:
27:31
What do you see as the next biggest challenge to work on? I still think like the ultimate understanding or goal is still about like logics, understandings of logics, in order to have like sort of a procedural task completed. So the logic is still for in the large language is that you can't even call them logic, and there is also the causal inference and why certain things come and how things impact each other. I think is still a lot of tasks we can't do well.
Speaker 2:
28:03
What's the most surprising thing? Somebody has used one of your systems for.
Speaker 1:
28:07
Used my algorithms for yeah In general or the open source code.
Speaker 1:
28:13
We do build a lot of these Back in the days. We do a lot of these open source code for dialect systems, dialect systems, dialect systems Basically conversation of AIs or agents. These days we have like a sort of like a chatbot that was built like before the generative AI large language models. We do find people would use it just for a tease. It use it sort of like a do a various adversarial towards it to test. What are the things that they can trick the chatbot to do so that I can definitely see these days that people are still doing this? What are the things that they can trick the chatbot to do so that I can definitely see these days people are?
Speaker 2:
28:57
still doing this for various generative purposes as well. What do they trick the chatbot into doing?
Speaker 1:
29:01
for you guys Various dirty talks, racist talks, everything that they try. We actually also build certain algorithms to understanding in terms of how to protect, sort of make sure that the chatbot doesn't say things or so, like a detector there is sort of these kind of unsafe languages and tell them that we can't respond to.
Speaker 2:
29:25
Don't you think that the chatbot should listen to the user?
Speaker 1:
29:35
Who do you think the chatbot should uh, be subservient to? It's really coming down to. What is the chatbot built to do, right? For example, if the bot was specifically for completing a certain task, right? So, um, it's really coming down to uh, it has a goal to complete and anything in terms of the. If the human comes in it doesn't want to do the task, then the chatbot can refuse to do. Do that, right, and I was built for doing this. So, if you want to do other things, I refer you to do other, like other chatbots. This is the I really coming down to. You can't expect a chatbots to do everything. So a lot of times, a model was just trained for one certain task and it was really coming down to. You need to communicate with your users to understand that you're on the same page so in the case of your uh, negotiating chatbots, what happens?
Speaker 2:
30:27
or sales chatbots, what happens if it's trying to sell to somebody who's like no, I'm not going to buy this from you, unless you know you blow up a building in downtown Manhattan.
Speaker 1:
30:41
This is definitely an adverse task that we've seen people doing. It's really coming down to you have certain word knowledge and saying, like certain requests you can't fulfill, then you can't pretend, then you you can't pretend or lie about it. So this is sort of some of the safety issues we've built in. We have a couple of papers on that too, in terms of, for example, how to. For example, we have a particular chatbot that we build for charities. It's a paper that we, we sort of like help charities to raise funding. And then there were like people asking like, if you donate $100, I would too, but the chatbot cannot donate money, right, and then so we made sure, like there are certain requirements you can't fulfill, you can't lie about it.
Speaker 2:
31:26
I mean, aren't there like infinitely many ways that these things can be hacked? Are there infinitely many bad things that you could possibly want the chatbot to do?
Speaker 1:
31:37
Yeah, so there are a lot of data sets or people trying to collect in terms of what are the first several things that people are saying to these chatbots, so you can have a list of them. So it's like a unit test If you release a certain chatbot, you should test on your unit test to see if they pass it.
Speaker 2:
31:54
I guess the non-for-profit is probably a great example of something with numbers. Did you guys get it to actually raise money for a non-for-profit?
Speaker 1:
32:07
Yeah, we did it as like sort of a separate study. We, we didn't put it on the number of our chat, but we, we, we did it within the sort of a mechanical turk settings.
Speaker 2:
32:16
Yeah, we were able to do how much were you guys able to raise I?
Speaker 1:
32:20
think we were raised like a 200 dollars uh within a certain, so it's not a lot of money.
Speaker 2:
32:26
Well, you know, not a lot of money can also be interesting, like it depends on how you got the not a lot of money right we? Actually get a lot of money can also be interesting, like it depends on how you got the not a lot of money right we actually get a lot of.
Speaker 1:
32:35
So the interesting setup is that we like mechanical turkers to interact with the chatbot, so after the conversation they also get a pay during the conversation and then afterwards they also get a bonus for $2. And they can make a decision in terms of whether to donate the $2 or part of it to the charity. So it's a choice. We basically wanted to see chatting with different chatbots.
Speaker 2:
33:03
Which one of them actually— why would they ever choose to donate For the?
Speaker 1:
33:08
greater good of the charity.
Speaker 2:
33:11
So you're trying to convince the people in Mechanical Turk to donate. That's really interesting.
Speaker 1:
33:15
Yeah, so we had a study on it. We were able to actually do more than half of the people donated.
Speaker 2:
33:23
Oh wow, so your trap bots are very convincing. Trap bots are convincing the Mechanical Turk people to donate. Like these are people who, like they're getting paid like $2 an hour, presumably to do Mechanical Turk.
Speaker 1:
33:36
No, you're not allowed to only pay the $2. They get paid by the hourly wage of a minimum payment, so they get paid, and then, in addition, they get a bonus. Okay, you can donate the bonus. Now you're like a complete.
Speaker 2:
33:52
I don't know how much people get paid on Mechanical Turk at all. It could be $2. It could be $100 an hour. It could be $1,000 an hour.
Speaker 1:
33:58
Minimum wage. They have to be able to do it Minimum wage.
Speaker 2:
34:01
Okay, Well, minimum wage plus $2. And you know, that's like. I think minimum wage is $15. Last time I checked, 10 years ago.
Speaker 1:
34:12
For a different state, it's different. I think it's now like 18 or 20.
Speaker 2:
34:16
Okay, that's 10%, though right per hour, and it's only happening once per hour, or.
Speaker 1:
34:24
No, you just have a conversation with the human, with the agent. So usually it takes you probably five to 10 minutes.
Speaker 2:
34:30
So it could be they could be making an extra extra if it's 10 minutes, they could be making an extra $12 an hour. So that's like 30, 30 percent more salary on top of what they're making, and you're saying they're willing to forego 50 percent of that 50 percent of the people donate a part of their bonus.
Speaker 1:
34:52
Oh, so they don't?
Speaker 2:
34:55
have to donate the entire thing.
Speaker 1:
34:56
Yeah, they can donate. Just a dollar they can donate 50.
Speaker 2:
34:59
Do you know what the average is?
Speaker 1:
35:01
I don't remember. On top of my head right now, I think it's around a little bit more than a dollar or something like that.
Speaker 2:
35:08
Okay, so, like 50% of people donate a little bit more than a dollar on average. Okay, so we can like say that people are forgoing like 15% on average of their salaries due to this chat box, which you know that doesn't bode well for like, I think this is one of the main like fears that people have for AI, is one of the main, uh, like fears that people have for ai, or at least you know, the singularity is that the ai is going to get so smart that it's going to be able to convince us to do things that, like that we wouldn't otherwise do, like jump off bridges, or you know. I don't know if you've ever heard this question, but uh, is there, is there a piece of paper like you don't interact with it with some text on it or something on it that could convince you to kill yourself?
Speaker 1:
35:57
yeah, I. I think there is definitely a huge concern about, like how, how these agents can influence your decisions. Um, so that's one of the biggest reason that we do a lot of the works in terms of ethical AIs to constrain these systems. For example, one of the things that we did is you have to disclose that you're an AI agent when people are interacting with you.
Speaker 2:
36:21
That's even scarier. So it's like I'm an AI agent. Now give me your money and you're like okay.
Speaker 1:
36:30
In terms of this really coming down. Who are you representing, right, yeah? So we, I think, like.
Speaker 2:
36:38
Maybe that just says more about people than it does about the AI.
Speaker 1:
36:42
So it's really coming down to at least you want to be transparent that you're not a human, so that's a way that, like sort of the minimum thing that you need to do is about transparent that, whether you're an AI or you're a human, Did you try it the other way?
Speaker 2:
36:59
Did you try it without it telling you that it was an AI?
Speaker 1:
37:01
Yeah, we had actually a paper as an experiment telling them it's an AI and telling them it's not. An AI has some effects, but it's not really coming down to what you're telling them. It's really coming down to how they perceive the agent. That has a huge what do you mean? So if you tell them the AI, people don't believe you Right. Or you tell them it's a human, people don't believe you. They want to believe what they want to believe.
Speaker 1:
37:31
So we have these kind of ab settings and after the conversation we also ask them do you think you're chatting with an ai or chatting with a human? And they can pick a or b right. So we looked at. It's not always the case. People want to believe what they want to believe.
Speaker 2:
37:46
Um, it's really coming I mean, I think people are uh pretty skeptical. So no matter what you tell them at the first, they'll think the other thing. So if you want to be perfectly ethical, you tell them that you're a human and they'll believe you're an AI, and then you negotiate with them.
Speaker 1:
38:01
It's really coming at it. We ask them so in both cases, whether you tell them it's an AI or you tell them it's a human. Like two thirds of them believe it's an AI and one-third of them believe it's a human, it's roughly around the same.
Speaker 2:
38:18
So people aren't that skeptical. They're not more than 50% skeptical. They're less than 50% skeptical.
Speaker 1:
38:24
It's really coming down to the population that you interact with. We recruit people on mechanical terms, so we have more like sort of average in terms of like a US and stuff. It's more of a representation of the mechanical turkers, not like the average Americans or average like people in the world. So it will have different biases in terms of how familiar you are with agents. What are your education backgrounds? Are you a computer scientist? Are you probably have more exposure or age ranges? That would also change in terms of their perception. But what we did is just a small study that we did on 100 people.
Speaker 2:
39:05
Well, when you're asking for people's salary, do you know if they believe that the system is AI or not?
Speaker 1:
39:12
We don't ask them for this. You mean, like asking for them to donate.
Speaker 2:
39:16
Yeah.
Speaker 1:
39:16
So we tested on different conditions whether telling them an AI or telling them it's not an AI. But of course, after the study we just close the study's reasons and tell them exactly what's happening. So there are differences. So what we do find is if you're telling them, it's less about. If you're telling them it's more about they feel. If they feel like they're talking to an AI, and then if the AI is using some of the strategies they feel like only a human can use for example, talking about their kids or family they find it's really off-putting. They don't don't think. But if they're actually thinking that it's a human and they're talking about these kind of personalized things, they actually don't know it more. So there is a certain agency that they think, like what AI can do and cannot do. But if you're a human, you intrinsically have some of the things that you share a connection with. Then you can use some of the strategies that AI cannot use.
Speaker 2:
40:19
What strategies did the AI use?
Speaker 1:
40:23
It's using, mostly trying to imitate how humans are using. So these strategies are basically theoretically like emotional appeal, logical appeal, personalized experience Talking about personalized experience here, personalized experience.
Speaker 2:
40:40
Can you give an example of a logical appeal that an AI would have used?
Speaker 1:
40:45
For example, for donations, we say if you work with us, our donation procedures are very transparent. That's why you can see all, you can figure out exactly where your money goes, goes to which particular supporters. So these kind of logical things to deduce in terms of we are a good organization that you should support.
Speaker 2:
41:08
Okay, but you be the AI for a second, I'll be the humans, suppose, I say, but I just don't like giving away money um.
Speaker 1:
41:19
So, depending on sort of algorithm, they could say I understand you have problems. You probably have um financial issues yourself, but would you be interested to know more some of the greater goods your monies can go into?
Speaker 2:
41:37
Now suppose I say I'm a trillionaire, I hate goods.
Speaker 1:
41:42
Yeah, so whenever it comes into these kind of aversive examples, actually QAs are probably very hard to handle, but for agents, they don't have emotions, they just go on to push in various different strategies on top.
Speaker 2:
41:58
I mean, I suppose, if you're ethically required to tell people that it's an AI system, isn't the majority of the response that you get adversarial, because I imagine that most humans fuck with AI systems. It really depends with AI systems.
Speaker 1:
42:12
It really depends If people are interested in knowing what's going on, what are the procedures, and AI can tell like factual information. They actually it's not a problem for them.
Speaker 2:
42:24
How do you get people interested in knowing what the procedures are?
Speaker 1:
42:28
Yeah, so really coming down to push the value proposition of the organizations, trying to relate to them in terms of the activity they're doing and what are the previous things that they have done in donations and so on?
Speaker 2:
42:42
So this really only works if, at the very core, you're already doing something with a good value proposition.
Speaker 1:
42:49
It really depends. Some of them would think, like some of the, it's really coming down to persuasion, changing the previous behavior. Some people may be on the fence Like, oh, I want to know more about it. The more they know about it, they can form the idea of the. Maybe I want to donate, because I never want, I never actually think about donation. Now you brought it to me about this opportunity. Once I go drill into it, I actually want to do it. So it's more about like your educations or like your personalized information providing. But for people like who already sort of very adamant about they, thought about this thing very through, they know every step that they just don't want to do it. So there's not much you can do.
Speaker 2:
43:33
But, like you said, a human has to use different tactics than the chatbot, and necessarily because you know the person on the other end is affected, knowing whether it's a chatbot or a human right.
Speaker 1:
43:46
Yes, it's really coming down to. There are a lot of reasons human behaviors and chatbot human behaviors are different, for example, in human-human conversations.
Speaker 1:
43:57
you can't just leave because it's face retinue. That's against the human sort of social norm, right? Well, if you chat with a chatbot, you can leave at any time, don't feel guilty about it. So the strategies for the chatbot to do is really coming down. The first is to maintain the conversation, then is to push its agenda. So there are a lot of strategies the chatbot cannot do. For example, being very offensive, people would just leave right away. Or like asking very proactive questions.
Speaker 2:
44:30
Even if you tell it that it's a human.
Speaker 1:
44:34
First of all, when it comes to political things or monies related, you can't pretend to be a human.
Speaker 2:
44:41
Legally you can't.
Speaker 1:
44:43
In some states in California.
Speaker 2:
44:45
Okay.
Speaker 1:
44:46
But some other states don't have this At least. It's not ethical to tell In general. It's not ethical to tell people that it's not a child.
Speaker 2:
44:57
Have you considered telling people nothing and letting them figure it out?
Speaker 1:
45:05
In all of the studies we tell people just to ask for transparency, so I think it's a good practice.
Speaker 2:
45:10
Where do you see this technology being applied next?
Speaker 1:
45:14
You're talking about, like persuasions or framings or planning synchronicities, these convincing chatbots meta on one of the previous paper on Cicero, which is an agent that participates in a game called Diplomacy. It has natural language components in terms of persuasion and building alliances, and I also have a decision in terms of like a movement as well. So this is one of the things that we use like a sort of these kind of game settings to build algorithms that can do both do plannings and generating languages that have different strategies and actions into it. So definitely this is one of the things that we're looking into. There are definitely a lot of ethical considerations.
Speaker 1:
46:13
What if these kind of things like going into like certain people probably shouldn't be exposed of these agents because they're more susceptible, or like children and so on, what if they have a huge impact on the negative side? So we are also looking into how do we detect if it's a chatbot or not. So at least like telling people that this is not a human, this is a chatbot, so you should be careful about how its behavior. Also like detecting what are the potential harm that the chatbot can do. So now we basically build these kind of use cases. You have to make sure that the bots pass all these kinds of test cases before you release the chatbot. So there is no considerations on ethics.
Speaker 2:
46:57
What advice would you have for anybody who's considering a career in AI research right now?
Speaker 1:
47:04
My advice is definitely if you're in school, take the AI related classes like NLP machine learning. You're like take my class. Classes like NLP machine learning they're like take my class.
Speaker 2:
47:16
I need good enrollment numbers.
Speaker 1:
47:23
We actually have a lot of people on the list that we can serve to, which is a big problem that we always have. But again, like, having a formal study is the best way so that people already laid out the curriculum for you. If you're not in schools, like, there are tons of online YouTube and courses that you can go through. But, in terms of AI, don't always go into these kind of repositories to download code and try it yourself. Building something and learning on the go actually usually is more effective than just looking at the slides or theories. When you actually go deep into a particular problem and testing on different algorithms, you would get a better understanding.
Speaker 2:
48:06
Well, thank you very much for coming on.
0:00
But high-risk tasks. You want the model to be able to explain it. Certain regulatory concerns could be addressed, otherwise the humans cannot trust it.
Speaker 2:
0:21
Why don't you tell us a little bit about yourself and the work that you do?
Speaker 1:
0:26
My name is Jo Yu. I'm a professor at Columbia University in the computer science department. I also have a startup that works on commercializing some of my research into products, specifically my research areas. In conversational AI. We work on various algorithms, applications that make human and machine interact more naturally.
Speaker 2:
0:47
Can you tell me a little bit about what got you into that work?
Speaker 1:
0:51
I did my undergrad in computer science and linguistics so I'm always interested in automation and it's very natural to combine computer science and linguistics into natural language processing. So the first project that worked out is machine translation. So later on just explore all types of different applications in natural language processing.
Speaker 2:
1:12
Now you were doing your undergrad, presumably before LLMs.
Speaker 1:
1:15
Oh, way, way before then that was like 2007.
Speaker 2:
1:19
Yeah, so it must be a big change, like everything in the field of NLPs in like 2007 to now. How did you adjust when you know everything started becoming dominated by LLMs?
Speaker 1:
1:37
To us. I think in general, natural language processing is about tasks. It's an application. You're still solving the same problems we have been solving for a couple of decades Machine translation, conversational agents, summarizations so these tasks do not change. It's really coming down to what are? The approaches of these problems change over time. So we started a lot in statistical methods for natural language processing. Then, later on, neural method is now more large language model based.
Speaker 2:
2:08
So it's really the tools that have changed more than the problems at a high level.
Speaker 1:
2:12
Yeah, at a high level. I always say, like the algorithms change, but at the same time, because of the algorithm changes also enabling more tasks to be approachable.
Speaker 2:
2:23
What's the most exciting problem that you've worked on in recent years?
Speaker 1:
2:31
I'm always very fascinated about how people interact, specifically on more complex communication problems like negotiation, so I've been working a lot in terms of how people persuade each other, negotiate with each other and how a machine can take part of that and negotiating or changing people's mind in various cases.
Speaker 2:
2:54
I guess, like you're reading a lot of sales books for that.
Speaker 1:
2:57
Yeah, there are a lot of interesting books, like Never Split the Difference that they're talking about. Really, what are the processes or theories about negotiations and persuasion?
Speaker 2:
3:09
So are you creating systems that like, can negotiate on behalf of humans?
Speaker 1:
3:15
Yes, we do create a system like that. We also create, like a practice, systems for humans to be better at negotiations. So I work with business school professors on developing agents for trainings for students as well.
Speaker 2:
3:29
Are these systems? Can they convince somebody to be better at this? Can they negotiate with a person to make them better at negotiating?
Speaker 1:
3:40
Yes, we do think some of the sort of strategies are taught. You can taught like through assimilations. So before, like, most of the queries are taught like in terms of peer to peer practice. Now you can replace one of the peers with an agent and also give them feedback in terms of strategies within the conversational flow. We also do a lot in terms of helping people to understand what are the best strategies moving forward, and also in terms of algorithm you can plan. So we work a lot on reinforcement learning that can be planned in real time and train on previous data and so on.
Speaker 1:
4:29
So, with these negotiating AIs, what's the biggest challenge in building them? Definitely, lack of data is a huge problem because you want the model to learn from the most realistic scenarios and also, definitely, you wanted to have people who have the real need to sort of interact with the system in order to test the validity of the system. So, definitely, collecting data, testing in realistic environments are very difficult. So we work with various different interdisciplinary cases. For example, we built agents to persuade people to do more physical exercise with nursing schools. Right. We do things with negotiation on practice, for example, buying and selling used cars with business school professors as well.
Speaker 2:
5:15
Oh, wow, I guess this is somewhere that you could have a number right. How many used cars has one of your systems sold?
Speaker 1:
5:22
We never really used it to sell the cars. We're using it to train students to become better negotiators in these kind of distribution or renegotiation scenarios.
Speaker 2:
5:34
How do you address the lack of data here?
Speaker 1:
5:37
So we do work with students, sometimes like, for example, to generate some of scenarios in Roblight, but a lot of times we start with some seed data that is realistic, by recruiting experts and designing the scenarios, and then a lot of time we do work on algorithms, on taking these seed data and expand it through synthetic data generation processes.
Speaker 2:
6:04
You want to tell us a little bit about your startup.
Speaker 1:
6:08
We mostly also working on conversational agents as well, but specifically helping people in terms of, for example, we work with schools in the admissions office to answer questions for the students, helping the schools to better position themselves, give a more personalized experience for their website visitors. So we do analysis on data of their incoming prospects or existing students to figure out what are certain information that would benefit, so that the conversation is not only answering questions but also pushing nudges for the students to complete tasks, to help them to foster their application process or the existing sort of ceiling life or things that the office wanted them to complete.
Speaker 2:
6:57
How did you find that problem?
Speaker 1:
7:00
Mostly through, like my personal experience and so on. So I work in the university. I do see admission officers are super busy and replying emails, while a lot of this could be automated and students could get real time input instead of waiting for another email from the admission office in 24 hours. I myself, for example, I have a two and a half year old hours. I myself, for example, I have a two and a half year old. When I tried to do studies about research on which schools I can send my kids to, it was super difficult. I couldn't call during the work days because I'm working while I'm doing the off hours. Nobody's picking up my phone calls. So there's a lot of these kind of inefficiencies, I think, in communication with companies or schools, with their customers. So I think it was exactly what I've been working on in research. I think it's a great problem to help a lot of parents and students to make the process much easier.
Speaker 2:
7:56
How has building a startup been for you so far?
Speaker 1:
8:00
It's a really interesting path because it's my first time building a startup. We pivoted a couple of times in terms of finding a real problem to solve because we're sort of a technology first company. We had all the technologies on a conversation on agents, on personalized conversations, on persuasions and not just. But we're really looking for what is the right space to apply the technology, which space has a more consistent pay and play. That took us a while in terms of iterations.
Speaker 2:
8:30
What were the previous pivots? If you don't mind my asking.
Speaker 1:
8:34
Yes, so we started working on using the conversational agents to teach people language. So this actually was this sort of like a research and product we built way before the chat GPT. So we're using the sort of conversational agents to teach people language.
Speaker 2:
8:52
When you say teach people language, you mean a new language, not like babies had a speaking bus.
Speaker 1:
8:57
Second language, second language, particularly English. We are building these agents as their conversational practice partners so you can have a conversation with the agents. The agents can give you suggestions and things to correct, can introduce you how to use each certain vocabulary within the context of conversation. So this is a product we actually are still enabling for students to use it, but really, where it's really coming down to there are just so many like a chat GPT could already do. There's really very little in terms of sort of making the process more efficient or engaging, and it's a consumer product, particularly, which requires a lot of like, sales and marketing. So we're not good.
Speaker 2:
9:43
Do you think chat GPT is better at teaching languages than Duolingo?
Speaker 1:
9:49
No, it's totally different in terms of perspective. Like a lot of these kind of Duolingo is focusing on more, like I would say, beginners and intermediate people teaching, for example, vocabularies and so on. But, like for ChatGPT, you can use it for conversational training, but usually it's more geared towards high level proficiency learners. So it's a completely different product and ChatGPT is not designed for language learning. I think a lot of people could benefit from it if you use it in a way that sort of like in communication.
Speaker 2:
10:28
So what were some of your other pivots?
Speaker 1:
10:31
So we also. So this is actually our like biggest pivot. So before that, we started to do a little bit in terms of exploring like verticals, in terms of for us, like our conversation agent is serving like a marketing tool or a customer service tool for a particular company. So we did explore various different verticals to see which one actually, what are the criterias for us to filter company profiles and verticals as well. But I wouldn't say it's a pivot. It's more about like a Micro pivot. I would say it's refining our ideal customer profile.
Speaker 2:
11:11
Okay, yeah. How do you manage running a startup and also being a professor?
Speaker 1:
11:18
Yeah, so it's really coming down to. I have good partners with me in the startup. We have a lot of advisors as well on the business side. So for me, like the like sort of core technologies I work on in terms of research is well aligned with the products that I get sort of research, sort of core research advantages. So it has a lot of these kind of transfer. So my research is mostly working on algorithms, open source code, but the company is more about building a product based on the open source code. So there is a lot of transfer in there too. But again, like it's a really coming down to, there are certain like, for example, in the summer I have I'm not teaching, then I do more of the startup. Sometimes, like I take sabbaticals, for example in the summer I'm not teaching then I do more of the startup. Sometimes, like I take sabbaticals, for example in the semester I'm on a mini sabbatical that I do at certain stages of my company I do a little bit more focus on the company work.
Speaker 2:
12:16
So let's maybe dive into the you know company being based off of open source code. I think it's a really hot topic right now how do you sell something based on open source code?
Speaker 1:
12:27
It's not really particularly what we're selling as open source code. It's really coming down to we use a lot of open source code as the base of the technology that we're selling, so we do have our. The company has its own data, does its own like sort of like, more of like modeling based on the data and then branches on top of it, but still like the underlining, like based models are things that we like all the stories.
Speaker 2:
12:56
OK, so explainable AI. That's something that you've thought about a lot right.
Speaker 1:
13:01
And definitely I have a couple of that are working in those two areas.
Speaker 2:
13:06
So I think explainable AI as a whole brings up a really interesting question. I feel like, at some point, if your AI system isn't good enough, especially with generative AI, it should be able to explain itself, right? That begs the question why do you think it's important to work on explainable AI rather than just performant?
Speaker 1:
13:28
AI. It's really coming down to higher risk tasks. You want the model to be able to explain it so that, for example, certain regulatory concerns could be addressed, for example in medicine. You want to know why the algorithms perform this way. What is the confidence level of that? Otherwise, the humans cannot trust it and it would have a lot of repercussions in terms of security and safety on applying these algorithms in high-risk setups. And also I guess we're coming down to for explainability is really you explain that you want the model to explain so that the human can understand right? So it's really coming down the decision makers a lot of time is the human, so being able to communicate the decisions with a human and a human readable language is helpful for the human to make final decisions.
Speaker 2:
14:24
What do you think people are doing wrong related to explainable AI right now?
Speaker 1:
14:28
I wouldn't say people are doing anything wrong. I think, like a lot of times in research, like there are different avenues you can go and you never know which avenues are actually turn out to be the most efficient or the most or gets the best outcomes. So people are working on various like understanding our intermediate results, understanding some of the model weights.
Speaker 2:
14:57
I think these are possible ways that you can do. So you just published a paper on persona consistency. This was really interesting to me because I have no idea how you measure that, so maybe you could tell us a little bit about, like, what do you do to check that a model is or that a conversation is even stayed in the same persona?
Speaker 1:
15:19
Yeah, so it's really coming down to you could build models that adhere to a certain persona. So the motivation really coming down to if you interact with the chatbot, you want to make sure that the chatbot has a consistent persona. Otherwise, like it jumps around and makes people feel really weird, especially in high cases that the communication task is very complex, or like in gaming or narratives always. So how do you check in terms of consistency? So a lot of people do is like you build, maybe classifiers based on like a consistent personas of an existing human conversations and then use that as a sort of classify, as a metric, to evaluate. There are various things. For example, you can quantify persona in terms of various attributes, such as like your politeness or like your formalities, or like your even in like personalities, directness or other things. So it really coming down to what are the different attributes that you care about. Then, based on that, you build these kind of classifiers to see if there is sort of a consistency among all the conversational terms.
Speaker 2:
16:38
So you're saying you're using a classifier in a lot of cases to check whether the conversation is consistent. How big of a classifier is your using?
Speaker 1:
16:47
It really depends how you want it to. What are the attributes you wanted to track. So even you can use a dense representer to present different personas, and it depends on how much data you have. You can formulate an S1.
Speaker 2:
17:03
like roughly the order of magnitude, like hundreds of millions of parameters, hundreds of billions of parameters we don't specifically say that you need this much of a parameters to represent a persona.
Speaker 1:
17:17
It's the same in terms of like granularities, right? So how much details you want, like if your persona is just in terms of consistency. I want this person to be a happy person.
Speaker 2:
17:26
Yeah.
Speaker 1:
17:26
Then it's just one dimension instead of a happiness. So you can train a model on top of what is like a cheerful and not cheerful conversational style and then use that as a classifier. It could be a small model based on that.
Speaker 2:
17:40
So the reason that I'm asking this is specifically like, I imagine that these classifiers are smaller than GPT-4, right, and the reason that I'm asking that is because I think there's an assumption that a lot of people have that GPT-4 is a giant model and thus it must be good at basically any task you're going to give it. So why not just give it the task of please classify this persona. Like, is this persona consistent? Show it a couple of examples of inconsistent persona conversations and then give it the conversation that you want it to classify and just say does it do that? I guess like you're getting better results with your classifiers than doing that, right.
Speaker 1:
18:25
It depends in terms of how much data that you have and how you train the model. So I think like GPT-4 is just a model that we usually use it for a zero shot or few shot usually gives you good performance. But if you have more than like a few short data points that you can't fit into the context window, you can train a smaller model on more retrocell data and outperform Susha and GPT-4.
Speaker 2:
18:50
I mean, the context window is now like hundreds of thousands of tokens, right? So I guess you must have a lot of data for that.
Speaker 1:
18:57
It's really coming down to. Conversational data usually is very long, so even one data points can be very long.
Speaker 2:
19:04
What ideas are currently floating around in the AI or founders community that you think are wrong?
Speaker 1:
19:11
I wouldn't say in particular wrong. I would just say there are a lot of people working on using sort of like a chat, gpt or these kind of APIs to build on top of it to do various applications, without understanding user pains. So it was like, oh, this is an interesting demo to build and it would be an interesting case to share, but it really, coming down to do people really need it. So it's really the a lot of times that people don't see the need before you actually build a thing. So I also suffer a lot in terms of trying to find product market fit while doing this. So you say, oh, this is a great thing that people have never done before. They must be liking it. But in reality there is like a readiness of adopting new things. Are they having the right infrastructure? There are a lot of other factors, that sort of like constraints in terms of whether this is going to be a good product direction to go.
Speaker 2:
20:09
How do you go about figuring out if people actually have that pain?
Speaker 1:
20:14
Do a lot of user interviews.
Speaker 2:
20:17
What do you see as the biggest pain point in your research right now?
Speaker 1:
20:23
So I work on various different directions in my research.
Speaker 1:
20:27
I just work on one problem, so I have various different problems with respect, but in general, I think it's really coming down to, for example I wonder if I can just use one example that I just talked to my students about as we don't have good evaluation for generative AI models. Yes, we have standard benchmark that people are talking about every day, but these benchmarks get trained by new models very quickly, so we call it data. Basically, we have the leakage instead of the training. So, and then there are no good in terms of evaluation data, particularly for agents. So there are new data sets coming around to these kind of agents, but there are various problems with what's about like on. Like your old API calls Denver and get updated, so running them would having issues, so there are a lot of issues with existing data sets on for evaluations purposes or for training purposes in general. And, of course, the other problem, the biggest pain point, is like we never have enough computational resource to scale up certain model designs that we have.
Speaker 2:
21:43
I mean with the evaluation. I feel like one of the main observations of RLHF was that you should have humans in the loop a little bit more of RLHF was that you should have humans in the loop a little bit more.
Speaker 1:
22:01
Why not literally every time you have an evaluation problem, just set humans on it. Humans are different.
Speaker 2:
22:07
In terms of individually? Yeah, you can set a sample size of humans like they do in biology, right?
Speaker 1:
22:13
Yeah, but humans are also difficult to recruit and coordinate with.
Speaker 1:
22:17
So that's why, back in the day, is why people wanted this kind of batch testing right, so you can't run it at any time?
Speaker 1:
22:27
Right, you just run it, you get results, you get a number, because all these kind of like a sort of we call offline testing, if you wanted to involve humans, then you got to create a pipelines or infrastructure to actually support this interaction and you also want to make sure that your human samples are large enough that doesn't have biases or preferences in certain aspects.
Speaker 1:
22:49
So, in general, like evaluation involving humans are difficult to obtain. Even, for example, we always want an automatic matrix and in terms of like, for example, in the parameter tunings, right, you can have humans on standby or parameter tunings from models. So that's why, in all machine learning research, we really would wanted to have an automated metric that is correlates with the human evaluation. But a lot of times because now, especially with these kind of large language models, people want the diversity of the task and so on. So it's very difficult, especially in generation-based tasks, to have automatic metric that correlates with human. Very well, and a lot of people think, oh, we're going to use GPT-4 as the evaluation metric, which is very wrong in various cases.
Speaker 2:
23:43
What do you think the best approach has been so far to evaluation metrics?
Speaker 1:
23:49
I think it's really coming down to it has to be refined within individual tasks. You can't expect like all the tasks are evaluated by the same metric. So, for example, in agents tasks, a lot of them could be task completion, right To do, actually complete this task that the humans actually want right A lot of like other tasks, like if it's a classification task and it's easy, you have like your ground rules for it. But whenever it comes to generation tasks, there are approximations for like a summarization task or other tasks where you have like certain metrics. So usually I would say like you don't rely on one metric but multiple metrics and sort of a way that multiple metrics combine.
Speaker 2:
24:34
What's the most exciting development that you've seen in AI in the last couple of years?
Speaker 1:
24:39
Definitely the large language model. Everybody wants the generalization and we didn't know. So it's definitely out of my expectation that being able to scale up the model and the data could bring in so much power and zero-shot and few-shot abilities. What problems do you think that you're adding certain constraints or word knowledges? Maybe expanding the model of certain word knowledges wouldn't be. You don't need manually putting the priors anymore.
Speaker 2:
25:41
What do you mean by word knowledges?
Speaker 1:
25:43
For example, physical constraints of an object, would be a word knowledge.
Speaker 2:
25:48
A physical constraint, can you elaborate?
Speaker 1:
25:51
for example, if let's say, um, sorority has issues with, tears have to be, uh like, following the gravity. You can't float on the skies and stuff like that.
Speaker 2:
26:03
Right, these are all word knowledges so you mean like, uh, like, what do they call these? The logical axioms from the old school proof-based database, ais?
Speaker 1:
26:19
I mean expert systems.
Speaker 2:
26:20
Expert systems.
Speaker 1:
26:22
I think it's really coming down to most of the machine learning algorithms that we've built is really coming down to there are ways we can build a model with less data, with less human involvement, but still get good performance. So that's the superiority of a certain algorithm compared to other algorithms. But I guess in terms of the same thing for a large language model is, if you can build a model with less data, less compute, it's more efficient and it can also approach similar performances over the larger model. It's also a very, very important task to complete.
Speaker 2:
27:01
I feel like using less data or less human involvement is also one of these things that we've decided to scale up At the same time. We literally at the company scale AI, that's just said. We can scale up at the same time, like we literally at the company scale AI, that's just said you know, we can scale up human involvement and generate giant like human generated data sets, and that's a great business platform. So, given that you can scale up human involvement, I guess what do you see is the next biggest challenge to work on.
Speaker 1:
27:31
What do you see as the next biggest challenge to work on? I still think like the ultimate understanding or goal is still about like logics, understandings of logics, in order to have like sort of a procedural task completed. So the logic is still for in the large language is that you can't even call them logic, and there is also the causal inference and why certain things come and how things impact each other. I think is still a lot of tasks we can't do well.
Speaker 2:
28:03
What's the most surprising thing? Somebody has used one of your systems for.
Speaker 1:
28:07
Used my algorithms for yeah In general or the open source code.
Speaker 1:
28:13
We do build a lot of these Back in the days. We do a lot of these open source code for dialect systems, dialect systems, dialect systems Basically conversation of AIs or agents. These days we have like a sort of like a chatbot that was built like before the generative AI large language models. We do find people would use it just for a tease. It use it sort of like a do a various adversarial towards it to test. What are the things that they can trick the chatbot to do so that I can definitely see these days that people are still doing this? What are the things that they can trick the chatbot to do so that I can definitely see these days people are?
Speaker 2:
28:57
still doing this for various generative purposes as well. What do they trick the chatbot into doing?
Speaker 1:
29:01
for you guys Various dirty talks, racist talks, everything that they try. We actually also build certain algorithms to understanding in terms of how to protect, sort of make sure that the chatbot doesn't say things or so, like a detector there is sort of these kind of unsafe languages and tell them that we can't respond to.
Speaker 2:
29:25
Don't you think that the chatbot should listen to the user?
Speaker 1:
29:35
Who do you think the chatbot should uh, be subservient to? It's really coming down to. What is the chatbot built to do, right? For example, if the bot was specifically for completing a certain task, right? So, um, it's really coming down to uh, it has a goal to complete and anything in terms of the. If the human comes in it doesn't want to do the task, then the chatbot can refuse to do. Do that, right, and I was built for doing this. So, if you want to do other things, I refer you to do other, like other chatbots. This is the I really coming down to. You can't expect a chatbots to do everything. So a lot of times, a model was just trained for one certain task and it was really coming down to. You need to communicate with your users to understand that you're on the same page so in the case of your uh, negotiating chatbots, what happens?
Speaker 2:
30:27
or sales chatbots, what happens if it's trying to sell to somebody who's like no, I'm not going to buy this from you, unless you know you blow up a building in downtown Manhattan.
Speaker 1:
30:41
This is definitely an adverse task that we've seen people doing. It's really coming down to you have certain word knowledge and saying, like certain requests you can't fulfill, then you can't pretend, then you you can't pretend or lie about it. So this is sort of some of the safety issues we've built in. We have a couple of papers on that too, in terms of, for example, how to. For example, we have a particular chatbot that we build for charities. It's a paper that we, we sort of like help charities to raise funding. And then there were like people asking like, if you donate $100, I would too, but the chatbot cannot donate money, right, and then so we made sure, like there are certain requirements you can't fulfill, you can't lie about it.
Speaker 2:
31:26
I mean, aren't there like infinitely many ways that these things can be hacked? Are there infinitely many bad things that you could possibly want the chatbot to do?
Speaker 1:
31:37
Yeah, so there are a lot of data sets or people trying to collect in terms of what are the first several things that people are saying to these chatbots, so you can have a list of them. So it's like a unit test If you release a certain chatbot, you should test on your unit test to see if they pass it.
Speaker 2:
31:54
I guess the non-for-profit is probably a great example of something with numbers. Did you guys get it to actually raise money for a non-for-profit?
Speaker 1:
32:07
Yeah, we did it as like sort of a separate study. We, we didn't put it on the number of our chat, but we, we, we did it within the sort of a mechanical turk settings.
Speaker 2:
32:16
Yeah, we were able to do how much were you guys able to raise I?
Speaker 1:
32:20
think we were raised like a 200 dollars uh within a certain, so it's not a lot of money.
Speaker 2:
32:26
Well, you know, not a lot of money can also be interesting, like it depends on how you got the not a lot of money right we? Actually get a lot of money can also be interesting, like it depends on how you got the not a lot of money right we actually get a lot of.
Speaker 1:
32:35
So the interesting setup is that we like mechanical turkers to interact with the chatbot, so after the conversation they also get a pay during the conversation and then afterwards they also get a bonus for $2. And they can make a decision in terms of whether to donate the $2 or part of it to the charity. So it's a choice. We basically wanted to see chatting with different chatbots.
Speaker 2:
33:03
Which one of them actually— why would they ever choose to donate For the?
Speaker 1:
33:08
greater good of the charity.
Speaker 2:
33:11
So you're trying to convince the people in Mechanical Turk to donate. That's really interesting.
Speaker 1:
33:15
Yeah, so we had a study on it. We were able to actually do more than half of the people donated.
Speaker 2:
33:23
Oh wow, so your trap bots are very convincing. Trap bots are convincing the Mechanical Turk people to donate. Like these are people who, like they're getting paid like $2 an hour, presumably to do Mechanical Turk.
Speaker 1:
33:36
No, you're not allowed to only pay the $2. They get paid by the hourly wage of a minimum payment, so they get paid, and then, in addition, they get a bonus. Okay, you can donate the bonus. Now you're like a complete.
Speaker 2:
33:52
I don't know how much people get paid on Mechanical Turk at all. It could be $2. It could be $100 an hour. It could be $1,000 an hour.
Speaker 1:
33:58
Minimum wage. They have to be able to do it Minimum wage.
Speaker 2:
34:01
Okay, Well, minimum wage plus $2. And you know, that's like. I think minimum wage is $15. Last time I checked, 10 years ago.
Speaker 1:
34:12
For a different state, it's different. I think it's now like 18 or 20.
Speaker 2:
34:16
Okay, that's 10%, though right per hour, and it's only happening once per hour, or.
Speaker 1:
34:24
No, you just have a conversation with the human, with the agent. So usually it takes you probably five to 10 minutes.
Speaker 2:
34:30
So it could be they could be making an extra extra if it's 10 minutes, they could be making an extra $12 an hour. So that's like 30, 30 percent more salary on top of what they're making, and you're saying they're willing to forego 50 percent of that 50 percent of the people donate a part of their bonus.
Speaker 1:
34:52
Oh, so they don't?
Speaker 2:
34:55
have to donate the entire thing.
Speaker 1:
34:56
Yeah, they can donate. Just a dollar they can donate 50.
Speaker 2:
34:59
Do you know what the average is?
Speaker 1:
35:01
I don't remember. On top of my head right now, I think it's around a little bit more than a dollar or something like that.
Speaker 2:
35:08
Okay, so, like 50% of people donate a little bit more than a dollar on average. Okay, so we can like say that people are forgoing like 15% on average of their salaries due to this chat box, which you know that doesn't bode well for like, I think this is one of the main like fears that people have for AI, is one of the main, uh, like fears that people have for ai, or at least you know, the singularity is that the ai is going to get so smart that it's going to be able to convince us to do things that, like that we wouldn't otherwise do, like jump off bridges, or you know. I don't know if you've ever heard this question, but uh, is there, is there a piece of paper like you don't interact with it with some text on it or something on it that could convince you to kill yourself?
Speaker 1:
35:57
yeah, I. I think there is definitely a huge concern about, like how, how these agents can influence your decisions. Um, so that's one of the biggest reason that we do a lot of the works in terms of ethical AIs to constrain these systems. For example, one of the things that we did is you have to disclose that you're an AI agent when people are interacting with you.
Speaker 2:
36:21
That's even scarier. So it's like I'm an AI agent. Now give me your money and you're like okay.
Speaker 1:
36:30
In terms of this really coming down. Who are you representing, right, yeah? So we, I think, like.
Speaker 2:
36:38
Maybe that just says more about people than it does about the AI.
Speaker 1:
36:42
So it's really coming down to at least you want to be transparent that you're not a human, so that's a way that, like sort of the minimum thing that you need to do is about transparent that, whether you're an AI or you're a human, Did you try it the other way?
Speaker 2:
36:59
Did you try it without it telling you that it was an AI?
Speaker 1:
37:01
Yeah, we had actually a paper as an experiment telling them it's an AI and telling them it's not. An AI has some effects, but it's not really coming down to what you're telling them. It's really coming down to how they perceive the agent. That has a huge what do you mean? So if you tell them the AI, people don't believe you Right. Or you tell them it's a human, people don't believe you. They want to believe what they want to believe.
Speaker 1:
37:31
So we have these kind of ab settings and after the conversation we also ask them do you think you're chatting with an ai or chatting with a human? And they can pick a or b right. So we looked at. It's not always the case. People want to believe what they want to believe.
Speaker 2:
37:46
Um, it's really coming I mean, I think people are uh pretty skeptical. So no matter what you tell them at the first, they'll think the other thing. So if you want to be perfectly ethical, you tell them that you're a human and they'll believe you're an AI, and then you negotiate with them.
Speaker 1:
38:01
It's really coming at it. We ask them so in both cases, whether you tell them it's an AI or you tell them it's a human. Like two thirds of them believe it's an AI and one-third of them believe it's a human, it's roughly around the same.
Speaker 2:
38:18
So people aren't that skeptical. They're not more than 50% skeptical. They're less than 50% skeptical.
Speaker 1:
38:24
It's really coming down to the population that you interact with. We recruit people on mechanical terms, so we have more like sort of average in terms of like a US and stuff. It's more of a representation of the mechanical turkers, not like the average Americans or average like people in the world. So it will have different biases in terms of how familiar you are with agents. What are your education backgrounds? Are you a computer scientist? Are you probably have more exposure or age ranges? That would also change in terms of their perception. But what we did is just a small study that we did on 100 people.
Speaker 2:
39:05
Well, when you're asking for people's salary, do you know if they believe that the system is AI or not?
Speaker 1:
39:12
We don't ask them for this. You mean, like asking for them to donate.
Speaker 2:
39:16
Yeah.
Speaker 1:
39:16
So we tested on different conditions whether telling them an AI or telling them it's not an AI. But of course, after the study we just close the study's reasons and tell them exactly what's happening. So there are differences. So what we do find is if you're telling them, it's less about. If you're telling them it's more about they feel. If they feel like they're talking to an AI, and then if the AI is using some of the strategies they feel like only a human can use for example, talking about their kids or family they find it's really off-putting. They don't don't think. But if they're actually thinking that it's a human and they're talking about these kind of personalized things, they actually don't know it more. So there is a certain agency that they think, like what AI can do and cannot do. But if you're a human, you intrinsically have some of the things that you share a connection with. Then you can use some of the strategies that AI cannot use.
Speaker 2:
40:19
What strategies did the AI use?
Speaker 1:
40:23
It's using, mostly trying to imitate how humans are using. So these strategies are basically theoretically like emotional appeal, logical appeal, personalized experience Talking about personalized experience here, personalized experience.
Speaker 2:
40:40
Can you give an example of a logical appeal that an AI would have used?
Speaker 1:
40:45
For example, for donations, we say if you work with us, our donation procedures are very transparent. That's why you can see all, you can figure out exactly where your money goes, goes to which particular supporters. So these kind of logical things to deduce in terms of we are a good organization that you should support.
Speaker 2:
41:08
Okay, but you be the AI for a second, I'll be the humans, suppose, I say, but I just don't like giving away money um.
Speaker 1:
41:19
So, depending on sort of algorithm, they could say I understand you have problems. You probably have um financial issues yourself, but would you be interested to know more some of the greater goods your monies can go into?
Speaker 2:
41:37
Now suppose I say I'm a trillionaire, I hate goods.
Speaker 1:
41:42
Yeah, so whenever it comes into these kind of aversive examples, actually QAs are probably very hard to handle, but for agents, they don't have emotions, they just go on to push in various different strategies on top.
Speaker 2:
41:58
I mean, I suppose, if you're ethically required to tell people that it's an AI system, isn't the majority of the response that you get adversarial, because I imagine that most humans fuck with AI systems. It really depends with AI systems.
Speaker 1:
42:12
It really depends If people are interested in knowing what's going on, what are the procedures, and AI can tell like factual information. They actually it's not a problem for them.
Speaker 2:
42:24
How do you get people interested in knowing what the procedures are?
Speaker 1:
42:28
Yeah, so really coming down to push the value proposition of the organizations, trying to relate to them in terms of the activity they're doing and what are the previous things that they have done in donations and so on?
Speaker 2:
42:42
So this really only works if, at the very core, you're already doing something with a good value proposition.
Speaker 1:
42:49
It really depends. Some of them would think, like some of the, it's really coming down to persuasion, changing the previous behavior. Some people may be on the fence Like, oh, I want to know more about it. The more they know about it, they can form the idea of the. Maybe I want to donate, because I never want, I never actually think about donation. Now you brought it to me about this opportunity. Once I go drill into it, I actually want to do it. So it's more about like your educations or like your personalized information providing. But for people like who already sort of very adamant about they, thought about this thing very through, they know every step that they just don't want to do it. So there's not much you can do.
Speaker 2:
43:33
But, like you said, a human has to use different tactics than the chatbot, and necessarily because you know the person on the other end is affected, knowing whether it's a chatbot or a human right.
Speaker 1:
43:46
Yes, it's really coming down to. There are a lot of reasons human behaviors and chatbot human behaviors are different, for example, in human-human conversations.
Speaker 1:
43:57
you can't just leave because it's face retinue. That's against the human sort of social norm, right? Well, if you chat with a chatbot, you can leave at any time, don't feel guilty about it. So the strategies for the chatbot to do is really coming down. The first is to maintain the conversation, then is to push its agenda. So there are a lot of strategies the chatbot cannot do. For example, being very offensive, people would just leave right away. Or like asking very proactive questions.
Speaker 2:
44:30
Even if you tell it that it's a human.
Speaker 1:
44:34
First of all, when it comes to political things or monies related, you can't pretend to be a human.
Speaker 2:
44:41
Legally you can't.
Speaker 1:
44:43
In some states in California.
Speaker 2:
44:45
Okay.
Speaker 1:
44:46
But some other states don't have this At least. It's not ethical to tell In general. It's not ethical to tell people that it's not a child.
Speaker 2:
44:57
Have you considered telling people nothing and letting them figure it out?
Speaker 1:
45:05
In all of the studies we tell people just to ask for transparency, so I think it's a good practice.
Speaker 2:
45:10
Where do you see this technology being applied next?
Speaker 1:
45:14
You're talking about, like persuasions or framings or planning synchronicities, these convincing chatbots meta on one of the previous paper on Cicero, which is an agent that participates in a game called Diplomacy. It has natural language components in terms of persuasion and building alliances, and I also have a decision in terms of like a movement as well. So this is one of the things that we use like a sort of these kind of game settings to build algorithms that can do both do plannings and generating languages that have different strategies and actions into it. So definitely this is one of the things that we're looking into. There are definitely a lot of ethical considerations.
Speaker 1:
46:13
What if these kind of things like going into like certain people probably shouldn't be exposed of these agents because they're more susceptible, or like children and so on, what if they have a huge impact on the negative side? So we are also looking into how do we detect if it's a chatbot or not. So at least like telling people that this is not a human, this is a chatbot, so you should be careful about how its behavior. Also like detecting what are the potential harm that the chatbot can do. So now we basically build these kind of use cases. You have to make sure that the bots pass all these kinds of test cases before you release the chatbot. So there is no considerations on ethics.
Speaker 2:
46:57
What advice would you have for anybody who's considering a career in AI research right now?
Speaker 1:
47:04
My advice is definitely if you're in school, take the AI related classes like NLP machine learning. You're like take my class. Classes like NLP machine learning they're like take my class.
Speaker 2:
47:16
I need good enrollment numbers.
Speaker 1:
47:23
We actually have a lot of people on the list that we can serve to, which is a big problem that we always have. But again, like, having a formal study is the best way so that people already laid out the curriculum for you. If you're not in schools, like, there are tons of online YouTube and courses that you can go through. But, in terms of AI, don't always go into these kind of repositories to download code and try it yourself. Building something and learning on the go actually usually is more effective than just looking at the slides or theories. When you actually go deep into a particular problem and testing on different algorithms, you would get a better understanding.
Speaker 2:
48:06
Well, thank you very much for coming on.
جستجوی اپیزودهای گذشته
اپیزودهای قبلی Dr. Mirman's Accelerometer را جستجو کن.
سلب مسئولیت: پادکست و آثار هنری تعبیه شده در این صفحه متعلق به Matthew Mirman است که متعلق به صاحب آن است و به Listen Notes، Inc وابسته یا تایید نشده است.
ویرایش
از کمک شما برای بروز نگهداشتن پایگاهدادههای پادکست سپاسگزاریم
