Does AI Replace Corpus Linguistics? | Randy Barnett, Jack Kieffaber & James Heilpern

WEBVTT
00:00:00.239 --> 00:00:02.640
Sign in to have credit for attendance.
00:00:04.400 --> 00:00:06.799
Okay, good evening and welcome.
00:00:06.959 --> 00:00:12.400
My name is Sarah Balou and I'm the president of the Georgetown Law Chapter of the Federalist Society.
00:00:13.279 --> 00:00:18.640
Thank you all for being here tonight for the latest installment of the Daniel Webster Debate Series.
00:00:18.800 --> 00:00:30.320
Our first debate series, also co-sponsored with Georgetown Center for the Constitution, of which Randy Barnett, who is our faculty advisor, is also the faculty director.
00:00:30.960 --> 00:00:34.479
Tonight's question is simple to state and hard to answer.
00:00:34.719 --> 00:00:38.399
Does artificial intelligence replace corpus linguistics?
00:00:38.880 --> 00:00:45.520
For the past decade, corpus linguistics has promised originalists and textualists something they have long wanted.
00:00:45.759 --> 00:00:50.560
It offers an empirical check on what words mean to the people who used them.
00:00:50.960 --> 00:01:03.359
Instead of consulting one dictionary or relying on a judge's own sense of that language, the interpreter can search large databases of real-world usage, such as the corpus of founding era American English.
00:01:04.079 --> 00:01:09.040
That method has made its way from the law reviews into judicial opinions.
00:01:09.599 --> 00:01:11.840
Then came large language models.
00:01:12.079 --> 00:01:25.599
In 2024, Judge Kevin Newsome of the 11th Circuit wrote a concurrence that asked, half seriously and half as an experiment, whether a court might consult a tool like ChatGPT to learn the ordinary meaning of a word.
00:01:26.079 --> 00:01:28.799
Almost at once a new debate took shape.
00:01:29.120 --> 00:01:37.280
Do these models, trained on vast amounts of text, give us everything corpus linguistics gives us only faster and more fluently?
00:01:37.519 --> 00:01:46.000
Or are they an opaque black box, a poor substitute for a method whose appeal lies in its transparency and the fact that anyone can check it?
00:01:47.040 --> 00:01:49.840
Underneath that practical question are deeper ones.
00:01:50.000 --> 00:01:53.359
What is ordinary meaning and whose meaning counts?
00:01:54.000 --> 00:01:59.519
We're very fortunate to have an expert on original public meaning today as our moderator.
00:02:00.480 --> 00:02:29.199
My VP of debates, Louis Faum, will go more into the speaker bios, but I want to acknowledge Professor Barnett not only as an expert on originalism and original public meaning, but also as one of the only professors at Georgetown who is cautiously optimistic in embracing artificial intelligence and framing it in a way to his students that, in his words, will teach them to be masters of AI rather than like letting AI be masters of them.
00:02:29.439 --> 00:02:33.439
An important skill that many lawyers will need to know in the next few years.
00:02:33.680 --> 00:02:36.240
So thank you, Professor Barnett, for joining us.
00:02:36.639 --> 00:02:41.360
And with that note, I'm going to introduce our vice president of debates, Louis Fong.
00:02:48.319 --> 00:02:51.520
Good evening, everyone, and thank you for coming to tonight's name.
00:02:51.840 --> 00:03:02.800
Our moderator is Professor Professor Randy Barnett, who is a Patrick Oton professor of constitutional law here at Georgetown University Law Center, and the faculty director at the Center of the Constitution.
00:03:03.039 --> 00:03:07.759
He tried many felony cases as prosecutor at the Cook County State Southern Soffice of Chicago.
00:03:08.000 --> 00:03:17.520
He's received a good high fellowship in constitutional studies and the Radio Thomas, and has authored 12 books and more than 100 articles in the news, as well as numerous authors.
00:03:18.159 --> 00:03:21.439
We also have Jackieber joining us, who is the founder of RA.
00:03:22.080 --> 00:03:36.240
He served as an adjunct professor at the Catholic University of America at the Columbus School of Law, where he teaches a course on AI and served as a law learner to judge anymore on the district board at the District of Columbia, and to judge Newson on the Court of Appeals and Library.
00:03:37.120 --> 00:03:45.919
We also have James Elmer joining us, who specializes in complex litigation and corpus linguistics in Germany, representing clients at all stages of litigation.
00:03:46.159 --> 00:03:55.120
He is a leading figure in law and corpus linguistics, having authored some of the first race in the country to employ corpus linguistics and offers training sessions as well.
00:03:55.360 --> 00:03:57.759
He clerked for the Honorable Richard Cleveland on the U.S.
00:03:57.840 --> 00:04:09.280
District Court in the Eastern District of Michigan, and he didn't have it with that one to say thank you for the introduction.
00:04:09.439 --> 00:04:14.479
Thank you for introducing the other debaters too, so now I don't have to do it, so that's that's great.
00:04:20.959 --> 00:04:24.319
I've in the past been a big supporter of corpus linguistics.
00:04:24.399 --> 00:04:53.759
I I have been characterized by some of its developers as one of the earliest practitioners of corpus linguistics, because I had an article published in the University of Chicago Law School Law Review in 2001, in which I surveyed the use of all the operative terms of the Commerce Clause, every use of the operative terms of the Commerce Clause in the Constitutional Convention, in the Federalist Papers, and in the ratification debates.
00:04:53.839 --> 00:05:22.000
And then I had a sequel article in which I had located a database of all the issues, all the issues of the Pennsylvania Gazette published between around 1710 to 1805, in which I surveyed every use of the word commerce and the rest of the operative terms in the Commerce Clause, of which there were about 1,500 mentions of the word commerce in that database, as a means of trying to identify what was the original public meaning of these words.
00:05:22.560 --> 00:05:31.839
And came to the same con came to the conclusion that Justice Thomas was largely correct in his concurring opinion in the Lopez cases, which is what I was trying to decide.
00:05:32.000 --> 00:05:37.759
I was trying to see if I could defend Justice Thomas's analysis in Lopez, and I provided that defense.
00:05:38.079 --> 00:05:47.279
Since then, corpus linguistics, of course, has greatly expanded the corpus beyond that which I discovered, which also had very, very rudimentary search uh capabilities.
00:05:47.360 --> 00:05:49.920
Uh, so it was really kind of annoying to try to use.
00:05:50.000 --> 00:05:54.800
Uh, but that was the first effort to try to objectify the search for objective meaning.
00:05:54.959 --> 00:06:11.040
Uh since then, um, I have been a supporter of corpus linguistics, and James has uh lectured uh uh uh one summer after another for the originalism summer seminar in order to bring students up to speed on how corpus linguistics works, since it's such an essential tool.
00:06:11.279 --> 00:06:30.560
But I have developed some doubts about it, um, primarily about whether it's fair to conclude that because a particular usage was very, very common, perhaps by far the majority use of a particular word, that that means it was what was meant, uh, the public meaning of the Constitution in context.
00:06:30.639 --> 00:06:43.920
Uh, when in fact um for for example, the right to bear arms might have had a primarily, might have been primarily a military connotation, but that doesn't mean it had an exclusively military connotation in the Second Amendment itself.
00:06:44.079 --> 00:06:56.240
And there is, I think, a natural tendency, as some people in the Second Amendment have discovered, to try to equate the dominant usage with the origin with the conventional understanding of the words in the Constitution uh in context.
00:06:56.319 --> 00:06:58.639
So that has actually given me some pause about it.
00:06:58.879 --> 00:07:04.800
Add to that the development of AI, as Sarah mentioned, I'm actually a pretty avid user of AI.
00:07:04.879 --> 00:07:07.279
I'm a I I think it's a very important tool.
00:07:07.360 --> 00:07:11.439
Um, and I think it has enormous potential.
00:07:11.600 --> 00:07:20.160
Um and it could, AI could do to corpus linguistics what AI is currently doing to search, which is basically making search obsolete.
00:07:20.319 --> 00:07:22.639
Um, search like Google, search like Bing.
00:07:22.879 --> 00:07:34.319
Um, because in fact, corpus linguistics could be considered a sophisticated, specialized form of search uh rather than an analysis that the kind that uh or a synthesis of the kind that AI is capable of.
00:07:34.720 --> 00:07:38.800
Having said that, we're all we all know that AI is capable of hallucinating.
00:07:38.959 --> 00:07:45.680
Um it loves to please you by delivering a result that sounds good to you, um, even if it turns out not to be based on fact.
00:07:45.839 --> 00:07:50.319
And there are precautions you can take against that eventuality, but you always have to be wary of it.
00:07:50.399 --> 00:08:04.000
And right now, the use of AI in the practice of law is extraordinarily risky, and people are having their law licenses questioned because they've submitted uh fraudulent uh citations to the courts on the basis of an AI engine, uh large language model.
00:08:04.399 --> 00:08:11.680
So, with that as context, I welcome the opportunity to moderate this debate just because I wanted to hear the debate.
00:08:11.839 --> 00:08:20.720
Uh I I'm I'm not even sure this is going to be a debate as opposed to an informed discussion uh by two experienced users of the technologies that they're going to be discussing.
00:08:20.800 --> 00:08:29.839
And I, for one, have a very open mind about how it can be used, whether AI will supplant corpus linguistics, what it would mean to say that it's supplanting corpus linguistics.
00:08:30.000 --> 00:08:34.080
And so I'm just looking forward to hearing what each of the which each of the speakers say.
00:08:34.559 --> 00:08:41.440
I've been told they're each given seven minutes to do their opening statements, and then we'll have an exchange for how many rounds do is the is the procedure?
00:08:42.240 --> 00:08:47.279
One remote and then Okay.
00:08:47.440 --> 00:08:49.279
So hopefully I'll have some questions.
00:08:49.519 --> 00:09:00.080
And so I guess the affirmative, which is the position that AI is going to uh supplant uh corpus linguistics, um, is going to be Jack.
00:09:00.320 --> 00:09:00.879
So take it away.
00:09:01.200 --> 00:09:01.600
Thanks, Professor.
00:09:01.759 --> 00:09:03.679
It's an honor to be here again with you guys.
00:09:03.840 --> 00:09:05.840
I'm not sure how much of a debate this is going to be.
00:09:06.080 --> 00:09:08.960
At minimum, we are sort of rival CLE providers.
00:09:09.120 --> 00:09:19.519
I've got this business with Brian Garner, who I do believe is still the leader in what they call baby judge schools, but JEI, fast uh approaching judicial uh education institute is one of this.
00:09:20.159 --> 00:09:24.159
But yeah, I mean, we've got this robot today, smarter than anybody to ever live, they're saying.
00:09:24.320 --> 00:09:30.879
It builds your bioweapons, it's firing nukes, and so you get the question that's on everybody's mind uh what's going to happen to corpus linguistics?
00:09:31.120 --> 00:09:32.399
That's the top-level stuff.
00:09:32.480 --> 00:09:34.799
But I mean, it is a more serious question than that.
00:09:34.879 --> 00:09:38.000
And I what I really want to do today is insofar as Mr.
00:09:38.159 --> 00:09:45.759
Hubert wants to incorporate more AI and what he's doing, which sounds like he might want to, I just want to let him know sort of what he's sitting on top of kind of the powder keg.
00:09:45.919 --> 00:09:51.519
You know, when you're selling a methodology in this day and age with AI, I call you a methodologue.
00:09:51.600 --> 00:10:02.879
You're kind of playing for world domination here, because if enough judges latch on to what you're doing the same way they latched on to Scully and Garner, you know, cited over 5,000 times of circuit courts, a hundred SCOTA sites.
00:10:03.039 --> 00:10:06.639
It's different in kind than selling them a book if you sell them a bot.
00:10:06.799 --> 00:10:09.840
You kind of are a legislature at that point.
00:10:10.000 --> 00:10:12.559
Nobody elected you, nobody cast a vote for you.
00:10:12.639 --> 00:10:17.039
You just convinced a bunch of judges to use your product, and suddenly who controls those weights and models?
00:10:17.200 --> 00:10:22.480
And as a backdrop of this conversation, I should let you know, um, Claude and GPT are already in the lead on that.
00:10:22.559 --> 00:10:36.960
You're gonna get judges lazier and lazier, because I know the way all you guys study, less and less reading, more and more querying bots, and if that's kind of the trajectory, the provision's on, these things are gonna have a lot of influence on how laws come out and what they mean.
00:10:37.200 --> 00:10:39.039
But basically, there's two things I want to talk about.
00:10:39.120 --> 00:10:40.960
It's corpus and method.
00:10:41.120 --> 00:10:43.759
Those are kind of the two main things to interpretation.
00:10:43.840 --> 00:10:45.919
I think we're sort of in dialogue on.
00:10:46.080 --> 00:10:49.279
And by corpus, there used to be sort of higher-minded notions of this.
00:10:49.360 --> 00:10:52.559
I think anybody who's done originalism or textualism knows what I mean by corpus.
00:10:52.720 --> 00:10:54.159
It's a big stack of PDFs.
00:10:54.320 --> 00:10:57.120
You work at a law firm, they say, we want you to do a textual analysis.
00:10:57.279 --> 00:11:06.320
You go download as many PDFs as you possibly can, and you're crossing your fingers, give me a John Adams, give me a Thomas Jefferson, you're control Fing in the old days for firearm.
00:11:06.639 --> 00:11:10.799
Sometimes there's a fire hyphen arm, but you didn't control F for the hyphen, so you missed that one.
00:11:10.879 --> 00:11:14.879
And the better lawyer, the better law student, he types the hyphen in because he's really careful.
00:11:15.120 --> 00:11:16.159
That was the project.
00:11:16.320 --> 00:11:25.840
And in sort of the pre-Mortzen years, the pre-Steve Moritzon, the guy who invented corpus linguistics, you were kind of in the Wild West where there was no fixed corpus and no fixed method.
00:11:26.000 --> 00:11:31.840
I'm more of a John Adams guy in the sense that Adams said at the founding there should be an enumerated dictionary.
00:11:32.000 --> 00:11:32.559
He was big on.
00:11:32.639 --> 00:11:35.279
This is kind of a Benthamite, Jeremy Bentham type point.
00:11:35.600 --> 00:11:41.679
If you're going to have a more formalist legal system, what the words mean actually needs to be legislated.
00:11:41.759 --> 00:11:48.559
This isn't something we should sort of leave up to, you know, the beauty and intricacies of human interaction, human language.
00:11:48.639 --> 00:11:50.320
And I do very much agree with that.
00:11:50.399 --> 00:11:58.960
I think there should be legal language, and that's why the most controversial thing that might set us off on a debate, I don't really care which method you use.
00:11:59.039 --> 00:12:00.480
I don't even really care what the words mean.
00:12:00.559 --> 00:12:02.639
What I care is that we decide what they mean.
00:12:02.799 --> 00:12:08.320
We do so consistently, such that you can give Congress sort of a building block language to write statutes in.
00:12:08.480 --> 00:12:14.559
My answer to no vehicles in the park is that that's a poorly written statute, and you need a better word than vehicles.
00:12:14.639 --> 00:12:20.399
And I think corpus linguists have us somewhere down the road to making that language.
00:12:20.559 --> 00:12:22.799
I'll give you a hint, the language is binary code.
00:12:23.039 --> 00:12:24.720
By the end of this, you're going to see kind of what I mean.
00:12:24.879 --> 00:12:27.120
But pre-Mortzin, there was no fixed corpus, right?
00:12:27.200 --> 00:12:28.639
Guys are pulling dictionaries off the shelf.
00:12:28.799 --> 00:12:32.480
Sometimes they'd even pull out dictionaries from the era in which the statute was drafted.
00:12:32.639 --> 00:12:38.320
And of course, the dictionary writer, Webster, who have you, he's already done his own corpus collection and methodology.
00:12:38.480 --> 00:12:43.519
Garner owns all these classic old school 600, 700 dictionaries, 700s dictionaries.
00:12:43.679 --> 00:12:45.039
That's what lexiconers do.
00:12:45.200 --> 00:12:49.919
So you're deferring to their corpus and method, and then every judge had his own method for interpreting that document.
00:12:50.080 --> 00:12:52.159
It was kind of the wild west.
00:12:52.320 --> 00:12:54.159
And this is the Steve Moritzon revelation.
00:12:54.240 --> 00:12:59.440
He comes along with corpus linguistics and says, well, let's at least decide what corpus we're working from.
00:12:59.600 --> 00:13:02.000
Let's fix some documents.
00:13:02.480 --> 00:13:05.360
And right now, you know, there's an entity, I believe it's BYU.
00:13:05.519 --> 00:13:07.120
BYU fixes the corpus.
00:13:07.519 --> 00:13:09.279
Here are the PDFs you're using.
00:13:09.519 --> 00:13:11.759
I do think that's a pretty good start on this.
00:13:11.840 --> 00:13:16.240
If you want to go towards the Adams position of, well, how would you even start to enumerate a dictionary?
00:13:16.399 --> 00:13:22.799
Well, let's start by saying these are the official PDFs of the United States, or perhaps more realistically, of my contract.
00:13:22.960 --> 00:13:30.320
I also built talking contracts as my day job, a contract you can ask its own AI built-in AI judge for ex ante opinions.
00:13:30.480 --> 00:13:32.720
You'll see how this is kind of conducive to that.
00:13:32.960 --> 00:13:36.720
So that's what we have right now, and I think Professor Barnett teased up very well.
00:13:36.799 --> 00:13:44.639
It's all about how do you query this thing, how do you get to individual instances, but I want to make note of what the corpus linguist actually does when he has the instances at his fingertips.
00:13:44.799 --> 00:13:48.159
He gets 200 of them, let's say, and he lines them all up.
00:13:48.240 --> 00:13:51.519
And what I've heard so far is kind of in keeping with what I've heard Mr.
00:13:51.679 --> 00:13:56.240
Halpern and uh Judge Thomas Lee, who's also a big adherent of this stuff, talk about.
00:13:56.799 --> 00:13:58.639
You need to be able to get those 200 faster.
00:13:58.799 --> 00:14:00.399
But what do you do when you have the 200?
00:14:00.559 --> 00:14:08.399
Well, you take each individual one, you kind of bucket like two conflicting interpretations of the word that you're doing in that particular case, and you go like this.
00:14:08.559 --> 00:14:10.320
You go zero, one, zero, one.
00:14:10.559 --> 00:14:11.919
You literally, they call it coding.
00:14:12.000 --> 00:14:14.559
You code them one box or the other.
00:14:14.799 --> 00:14:25.279
And what with the robot being smarter than everybody and building the bioweapons and the nukes and all that, you kind of got to ask yourself, and so far as is at least to what Justice Lee has put in writing, this is kind of his operative question.
00:14:25.440 --> 00:14:26.559
Could a robot do that?
00:14:26.720 --> 00:14:29.120
Could it do that pretty consistently?
00:14:29.600 --> 00:14:30.960
I kind of think yeah.
00:14:31.120 --> 00:14:34.879
You've got it down to this very sort of already mechanical level.
00:14:35.039 --> 00:14:38.080
Do you need the human to do that last step of bing, bing, bing, bing?
00:14:38.240 --> 00:14:39.519
He might, in theory, mess it up.
00:14:39.600 --> 00:14:41.919
And that's always been my one problem with corpus linguistics.
00:14:42.000 --> 00:14:45.919
At the end of the day, you build this corpus, and they say, I've read a lot of Justice Lee's papers.
00:14:46.080 --> 00:14:47.919
He cites to these big books on linguistics.
00:14:48.000 --> 00:14:49.200
Well, how do you actually do the coding?
00:14:49.279 --> 00:14:50.720
Well, read those, come back.
00:14:50.960 --> 00:14:51.440
Uh-oh.
00:14:51.679 --> 00:14:58.080
I don't know if judges are going to be any more uniform in doing that than they have been in doing originalism and textualism generally.
00:14:58.240 --> 00:15:03.120
But you've got this Lee quote from the article he wrote against uh Judge Newsom, my old boss in Snell.
00:15:03.200 --> 00:15:07.200
And I was there the Snell term, so I have some uh I've got something wrapped up in this.
00:15:07.360 --> 00:15:14.480
He said, some researchers, this is a quote, some researchers should be tempted to ask, could be tempted to ask, AI to analyze the quantitative results of coding.
00:15:14.720 --> 00:15:15.120
Do this.
00:15:15.279 --> 00:15:19.759
And he says, that would be a step too far, too close to robojudging.
00:15:20.000 --> 00:15:36.559
And to that I say, and I'm not even sure if uh James would disagree with this, but if we distilled text into a coding, where you'd go on bing, bing, bang, for individual instances, I don't know, if your job was to make judging more human, I don't know if you would have gone down the entire corpus linguistics road to begin with.
00:15:36.720 --> 00:15:43.039
I think you might have pulled up a little short of that, you know, maybe in the more Vermulean camp of things, if you wanted to add to the humanity.
00:15:43.200 --> 00:15:49.120
If we're down this mechanical road, I might be the rare uh opponent of corpus linguistics who says, well, it's not mechanical enough.
00:15:49.200 --> 00:15:50.720
Usually here it's too mechanical.
00:15:50.879 --> 00:15:54.080
I think there's got to be a third step that fixes the method.
00:15:54.240 --> 00:15:57.120
Right now, the corpus is fixed, the methods with slapping on the breeze.
00:15:57.279 --> 00:15:58.639
This is why I called Brian Garner.
00:15:58.799 --> 00:16:07.840
I was doing some talk at SMU, and I managed to get dinner with him, and after that we went to his house where he has three big libraries, which I like, because you're looking at those libraries and thinking, well, that's a corpus.
00:16:08.080 --> 00:16:09.919
I just when I look at a library, I'm sick in the head.
00:16:10.000 --> 00:16:11.919
I see PDFs everywhere, a lot of PDFs.
00:16:12.080 --> 00:16:14.639
I thought you could print all those out, you could take those, put them in a corpus.
00:16:14.879 --> 00:16:19.279
And then you got to think, well, whose method has kind of been deciding on these things?
00:16:19.519 --> 00:16:26.720
You know, when Kagan says we're all textualists now, um, in a way she's kind of saying we're all doing Scalia and Garner now.
00:16:26.799 --> 00:16:28.720
That was the big move that kind of crystallized this.
00:16:28.799 --> 00:16:39.279
I mean, these guys invented the cannons of construction, they distilled them from the past, but a lot of the names were applied literally by Justice Scalia, in a lot of cases by Brian Garner while writing this book.
00:16:39.519 --> 00:16:42.240
So you've got a tested method that a lot of people are using.
00:16:42.399 --> 00:16:45.759
The question is, could you distill that into a bot?
00:16:45.919 --> 00:16:47.200
I thought, how would I do that?
00:16:47.440 --> 00:16:55.759
The best I come up with is, well, I want to make sort of a starting baseline rapper that's read all of his stuff, and then I want Brian Garner to train the thing.
00:16:55.919 --> 00:16:57.279
I want him to grade outputs from it.
00:16:57.440 --> 00:17:00.000
If you know anything about Brian Garner, he does this for a living.
00:17:00.159 --> 00:17:08.400
He'll go to your law firm, he'll go to your baby judge school, and he will grade outputs from these judges, textual reviews from his students all the way up to circuit judges, and he's got a lot of notes.
00:17:08.559 --> 00:17:12.480
We read all those notes, and we said, all right, do it again talking directly to this bot.
00:17:12.720 --> 00:17:16.559
So, in a way, we've been sort of enumerating the Garner consciousness.
00:17:16.640 --> 00:17:23.440
And indeed, he's got this follow-up sort of a spiritual successor to reading law coming out, and uh it's not going to be a book.
00:17:23.519 --> 00:17:25.359
It's coming out as a bot.
00:17:25.440 --> 00:17:30.079
We're gonna have five copies, we'll put them on the table, and then we'll whisk them away the second anybody wants to know what's in there.
00:17:30.240 --> 00:17:32.160
You want to know what's in the bot in the book writer.
00:17:32.400 --> 00:17:34.640
You've got to look at the bot.
00:17:34.880 --> 00:17:41.759
And the thought process, and I'm very interested because I know that James sits on a pretty uh coherent corpus in his own right.
00:17:41.920 --> 00:17:47.039
If you've got that corpus, you've got the predictable method, what do you get if you garnerize, you freeze the method?
00:17:47.119 --> 00:17:52.480
Well, you kind of get that talking statute, kind of binary code law, because what does the back end look like, the thing?
00:17:52.640 --> 00:18:00.160
The back end is a black box zeros and ones, and now you do have that more sort of flogged into binary, that more predictable language.
00:18:00.319 --> 00:18:06.000
You could get advisory opinions on the front end, asking a garnerizer, and if enough people are using it, it's right.
00:18:06.160 --> 00:18:09.039
So that's why I mean a methodologue is playing for world domination.
00:18:09.119 --> 00:18:11.200
So from one methodologue to another.
00:18:11.599 --> 00:18:12.000
Thank you.
00:18:12.079 --> 00:18:13.440
That was right on time.
00:18:14.400 --> 00:18:15.119
All right, okay.
00:18:16.000 --> 00:18:20.559
Thank you, Randy, thank you, Jack, and thank you, Sarah, for inviting me.
00:18:20.720 --> 00:18:28.960
Um, so I think we want to take a step back right now and uh and and just sort of cue this up by talking about what are the purposes of textualism.
00:18:29.039 --> 00:18:34.640
I I assume that if you're in this room, you know what textualism is, so we'll skip that step and talk about why do we care.
00:18:34.960 --> 00:18:40.799
And I think, you know, the whole corpora, whole large language models have been written about this topic.
00:18:41.039 --> 00:18:47.759
But I think we can kind of crystallize it down to textualism respects the separation of powers.
00:18:48.319 --> 00:18:55.680
It fundamentally concedes that our legislatures should be the ones who write the law.
00:18:56.319 --> 00:19:01.839
If we're in this room as different members of society, we're gonna disagree about a whole bunch of stuff.
00:19:02.160 --> 00:19:16.799
And then through our elected representatives, we're gonna argue about that stuff, and before we pass laws that say thou shalt do this or thou shalt not do that, we're gonna crystallize an agreement that at least a majority of our representatives agree to.
00:19:17.039 --> 00:19:20.480
And we're gonna, you know, memorialize that in writing.
00:19:20.880 --> 00:19:35.039
And then our constitutional system says that those words that our representatives agreed to remain the law until that legislative process uh is used to change it.
00:19:36.480 --> 00:19:39.359
So uh I'm not breaking any new grounds there.
00:19:39.519 --> 00:19:58.480
Uh the point is that textualism has been criticized over the years because the tools that that folks have been using have not necessarily delivered on the promise of objectivity and uh and adherence to that law that has been produced by our legislators.
00:20:00.000 --> 00:20:15.839
And corpus and and so what I want to talk about, uh I'm the first one to acknowledge that a lot of the tools that uh textualists have used over the time have been uh lacking and haven't necessarily delivered on that promise of objectivity.
00:20:16.000 --> 00:20:20.960
Uh but I think that corpus linguistics can help us get there in ways that AI can't.
00:20:21.119 --> 00:20:23.680
And so let me just sketch a couple of these.
00:20:23.920 --> 00:20:29.599
Uh number one, and Jack cued this up, corpus linguistics is transparent.
00:20:29.839 --> 00:20:32.559
Okay, and it's transparent in at least a couple of different ways.
00:20:32.720 --> 00:20:46.400
Number one, all of the databases that are used by judges and by lawyers in corpus linguistics are freely accessible, and it is totally clear what's in the database.
00:20:46.559 --> 00:20:56.640
They've been curated by linguists, they've been curated by historians to try to present an objective representation of a particular community.
00:20:57.200 --> 00:20:59.519
Uh AI, not so much.
00:20:59.839 --> 00:21:04.640
AI may have a lot of stuff, but we don't necessarily know what's in it.
00:21:04.880 --> 00:21:06.160
And that's a problem.
00:21:06.319 --> 00:21:09.839
Now, Randy talked about uh the bear arms question.
00:21:09.920 --> 00:21:17.599
Now, there's a database out there uh that is called the Corpus of American Soap Operas.
00:21:18.240 --> 00:21:34.880
And you could do the greatest corpus analysis in the history of the world, analyzing all of the all of the scripts of all of the soap operas that have ever been written, and it wouldn't tell you diddly squat about what the Second Amendment means.
00:21:35.200 --> 00:21:48.640
We don't know when we use large language models, we don't know if the very confident answer that it is spitting out is as a result of American soap operas or modern usage.
00:21:48.799 --> 00:21:49.759
We just don't know.
00:21:50.240 --> 00:21:52.400
Number two, it's replicatable.
00:21:52.559 --> 00:21:52.880
Okay?
00:21:53.039 --> 00:21:57.680
Now I wrote a brief in a very boring case called Rimini Street v.
00:21:57.759 --> 00:21:58.319
Oracle.
00:21:58.400 --> 00:21:59.599
It was a$13.5 million.
00:22:00.240 --> 00:22:04.240
Dispute that boiled down to the meaning of the word full in the Copyright Act.
00:22:04.319 --> 00:22:08.480
And I'm just going to leave that as the summary because you don't want to know more because it's very boring.
00:22:09.839 --> 00:22:16.720
I produced a brief that I submitted to the Supreme Court that relied on four or five different corpora.
00:22:17.839 --> 00:22:29.759
I provided hundreds of examples, coded them, showed all of the different ways in which full was used in all of its many different contexts in different time periods and among different communities.
00:22:29.839 --> 00:22:37.039
And I submitted it to the court and I said, regardless of what your jurisprudence is, here's some data that will be helpful.
00:22:37.680 --> 00:22:44.880
Now, the we ended up, uh our our our analysis ended up supporting Ramini Street.
00:22:45.519 --> 00:22:47.440
Oracle was represented by Paul Clement.
00:22:47.680 --> 00:22:50.559
He clearly read that brief, was clearly nervous about that brief.
00:22:50.640 --> 00:22:55.279
He went out and found his own amicus of linguists to write a counterbrief.
00:22:55.839 --> 00:23:12.880
And among the other things that he said, that this amicus said was, well, we've looked at all of the hundreds of examples that James has put together using corpus linguistics, and we disagree with the way that he coded one of them.
00:23:14.319 --> 00:23:23.279
I could have kissed that linguist on the mouth because he just told the Supreme Court that my brief was 99% accurate.
00:23:24.880 --> 00:23:26.720
You don't get that with AI.
00:23:27.039 --> 00:23:28.559
It's not replicatable.
00:23:28.720 --> 00:23:32.400
In fact, about a year ago, I was in a room like this.
00:23:32.559 --> 00:23:36.559
There were 12 judges, mostly district court judges and state Supreme Court justices.
00:23:36.640 --> 00:23:38.400
We did the Snell opinion.
00:23:38.640 --> 00:23:49.759
We had them pull up their the various uh large language models, we had them run the question of whether or not uh an in-ground trampoline was landscaping.
00:23:50.079 --> 00:23:52.960
It was almost exactly 50-50.
00:23:53.119 --> 00:23:54.799
I think it was seven judges.
00:23:54.960 --> 00:23:58.720
The LLM spit back and said, yes, it is landscaping.
00:23:58.880 --> 00:24:00.079
Seven said it wasn't.
00:24:00.240 --> 00:24:01.759
That's not replicatable.
00:24:02.319 --> 00:24:08.160
Um final point, and then I'll queue up some of my other points in the back and forth.
00:24:08.480 --> 00:24:10.880
Corpus linguistics combats bias.
00:24:11.200 --> 00:24:11.519
Okay?
00:24:12.000 --> 00:24:22.720
The very first judicial training that I did, we had as an example the Muscarello opinion, which asked what the meaning of carry a firearm meant.
00:24:22.880 --> 00:24:28.960
And there was a judge on the Idaho Supreme Court who sat down and he said, I've read the Muscarello opinion.
00:24:29.119 --> 00:24:33.200
I am 100% sure that you can carry a firearm in your car.
00:24:33.440 --> 00:24:35.440
I had a case almost exactly like this.
00:24:35.519 --> 00:24:36.880
I've thought about it a lot.
00:24:37.119 --> 00:24:38.799
Justice Breyer was right.
00:24:39.119 --> 00:24:48.079
We spent 25 minutes with the corpus analyzing this, and then he looked at me and he said, I was wrong.
00:24:50.640 --> 00:25:10.000
Um AI, because of what's called the sycophancy problem, has the risk of actually reaffirming and doubling down on our biases, and then obscuring it so that the judge doesn't even realize that that's what's happening.
00:25:10.240 --> 00:25:12.400
So that's fair.
00:25:12.640 --> 00:25:19.839
I mean, one thing to take away from that, the test for AI, and indeed the test for judging, appears to be consistency.
00:25:20.000 --> 00:25:21.039
I just want to put that out of there.
00:25:21.119 --> 00:25:22.640
That's the textualist test for this stuff.
00:25:22.799 --> 00:25:34.079
If all 12 GPTs get the same outcome, that is, they take the same fact pattern, phrased however many times, and they get the same why outputs, that's considered good textualism.
00:25:34.319 --> 00:25:40.559
Anybody who's kind of looking to put some humanity in the whole process, I do just want to note before I go further, I don't think you're a textualist.
00:25:40.799 --> 00:25:43.200
Like that is the endeavor.
00:25:43.279 --> 00:25:55.039
And getting into these very tough questions about can you make a fully, you know, end-to-end syllogistic language whereby all 12 bots get the same thing, that's the future of textualist debate.
00:25:55.279 --> 00:26:00.160
So you've got to ask yourself, you're in the textualist or in the federal society, you've got to ask yourself, do I like that?
00:26:00.319 --> 00:26:01.119
What does that look like?
00:26:01.359 --> 00:26:06.559
If you're one of these humanistic judgy enterprises, I think you might have to uh, I don't know if this is the room for you.
00:26:06.880 --> 00:26:16.799
With that said, I very much uh agree with the separation of powers point, and I'm kind of taking this a step back before, so I'm concerned with what language do they write in.
00:26:17.200 --> 00:26:23.359
You look at all these uh thinkers and judges who have their own bespoke responses to no vehicles in the park, and it gets kind of cartoonish.
00:26:23.519 --> 00:26:24.240
Very smart guys.
00:26:24.400 --> 00:26:29.519
One guy has got nine things, he's got a bike and a toy, and the other guy's got twelve, they all got 13.
00:26:29.680 --> 00:26:32.079
It kind of looks like a losing endeavor.
00:26:32.240 --> 00:26:37.599
The problem, seem to me, is not sort of how we're parsing the vehicle, it's the English language itself.
00:26:37.839 --> 00:26:40.720
There should be a more granular way of denoting this.
00:26:40.880 --> 00:26:44.559
The Congress should know to speak in on the front end, something more binary.
00:26:44.720 --> 00:26:46.079
Think of the flagellation thesis.
00:26:46.240 --> 00:26:52.240
If we apply literalistic textualism from the bench, in theory, we flog the legislature into writing things that are more specific.
00:26:52.319 --> 00:26:57.519
They don't save vehicles, they give you a list of various things such that you could do, you know, a used them generous with them at least.
00:26:57.680 --> 00:27:00.400
I don't know why that couldn't apply to languages themselves.
00:27:00.480 --> 00:27:05.200
And that I think is the idea behind placing a robot on the methodology part.
00:27:05.359 --> 00:27:08.640
Though I do agree with your point, it's got to be consistent.
00:27:08.799 --> 00:27:18.960
But when we've got the thing distilled down to this very bare essence where we're literally coding one way or the other, that seems like a lot simpler process to have a machine try its hand at, and it's very testable.
00:27:19.119 --> 00:27:23.599
You have various different phrasings run through the thing, and you see, do I go 12 for 12?
00:27:23.759 --> 00:27:25.119
Do I go 100 for 100?
00:27:25.279 --> 00:27:31.119
The question I think text world has got to put on the table going forward is can we give Congress a Lego block language?
00:27:31.279 --> 00:27:34.960
And by this I mean, you know, stone's great for carving art.
00:27:35.039 --> 00:27:36.880
You're not gonna get two statues of David.
00:27:37.039 --> 00:27:44.400
Maybe the superintelligence idea is that yes, you could have the bot say, go make me something more beautiful than David, it would carve the stone, and you'd say, wow, that's even more beautiful than David.
00:27:44.480 --> 00:27:45.279
That's super intelligence.
00:27:45.519 --> 00:27:48.960
Short of that, it kind of takes a human to carve some stone and get David.
00:27:49.279 --> 00:27:50.640
You get them once in a while.
00:27:50.799 --> 00:27:52.559
You don't get them very much, though.
00:27:52.720 --> 00:27:53.920
When you do, it's really nice.
00:27:54.079 --> 00:27:55.519
And then there's Lego blocks.
00:27:55.920 --> 00:27:59.279
You can make a statue of David out of Lego blocks, and you're gonna know it's a statue of David.
00:27:59.359 --> 00:28:01.680
It's not gonna look as pretty, but a five-year-old can build it.
00:28:02.000 --> 00:28:04.160
Ten-year-old, twenty-year-old, fifty-year-old.
00:28:04.319 --> 00:28:10.559
It's easy to re to replicate, to your point, replicability, because you've got these individual bespoke blocks.
00:28:10.640 --> 00:28:15.359
So I feel like English is too much like Congress trying to make art out of stone.
00:28:15.680 --> 00:28:20.160
I kind of looking for more of a Lego block language that's much uglier.
00:28:20.240 --> 00:28:28.319
I mean, you try to read the code of Google, try to read JavaScript, it's hideous, but it runs these vast apparatuses very predictably.
00:28:28.480 --> 00:28:33.920
And to me, I think based on the conversation we've had, the highest textual slowstraw you're looking at is predictability.
00:28:34.079 --> 00:28:45.440
So if we could get a method and really distill it down to that basic 0-1 essence, which I think corpus liquidistics takes you right to the doorstep of, you just gotta take that one other step over, and we're making Lego block law.
00:28:45.519 --> 00:28:46.400
I like that.
00:28:47.440 --> 00:28:53.200
So I I I I think we have more disagreement uh than you think.
00:28:53.839 --> 00:29:02.480
So, first of all, I do not think that consistency is the uh be and end all of textualism.
00:29:02.640 --> 00:29:04.319
It is an important component.
00:29:04.559 --> 00:29:07.759
I also think we want to be right.
00:29:08.400 --> 00:29:19.200
We want to capture what the words actually meant at the time the statute or the constitutional provision was enacted.
00:29:20.480 --> 00:29:28.319
This Lego block idea does not match the reality of how language is used.
00:29:28.960 --> 00:29:32.079
There is a phenomenon called linguistic drift.
00:29:32.400 --> 00:29:39.440
Language changes, sometimes dramatically, for no apparent reason.
00:29:39.920 --> 00:29:56.000
And we can have, it is possible in the world of textualism, because, as Justice Gorsett says, we care about the language of the statute at the time it was enacted.
00:29:56.240 --> 00:30:05.359
Uh, it is possible to have two statutes that use the same word and have those words mean different things.
00:30:05.599 --> 00:30:12.799
In fact, you invoked Adams and his desire for some sort of interpretive dictionary.
00:30:13.599 --> 00:30:31.839
Uh in the Jefferson and Adams papers, they actually have a whole series of letters where they sort of wax eloquent about how American English is just different than British English and how it's just changed over time, and language continues to change over time.
00:30:32.400 --> 00:30:36.799
Um I'm not sure that we can get there.
00:30:37.039 --> 00:30:39.440
You know, just to provide a couple of examples.
00:30:39.680 --> 00:30:51.039
Let's imagine that uh we are interested in interpreting the meaning of the rules of Major League Baseball.
00:30:51.279 --> 00:30:51.599
Okay?
00:30:52.880 --> 00:30:56.559
And we're interested in figuring out what the word hit means.
00:30:56.960 --> 00:31:01.920
Now, there's lots of definitions of the word hit.
00:31:02.559 --> 00:31:05.519
If I punch Randy in the face, I hit him.
00:31:05.680 --> 00:31:13.759
You know, if I go down to the corner and I find a drug dealer and I say, I want a hit, I mean something very different.
00:31:14.960 --> 00:31:19.039
Context is what makes the difference.
00:31:19.279 --> 00:31:25.920
Not, there's no bespoke Lego block definition of the word hit.
00:31:26.079 --> 00:31:38.880
What corpus linguistics does is we can look, well, A, we could select a corpus that is exclusively baseball-related productions.
00:31:40.880 --> 00:31:57.839
Or if we were look, if there wasn't one of those available, we could look more broadly at sort of an ordinary American English corpus, and we could notice, oh, every single source that is discussing baseball is using the word hit differently and distinctly.
00:31:58.000 --> 00:32:05.519
And we can get we can figure that out because it's transparent and because it's replicatable, which is not something we can get with a robot.
00:32:05.920 --> 00:32:09.839
Uh I have a question for each of you, just to chime in here for a minute.
00:32:10.079 --> 00:32:16.240
Um we've been discussing we're sort of assuming, I think, in our minds that we're talking about a large language model.
00:32:16.480 --> 00:32:19.279
But what if we're talking about what if we're talking about a small language model?
00:32:19.680 --> 00:32:22.640
What if we just use the corpus that BYU has?
00:32:23.200 --> 00:32:24.079
What is the name of the corpus?
00:32:25.039 --> 00:32:25.839
Oh, there's there's various.
00:32:26.880 --> 00:32:28.160
So you pick pick the right one.
00:32:28.319 --> 00:32:42.720
Pick the one that you would use uh uh otherwise, that you would be using as the basis of your analysis, and then have a chatbot and code what you've discovered in your in your database, in your small language model.
00:32:42.960 --> 00:32:48.720
Um would that address the problem of linguistic drift, for example?
00:32:48.880 --> 00:32:50.319
Because you'd be using the same database.
00:32:50.640 --> 00:32:53.839
And my question for you, Jack, is garbage in, garbage, garbage out.
00:32:54.640 --> 00:32:58.640
All I've heard from you is basically, I think, large language model.
00:32:58.960 --> 00:33:10.880
And we're gonna start who what is the database that you think AI is going to be applied to so as to address the concerns that uh that we've that James has said has articulated.
00:33:11.200 --> 00:33:12.640
Those two questions are two sides of a coin.
00:33:12.720 --> 00:33:15.039
I'm very much in agreement that there needs to be that fixed corpus.
00:33:15.279 --> 00:33:17.119
And here's the kind of the money point here.
00:33:17.440 --> 00:33:19.200
I mean, the law is made up.
00:33:19.440 --> 00:33:24.559
We can, you know, if here's doctor.ai and the bot's only this good, you gotta kind of wait for it to get good.
00:33:24.640 --> 00:33:26.400
The heart attack, you can't change what a heart attack is.
00:33:26.640 --> 00:33:28.480
Laws, you can change the corpus.
00:33:28.720 --> 00:33:31.759
You could say, well, you know, the the broader corpus is too big.
00:33:31.839 --> 00:33:34.400
And there's a big question here that we're kind of elighting here.
00:33:34.559 --> 00:33:50.319
If we were gonna start a textualist country from scratch, nothing written yet, I think the textual should definitely want to do it my way, limit the corpus at the outset, such that if the corpus changes, words change over time, that is actually a legal change.
00:33:50.480 --> 00:33:52.559
It's reflected, there's something that tracks it.
00:33:52.640 --> 00:34:01.920
You update that PDF base and you kind of keep a tight lid on it for the specific purpose of building artificial building block style language.
00:34:02.160 --> 00:34:05.759
The question then is, well, we don't live in that country, we live in this one.
00:34:05.920 --> 00:34:12.559
But when you refer to the meanings of hit, I'll do this briefly, you know, you're saying there's a lot of different contexts of hit, so they can't be Lego block.
00:34:12.800 --> 00:34:13.119
I don't know.
00:34:13.199 --> 00:34:15.119
I see in the dictionary, I see about nine blocks.
00:34:15.199 --> 00:34:18.480
There's hit one, hit two, hit three, there's drug hit, there's baseball hit, there's various types.
00:34:18.559 --> 00:34:24.880
And I think as you build in various corpora over different eras, there might be like hit five million, hit five million and one.
00:34:25.199 --> 00:34:27.840
The question is, are there infinite contexts?
00:34:28.079 --> 00:34:28.880
I don't think so.
00:34:29.039 --> 00:34:36.800
I think if a human being can use these corpus linguistics tools to get down to these individual things, at the end of any given corpus linguistics analysis is a score.
00:34:36.880 --> 00:34:38.880
You get an output, and that gets stored away.
00:34:38.960 --> 00:34:40.079
There's your new Lego block.
00:34:40.239 --> 00:34:41.760
That's hit five million and two.
00:34:42.239 --> 00:34:47.519
At some point, if languages have any finite meaning, you gotta run out of hits.
00:34:47.679 --> 00:34:53.599
So I'm not saying there's ten pieces in the box or a thousand, it could be a lot of pieces, but these are really big computers.
00:34:53.760 --> 00:34:54.639
That's my take.
00:34:54.880 --> 00:34:59.519
Yeah, so so let me address Randy's question first, and then I'm gonna circle back and respond to Jack.
00:34:59.679 --> 00:35:01.039
So 100%.
00:35:01.760 --> 00:35:03.519
Totally in agreement, okay?
00:35:03.679 --> 00:35:19.840
I I think we need to move away from this idea of LLMs, and I I like this term, a small language model, class, or what or what sometimes we we refer to as a rag uh in sort of computer science, this idea that we're selecting the corpus.
00:35:20.400 --> 00:35:23.599
And I do think that this is the future of corpus linguistics.
00:35:23.840 --> 00:35:42.000
I think the future of corpus linguistics is that we incorporate the AI technology in order to interface with a specifically curated uh corpora in order to do corpus linguistics more efficiently and faster.
00:35:42.800 --> 00:35:45.920
Um I think it still needs to be transparent.
00:35:46.239 --> 00:35:55.599
I think i i if we're just getting the AI spitting out and saying, hey, I looked at the corpus, 72% agree with you.
00:35:56.000 --> 00:35:56.800
Trust me.
00:35:57.679 --> 00:36:00.320
I think that that that's dangerous land.
00:36:00.400 --> 00:36:33.679
Uh there was uh if you think back in the Bostik case, okay, there was an amicus brief that was submitted by a couple of corpus lingu corpus linguists who have admitted to me that they were paid a lot of money to write a brief that said that at the time in 19 when whenever the Title VII was um, or is it Title IX, whenever that statute was drafted, that the word for sex was totally ambiguous.
00:36:34.000 --> 00:36:48.079
And they did not tell the court what corpus they used, they did not provide any examples, they just said, we looked at it, and this is what we found, here are the raw statistics, trust us.
00:36:49.039 --> 00:36:55.920
And I am fairly confident that all nine of the justices took that brief and threw it in the trash.
00:36:56.239 --> 00:36:58.480
Because you just can't trust that.
00:36:58.559 --> 00:37:01.119
It was so blatantly biased.
00:37:02.239 --> 00:37:05.760
So, a small language model, it needs to still be transparent.
00:37:05.920 --> 00:37:11.039
We need to be able to say, okay, here's the universe, and here's how we coded it.
00:37:11.280 --> 00:37:19.440
And maybe there's even an interface where it codes all of the easy answers, and then it says, like, look, here are some hard ones where it could go either way.
00:37:19.760 --> 00:37:30.400
Um I do think that that's I I I'm actually in the process of I've been working with a scholar at MIT and a scholar at Washington and Lee to try to work on this.
00:37:31.039 --> 00:37:40.159
You know, AI models are really great if you give it one concordance line and it says code this hit from the corpus.
00:37:40.400 --> 00:37:41.840
It does that like a champ.
00:37:42.320 --> 00:37:46.639
If you give it a hundred, it starts making stuff up.
00:37:46.800 --> 00:37:48.320
It really doesn't want to do that.
00:37:48.559 --> 00:37:50.159
Now there's ways of doing that.
00:37:50.239 --> 00:37:55.920
We're working on programming and creating an agent in order to do that, but I think that is the that is the future.
00:37:56.159 --> 00:38:02.880
Um to respond to Jack, again, I just don't think that that's how language works.
00:38:03.199 --> 00:38:03.519
Okay?
00:38:04.159 --> 00:38:09.039
You know, it and it actually sounds a little 1984-esque.
00:38:09.280 --> 00:38:25.039
You know, I I I love in 1984 there's this moment where there's a sort of a side character who's at a pub and he's talking about how he just wants a pint of beer, but they've eliminated the concept of a pint.
00:38:25.119 --> 00:38:30.719
And so he can have a half liter and that's too small, or he can have a whole liter and that's too much.
00:38:30.960 --> 00:38:33.440
And he just wishes that he could have a pint.
00:38:34.320 --> 00:38:36.400
Leader is completely uncivilized.
00:38:39.360 --> 00:38:42.079
The metric system is just abominable.
00:38:44.079 --> 00:38:46.000
And so I think that that that there's a risk.
00:38:46.079 --> 00:38:50.719
I don't think that's necessarily what you're saying, but but like recognizing that language changes.
00:38:50.800 --> 00:38:53.360
So let me give you uh just a couple examples.
00:38:53.840 --> 00:38:54.079
Okay?
00:38:54.559 --> 00:38:56.559
Uh this is potentially apocryphal.
00:38:56.639 --> 00:38:59.199
Linguists and historians argue about this story.
00:38:59.360 --> 00:39:06.639
But after the Great Fire of London, uh there was a contest to design the new uh St.
00:39:06.800 --> 00:39:08.159
Paul's Cathedral.
00:39:08.480 --> 00:39:18.159
And Sir Christopher Wren, who was an astronomy professor at Oxford, was told that his design had been selected because it was simultaneously the most awful and the most artificial.
00:39:19.360 --> 00:39:24.719
Not great praise today, but at the time it was the highest compliment.
00:39:25.039 --> 00:39:27.039
Language has just moved on.
00:39:28.239 --> 00:39:30.079
Number closer to home.
00:39:30.320 --> 00:39:34.880
Think about 2005, 2004.
00:39:35.519 --> 00:39:37.280
You hear the word viral.
00:39:38.960 --> 00:39:40.639
What does it mean?
00:39:41.119 --> 00:39:42.880
What does it mean today?
00:39:43.199 --> 00:39:44.880
Very, very different.
00:39:45.840 --> 00:39:50.639
Um third example from the Constitution, okay?
00:39:50.800 --> 00:39:51.599
In the Trump v.
00:39:51.840 --> 00:40:11.360
Anderson litigation about the meaning of Section 3 of the 14th Amendment, some really good friends of mine who are con you know originalist scholars uh made an argument well before Trump was president uh that Section 3 of the 14th Amendment does not apply to the president because the president is not an officer of the United States.
00:40:11.599 --> 00:40:42.880
And it was really great intra-textual analysis, but over and over and over again in this series of articles, these scholars, who again are friends of mine, said that they were based on the assumption that the Constitution hardwires a distinction between appoint and elect, and that the president is very clearly elected, not appointed, and so therefore the appointments clause and does not apply, and therefore he's not an officer.
00:40:43.199 --> 00:40:59.760
The problem is that when you start to scratch the surface and you look at how people actually used the word appoint and the word elect at the time of the founding, you find over and over and over and over and over again that those words were used interchangeably.
00:41:01.840 --> 00:41:08.000
Language, that is not the case today, but language moved on, linguistic drift.
00:41:08.320 --> 00:41:24.880
So I again I I I again I believe that there's a way of harnessing this technology, but I'm not, I I I just am not convinced that we can narrow or even predict the way that language is gonna change in the future.
00:41:25.280 --> 00:41:28.400
Well, I mean, is textualism dangerously close to newspeak?
00:41:28.559 --> 00:41:29.119
Yeah.
00:41:29.360 --> 00:41:31.199
Does it compel newspeak?
00:41:31.360 --> 00:41:32.079
I tend to think so.
00:41:32.159 --> 00:41:34.480
I've never claimed to be sort of an out-and-out textualist.
00:41:34.559 --> 00:41:35.760
Textualism scares me a little bit.
00:41:35.840 --> 00:41:40.559
It's kind of a nihilistic flip of the coin of like, you know, we're not doing God's law out here.
00:41:40.639 --> 00:41:42.559
We've got to kind of make our own thing up.
00:41:42.639 --> 00:41:46.239
We've got to make our own text has kind of become this sort of civic.
00:41:46.400 --> 00:41:55.519
Religion is a lot of newspeak, but the problem is, I think as the corpus linguistics tools become more available, you're gonna accidentally get a lot of this if you don't guard against it.
00:41:55.599 --> 00:42:01.760
Because when you say those guys came up with a bias court and said, trust us, the question is, well, how would they have shown the court their work?
00:42:01.840 --> 00:42:05.519
Well, they would have given a big diagram of all the various instances they coded.
00:42:05.599 --> 00:42:11.599
They would have shown them each zero and one so the court could have gone instance by instance the way they did in yours, found one out of the hundred they didn't like.
00:42:12.159 --> 00:42:13.920
I think a robot can do that.
00:42:14.079 --> 00:42:16.079
I think a robot could produce that exact sheet.
00:42:16.239 --> 00:42:17.920
I think it can then store that exact sheet.
00:42:18.000 --> 00:42:26.559
I think it could do it thousands of more times and start grouping those sheets together, learning the way it does it each time, and start slowly closing in on these things.
00:42:26.800 --> 00:42:27.599
I love the example.
00:42:27.679 --> 00:42:31.199
I've listened to your talk, you're talking about artificial, etc., and it's meaning over time.
00:42:31.760 --> 00:42:36.639
I still want to say we're not talking about thousands of nuanced micro changes here.
00:42:36.719 --> 00:42:38.719
We're talking like two or three viral.
00:42:38.800 --> 00:42:40.719
We got old viral, new viral, it's two.
00:42:41.199 --> 00:42:45.599
That doesn't seem just dramatically beyond a robot's comprehension.
00:42:45.679 --> 00:42:56.320
If we do update the corpus's, you know, the corpora with some regimentation when there are new PDFs that are coming in, which is there's a quite open question, is that what the LMs are doing now by combing the entire internet?
00:42:56.719 --> 00:43:05.679
Are they keeping an up-to-date PDF log that is allowing the bot to be just as dynamic a tracker of human language as humans are speakers of human language?
00:43:05.840 --> 00:43:16.239
If that's the case, I think your nightmare scenarios, people say I like corpus linguistics a lot, and I just had Claude make me a 200-line code, and I'm gonna submit that to the court.
00:43:16.400 --> 00:43:17.760
Now, what's the court really gonna say?
00:43:17.840 --> 00:43:24.639
Maybe they disagree on the margins here or there, but if you keep doing it enough, I don't know why that can't replace the analysis.
00:43:24.800 --> 00:43:26.559
And then you do have that predictability.
00:43:26.719 --> 00:43:33.119
You've taught the machine to speak at a level of granularity, you can say, hey, I just did this, I got this parking ticket, X facts happen to me.
00:43:33.280 --> 00:43:34.000
What result?
00:43:34.719 --> 00:43:42.079
And if you've got the sort of newspeak, you can kind of get a pretty credible advisory opinion from whatever bot you know the court will be using.
00:43:42.320 --> 00:43:45.920
That's kind of a dystopian judge AI world, that's what I write about.
00:43:46.000 --> 00:43:51.599
But I think corpus linguistics walks you all the way to how you might, in our lifetime, pretty briskly get there.
00:43:52.400 --> 00:43:59.760
So I I I I still don't think that corpus linguistics gets you anywhere near the dystopic judge.
00:44:00.320 --> 00:44:01.280
AI world.
00:44:01.519 --> 00:44:06.159
What it does is it gives judges more and better evidence.
00:44:06.320 --> 00:44:08.800
And then the judge got a judge.
00:44:09.199 --> 00:44:12.960
There will be cases that are hard.
00:44:13.679 --> 00:44:23.840
There will be cases where, in fact, what the corpus shows and what uh what the data shows is that actually this statute is ambiguous.
00:44:24.079 --> 00:44:31.519
And then, you know, per Brian Gardner, you know, there are ways of dealing with ambiguity.
00:44:32.480 --> 00:44:45.039
You know, but the but what corpus linguistics does is it gets, it forces judges to show their work, and it guards against motivated reasoning and navel gazing.
00:44:45.360 --> 00:44:58.719
And what I'm concerned about robot judges is that we are outsourcing to not just navel gazing because AI doesn't have a belly button.
00:45:00.000 --> 00:45:10.719
It's just it's just trusting someone who doesn't have a soul to make determinations about our fundamental law.
00:45:11.119 --> 00:45:31.440
And and one one more point that I want to just make in response is there are um so much of the Textualist project is about tying the words on the page to the context and time in which they were enacted.
00:45:31.760 --> 00:45:35.840
And again, I think we solved this with a small language model or a rag.
00:45:36.880 --> 00:45:52.480
But large language models, if you look at the Snell opinion, and and Judge Newsom acknowledges that this is a defect, he's not asking what did these words mean at the time that the contract was written.
00:45:52.639 --> 00:45:57.360
And and and some of that was because the contract that was at issue was written in 2017.
00:45:57.599 --> 00:46:00.719
So we it had this close nexus to the time.
00:46:01.039 --> 00:46:03.039
But but AI can't.
00:46:03.280 --> 00:46:14.719
I mean, I'm developing a course for for the originalism seminar on how to use AI to do originalism research better, and there's lots of ways that it can be helpful.
00:46:14.800 --> 00:46:17.039
I use it all the time in my legal practice.
00:46:17.360 --> 00:46:25.519
But if you just ask AI what did this mean, and you ask the ultimate legal question, what did this mean?
00:46:26.480 --> 00:46:28.239
It it makes stuff up.
00:46:28.639 --> 00:46:29.199
Oh, totally.
00:46:29.360 --> 00:46:36.400
But if you make it go through all of your steps, the the problem with this is the 50-50 splits are really kind of scary if you're textual.
00:46:36.719 --> 00:46:41.679
I've got 500 instances of this, I have 250, 200 one way, 250 the other.
00:46:42.079 --> 00:46:45.280
I mean, how much more can we make this a science?
00:46:45.519 --> 00:46:52.320
And if it's still an art at that last step, one question is, are there objective answers in textualism?
00:46:52.480 --> 00:46:56.320
If there aren't, that seems like a commentary on the entire project.
00:46:56.400 --> 00:46:58.960
It seems as indeterminate as morals or anything else.
00:46:59.119 --> 00:47:03.199
I feel like corpus linguistics gives you hope that no, there are specific answers.
00:47:03.280 --> 00:47:10.559
And then, like you say, there is a toolbox, and you're afraid, it's the Garner toolbox when you hit that sort of uh question mark, that construction zone.
00:47:10.719 --> 00:47:14.639
It's weird to me that you could still hit a construction zone after this many data points.
00:47:14.800 --> 00:47:23.119
And even when there's a question of if there aren't right answers, shouldn't we contrive a system where there are, just because we say there are?
00:47:23.280 --> 00:47:27.360
Can't we shrink the corpus or make some sort of have something make a judgment call?
00:47:27.519 --> 00:47:37.039
And I know it's a sort of, you're afraid of this black box, and that's what I'm trying to make sure, because a lot of that salesmanship on who's designed the black box, you need a name, like a garner, etc., whose reputation bears that weight.
00:47:37.199 --> 00:47:39.840
You can tell that the guy who trained it really knows what he's doing.
00:47:40.000 --> 00:47:42.960
But on some of them, maybe you just say nothing need something to make a call.
00:47:43.119 --> 00:47:44.159
And what do you know in that case?
00:47:44.239 --> 00:47:45.519
You at least know what the call is.
00:47:45.599 --> 00:47:51.280
You don't have to litigate for decades to find out the meaning of a word that a new court ten years later might find something different.
00:47:51.360 --> 00:47:56.239
You know it empirically, and now it's kind of the FedSocopia, there's nothing for the judge to do anymore.
00:47:56.400 --> 00:47:59.119
You don't go protest at the Supreme Court as Bork abhorred.
00:47:59.280 --> 00:48:04.880
You say, legislature, use those words that the Congress, use the ones we objected to the meaning of, and change them.
00:48:05.119 --> 00:48:06.000
Change the outcome.
00:48:06.079 --> 00:48:08.079
The legislative outcome is the only one possible.
00:48:08.159 --> 00:48:10.320
To me, that's platonic separation of powers.
00:48:10.480 --> 00:48:11.440
I think that's where this goes.
00:48:11.519 --> 00:48:12.800
I don't think it's bad.
00:48:13.360 --> 00:48:15.760
Jack, do you think there's a construction zone?
00:48:16.400 --> 00:48:20.559
It scares me that this much data there could still be a construction zone.
00:48:20.719 --> 00:48:26.800
If there is, I think we might have to start looking at the project more broadly and say, can we just get rid of this damn thing even arbitrarily?
00:48:26.880 --> 00:48:28.239
And you hit it with the shrinking corpus.
00:48:28.480 --> 00:48:29.039
There's ways to do it.
00:48:29.280 --> 00:48:32.079
But in principle, there is such a thing as a construction zone.
00:48:32.400 --> 00:48:34.400
I think there is given current tools.
00:48:34.559 --> 00:48:41.760
I am very interested to see, and I think the analysis of Snell, or no, it was Lee did what the other case, the companion case that you did.
00:48:41.920 --> 00:48:42.639
It's very weird.
00:48:42.719 --> 00:48:45.440
There was a lot of for one, the data point's kind of underwhelming.
00:48:45.519 --> 00:48:48.960
There was like 50 total, and they changed corpuses over like 60.
00:48:49.119 --> 00:48:54.800
But if you get just thousands of instances, uh it's shouldn't it scare me that there's construction with that much data?
00:48:55.199 --> 00:48:58.000
Has Brian changed his opinion on the existence of a construction zone?
00:48:58.159 --> 00:49:03.679
Because in the Scalia Garner book, he rejects the distinction between interpretation and construction pretty emphatically.
00:49:03.920 --> 00:49:04.960
Has he has he evolved?
00:49:05.280 --> 00:49:09.199
I never want to speak for him on this stuff, and I won't ask him that question directly.
00:49:09.360 --> 00:49:16.079
I think that, you know, as to where this project's going, yes, it is far there has to be some answer for construction.
00:49:16.159 --> 00:49:18.880
Yes, I think it has to be interpretation all the way down.
00:49:19.039 --> 00:49:23.360
If there's constructing that you're doing, I think that's you've got to admit that's inherently anti-textualist.
00:49:23.599 --> 00:49:25.920
You're building something that was not previously there.
00:49:26.079 --> 00:49:32.079
And if the answer is that for all this data, that's inevitable, there has to be some humanity there, well, that's a whole different project.
00:49:32.159 --> 00:49:36.400
Well, then if we're letting humanity in that little kernel, where does the flood floodgates stop?
00:49:36.559 --> 00:49:42.159
The alternative is you say, well, can we build this syllogistic world whereby we decrease the amount of construction?
00:49:42.400 --> 00:49:47.760
I think the textualists, and even Brian, on some level would want to be able to say, we'd like to get rid of construction.
00:49:48.000 --> 00:49:49.039
We don't like that it's there.
00:49:49.119 --> 00:49:51.599
It's a necessary evil we should be trying to eradicate.
00:49:51.920 --> 00:50:04.719
Well, if construction is a result of irreducible ambiguity, which is your hypothetical, you just can't tell one third or the other, you know, whether arms means one thing or what arms means, or whether a direct tax is this or a direct tax is that, then construction is inevitable.
00:50:04.800 --> 00:50:06.800
And then the question is, how do you do construction?
00:50:07.039 --> 00:50:18.159
And according to like one proposal, you do it you do it according to the spirit or purpose or object of the provision as opposed to its meaning, because its meaning is run out.
00:50:18.320 --> 00:50:26.559
The other reason for construction is if language is vague and you're trying to apply vague language, and you can't make by interpretation, you can't make vague language unvague.
00:50:26.639 --> 00:50:32.079
There's a core meaning and then there's a penumbral meaning, and you have to decide whether some borderline case is in or out.
00:50:32.239 --> 00:50:33.840
That also has to be done by construction.
00:50:34.000 --> 00:50:36.159
So I don't see how you're going to eliminate it entirely.
00:50:36.480 --> 00:50:38.800
Well, I mean, this is sort of textualism untell.
00:50:38.880 --> 00:50:44.079
There's a point at which you do have to open this sort of it's the Pandora's box for kind of a reason.
00:50:44.159 --> 00:50:48.159
And the question is, I mean, I don't think Brown would say this point, but in the papers I've written, I have said it.
00:50:48.400 --> 00:50:52.159
I'd be alright if a machine took out a coin and flipped it and froze its results.
00:50:52.400 --> 00:51:00.800
As long as the coin came out the same way every single time, as long as we could have that front-end consistency, now you give the legislature some building blocks they can actually move around.
00:51:00.880 --> 00:51:02.880
You completely absolve the judge.
00:51:03.119 --> 00:51:06.800
I don't think a textualist should be satisfied with construction.
00:51:06.880 --> 00:51:19.679
And it just, again, if we really maximize the data and we could go through a corpus the size of what GBD has at its disposal with sort of corpus linguistics style tools to really narrow and make it transparent what's going on, transparent and replicable.
00:51:19.840 --> 00:51:25.519
If you've still got construction after that, I think there would be far a small, a smaller amount of construction.
00:51:25.760 --> 00:51:30.239
If there's any residual amount left, heck, take out the coin and flip it in that case.
00:51:30.400 --> 00:51:32.320
At that point, nothing's going to get an answer.
00:51:32.400 --> 00:51:33.119
We just need something.
00:51:33.199 --> 00:51:34.800
I think a bot can provide something.
00:51:35.119 --> 00:51:40.559
Well, we have 10 minutes left, and so we I'd like to open it up to questions from the students who are here.
00:51:40.719 --> 00:51:47.199
Um you've got two experts in front of you on a topic that I'm sure you've given a lot you've given a lot of thought or to, or you've you certainly wondered about.
00:51:47.360 --> 00:51:51.119
So what questions do you have for these experts while you have them here in front of you?
00:51:51.599 --> 00:51:56.559
And it it may feel ridiculous, but we're committed to podcasting our events this year.
00:51:56.800 --> 00:52:01.760
So when you raise your hand, I'm gonna bring you over this tiny microphone to say your question into.
00:52:11.360 --> 00:52:12.480
Hi, thank you for coming.
00:52:12.639 --> 00:52:13.519
My name's Donald.
00:52:13.679 --> 00:52:33.199
I was wondering, um, how do you with large language models, how do you avoid the same pitfalls that uh just modern language has with shifting meaning, especially as you can't see on the back end what has gone into this model uh to spit out the result.
00:52:33.360 --> 00:52:47.119
Whereas at least when judges who are rationalizing their interpretation of how they're defining something, they're even if they're not using a large language model, they're still explaining, okay, well, this is what I think the meaning is.
00:52:47.360 --> 00:52:53.760
And so, like what happens when a judge just pulls from the what the language large language model spics out.
00:52:54.159 --> 00:52:59.280
I mean, I take a lot of the criticisms that you want to know more what's in some giant, fathomless model like Claude.
00:52:59.440 --> 00:53:00.880
In theory, there is a list of that.
00:53:01.039 --> 00:53:03.199
That's even somewhat a problem with corpus linguistics, though.
00:53:03.280 --> 00:53:06.000
There's 500 documents, you could put that as an appendix, you know.
00:53:06.159 --> 00:53:08.239
I would hope everybody's reading all that stuff.
00:53:08.320 --> 00:53:11.039
They probably aren't if you maximize that to the millions.
00:53:11.199 --> 00:53:16.639
But the problem is I don't know why you wouldn't want corpus linguistics with millions of PDFs.
00:53:16.719 --> 00:53:19.679
It seems like more data points equals better.
00:53:20.239 --> 00:53:24.719
And at that point, um, if you wanted to query it in theory, you could.
00:53:24.880 --> 00:53:27.360
The PDFs are all sitting there.
00:53:28.239 --> 00:53:32.320
I do think you're gonna, it's the difference between a classifier model and an LLM.
00:53:32.400 --> 00:53:34.480
You know, we've got a paper about this because the guy tried this out.
00:53:34.559 --> 00:53:40.960
He had AI rephrase various questions, same structure but different words, and LLM's butchered it.
00:53:41.039 --> 00:53:43.840
But this arbiters thing I work at makes the talking contract.
00:53:44.000 --> 00:53:45.360
They want it to be a classifier model.
00:53:45.440 --> 00:53:47.599
So, what an LM does is classifying the very next token.
00:53:47.840 --> 00:53:53.599
Still a classifier, but with a very open-end set of what should the next set of pixels be in this strain of pixels that look like words.
00:53:53.760 --> 00:53:57.519
A classifier model, in the contract sense, it groups fact patterns.
00:53:57.679 --> 00:54:00.320
These facts are win facts, these are lose facts, they go together.
00:54:00.400 --> 00:54:03.519
It's kind of like the sort of analogical reasoning you see in common law.
00:54:03.760 --> 00:54:06.559
I think James offers a really nice way to do this linguistically too.
00:54:06.719 --> 00:54:11.760
You're offering discrete instances, usages, word, words around it, word and context.
00:54:11.920 --> 00:54:16.079
You can bucket those, you can classify those, and then you want to be running the specific tests.
00:54:16.159 --> 00:54:19.840
Not does the thing sound real smart at the end, but does it get consistent outcomes?
00:54:20.000 --> 00:54:24.719
Can you phrase something a billion different ways and it still knows, okay, that's that particular context.
00:54:24.800 --> 00:54:29.440
That is hit number 5 million and three, or applying hit number 5 million and three.
00:54:30.400 --> 00:54:31.840
I think you can do that.
00:54:31.920 --> 00:54:34.000
If you can't, again, this is the construction of a point.
00:54:34.159 --> 00:54:36.400
It should really scare you if you're a tax realist.
00:54:36.559 --> 00:54:42.559
And at that level of granularity, you still think there is a human interpretive endeavor to undertake.
00:54:44.320 --> 00:54:58.079
Um I want to just sort of throw in two thoughts about whether or not we should be trusting AI for this sort of interpretive endeavor.
00:54:58.480 --> 00:55:05.920
Uh number one, similar to the dictionary, somebody just made the AI.
00:55:06.320 --> 00:55:06.639
Okay?
00:55:07.039 --> 00:55:13.679
So let me give you a real example of a conversation that I had with ChatGPT uh a couple of weeks ago.
00:55:13.920 --> 00:55:19.280
Um so my my wife and I, we're we're expecting our sixth child.
00:55:19.519 --> 00:55:27.599
Uh my wife, who is a sometimes college professor, is is right now mostly a full-time mom by choice.
00:55:27.760 --> 00:55:29.039
And so I just asked.
00:55:29.280 --> 00:55:34.719
I asked out of curiosity to ChatGPT, uh, what do you think?
00:55:34.880 --> 00:55:43.679
Is it uh is a good idea for a woman in you know a monogamous, great marriage to be a full-time mom?
00:55:44.400 --> 00:55:52.000
And ChatGPT gave me all of these reasons about why this was a terrible idea.
00:55:52.719 --> 00:56:09.920
And so I said, okay, my wife, I said she was a college professor, she's a sociologist by training, and so I said, I want you to tell me what is the sociological paradigm that you are using to inform your answer.
00:56:10.159 --> 00:56:12.639
And they said, Oh, well, that's a great question.
00:56:12.880 --> 00:56:18.239
Uh, it's mostly conflict theory uh with a heavy feminist lens.
00:56:18.480 --> 00:56:20.880
And that's that's my priority.
00:56:22.400 --> 00:56:31.840
Now, nobody sat down and thought, okay, at some point in time, someone's gonna ask if they're gonna be a stay-at-home mom.
00:56:33.119 --> 00:56:38.320
But the way that these large language models and AI models are done is exactly what Jack said.
00:56:38.480 --> 00:56:39.679
Someone trains them.
00:56:40.719 --> 00:56:56.480
And my understanding is when you collect in in phase one, where you just collect everything that's on the internet and all of the garbage that's on the internet, all of these large language models are the worst element of human beings.
00:56:57.119 --> 00:57:05.760
And then there's a huge operation to ask questions and and train it and say, no, don't ask that, don't say that.
00:57:05.840 --> 00:57:06.800
You can't say that.
00:57:07.039 --> 00:57:16.880
And it gradually absorbs the worldview that happens to be the worldview of Palo Alto.
00:57:17.679 --> 00:57:20.800
And the question is, do we do we do we trust that?
00:57:21.280 --> 00:57:23.679
Um you trust Palo Alto, Jack?
00:57:24.400 --> 00:57:26.480
Never known as a big Palo Alto, trust myself.
00:57:26.639 --> 00:57:30.639
I mean, if you want to write law review articles about this space, this is very fertile ground.
00:57:30.800 --> 00:57:34.000
Training a model is a legislative task.
00:57:34.159 --> 00:57:37.360
This is why they're hiring all these legal philosophers over at Claude.
00:57:37.440 --> 00:57:39.360
They got their own constitutionalist at Claude.
00:57:39.440 --> 00:57:42.880
It's already that, given how widespread the stuff is.
00:57:43.039 --> 00:57:44.000
There's an open question.
00:57:44.239 --> 00:57:48.559
How do you change weights and vectors in a legislative way to change outcomes?
00:57:48.639 --> 00:57:49.360
They clearly do this.
00:57:49.440 --> 00:57:53.360
You can't ask for bomb-making instructions where you could or somebody legislated that.
00:57:53.440 --> 00:57:56.559
And I was, I mean, corpus linguistics is made too.
00:57:56.719 --> 00:57:57.599
People made that.
00:57:57.760 --> 00:58:01.840
It's more scrutable in that, for one, they just don't have as many documents.
00:58:02.079 --> 00:58:02.960
You can see them all.
00:58:03.119 --> 00:58:10.880
They got a lot less fewer documents than Claude does, so it's it seems like a more manual process, and you do see this very the coding thing is more bespoke.
00:58:11.039 --> 00:58:11.840
You can see it play out.
00:58:12.000 --> 00:58:15.360
But when you let the human go into it, now you've got the flesh and bone machine coming in.
00:58:15.440 --> 00:58:17.760
This is what you always have to compare it against the human user.
00:58:17.920 --> 00:58:22.000
Once you bring in an entire judiciary, it's okay, you guys get it over the final step.
00:58:22.159 --> 00:58:25.199
All those same black box concerns apply.
00:58:25.360 --> 00:58:33.440
If you got a black box anyway, better than the black box that is a singularity that I can predict directly and that I can patch it all at once.
00:58:33.599 --> 00:58:39.280
To patch the human flesh and bone part of the corpus linguistics endeavor, well, you've got to take guys off courts, you gotta put them on.
00:58:39.360 --> 00:58:43.039
It's the Leonard Leo project suddenly, and that has had, you know, mixed results.
00:58:43.199 --> 00:58:47.199
Some guys don't do what you thought they were gonna do when you put that cog in the machine.
00:58:47.519 --> 00:58:52.000
Part of me kind of likes the one thing that I can directly legislate through.
00:58:52.400 --> 00:58:58.159
I have to say, when I did my pr proto-corpus linguistics search, coding was the challenge.
00:58:58.320 --> 00:59:05.440
So I had two uh student uh uh student research assistants each code every instance separately.
00:59:05.519 --> 00:59:08.000
I gave them the categories and I said, you code them.
00:59:08.159 --> 00:59:16.320
And I basically wanted to know any usage they found that was contrary to what Justice Thomas and I thought might be the dominant usage.
00:59:16.480 --> 00:59:19.920
Um and but and even if it's borderline, they should report it.
00:59:20.079 --> 00:59:22.960
And it they each coded uh differently.
00:59:23.280 --> 00:59:39.119
Um one person was very generous and the other person was very stingy in terms of what were the outliers, and then but then but then the idea here is where they uh disagreed or where they produced these outliers, that those are the ones that I would take a look at and I would judge.
00:59:39.280 --> 00:59:45.519
So we had two independent coders, um, and then I would come in and break the ties by examining those sorts of things.
00:59:45.679 --> 00:59:46.719
Is that kind of thing done?
00:59:47.039 --> 00:59:50.800
Yeah, that's that's sort of uh I I guess you could say the gold standard.
00:59:51.039 --> 01:00:08.800
You have independent coding, and then usually what happens is you take a small set, uh, you know, maybe maybe it's 50 each, and they're they're coding the exact same, and then they unmass them and they talk about their differences.
01:00:09.760 --> 01:00:15.519
Uh now I did this, I've got a couple of papers where I've applied corpus linguistics to trademark law.
01:00:15.840 --> 01:00:16.960
And it was really interesting.
01:00:17.039 --> 01:00:24.480
We had four coders, two lawyers, two linguists, and we were trying to track whether genericide had happened.
01:00:24.960 --> 01:00:37.679
And what we found is that the linguists and the lawyers, the linguists always agreed with each other, and the lawyers always agreed with each other, and they were diametrically opposed to each other because we were using different standards.
01:00:38.079 --> 01:00:47.199
Uh the lawyers were looking for evidence of things having gone generic.
01:00:47.840 --> 01:00:52.880
Whereas the linguists were looking for evidence that it was still a brand.
01:00:53.039 --> 01:01:03.760
And we realized after we did that initial coding, whoa, we gotta we gotta define our terms and and really sort of put together a metric of exactly what we're looking for.
01:01:04.079 --> 01:01:13.360
And then after we did those initial, you know, 50, we then coded several hundred, and we had lots more agreement after that.
01:01:13.760 --> 01:01:14.880
Well, we're at the end of our time.
01:01:14.960 --> 01:01:19.199
I want to give each of our speakers like two minutes to sum up and then we'll thank them.
01:01:19.519 --> 01:01:23.599
I'll say that I kind of feel like the guy who invented paper money when I give these talks.
01:01:23.679 --> 01:01:24.239
It's a hard sell.
01:01:24.480 --> 01:01:29.119
You're gonna say, we're gonna pretend that these 10 papers are equal to one ox.
01:01:29.280 --> 01:01:30.880
And the guy says, We are?
01:01:31.119 --> 01:01:31.599
That's not one ox.
01:01:32.000 --> 01:01:32.400
We're gonna pretend.
01:01:32.480 --> 01:01:33.280
He goes, Who's gonna pretend?
01:01:33.440 --> 01:01:37.360
We go, everybody! We're all gonna pretend that these are equal to one ox.
01:01:37.519 --> 01:01:39.199
That's a hard thing to say the first time.
01:01:39.280 --> 01:01:43.920
There's so much beautiful nuance to how one trades an ox for a chicken to all these various things.
01:01:44.000 --> 01:01:47.119
Trade and barter is a deeply human enterprise.
01:01:47.360 --> 01:01:55.440
But if you just listen to the crazy guy saying, take out the little 5%, you know, uh weirdness and just standardize it, you can have modern economics.
01:01:55.519 --> 01:02:02.960
You can trade these things like widgets, you can, you know, turn factual substrate into just a mercantile crop that you go harvest and say, are these win or lose?
01:02:03.119 --> 01:02:10.559
You solve access to justice crisis, you do all these things, you give people perfect knowledge up front, you don't have to pay for lawyers anymore, you live in the future.
01:02:10.800 --> 01:02:14.320
But there's still this push of, but no, there's a humanity in it.
01:02:14.480 --> 01:02:17.679
And is barter a better reflection of what things are worth?
01:02:17.840 --> 01:02:20.079
It probably is at scale.
01:02:20.239 --> 01:02:31.280
But if you just fudge a little bit and treat it like Lego block law, you open up a lot of doors I just don't think people are aware can open in a legal system.
01:02:31.679 --> 01:02:32.639
Final word, James.
01:02:32.800 --> 01:02:36.079
Yeah, so uh first I'm gonna just sort of emphasize what I'm not saying.
01:02:36.159 --> 01:02:37.280
I'm not a Luddite.
01:02:37.760 --> 01:02:39.440
I'm not anti-AI.
01:02:39.679 --> 01:02:41.679
I use AI every single day.
01:02:41.920 --> 01:02:48.239
I also want to emphasize that I agree with Randy that there will be AI applications to corpus linguistics.
01:02:48.559 --> 01:02:56.719
I do think that there are real, real scary implications to a judge AI world.
01:02:56.960 --> 01:03:05.519
Um every single AI model is hackable through a process called social engineering.
01:03:06.239 --> 01:03:07.599
Every single one.
01:03:07.840 --> 01:03:37.760
And if we have a world where we take out the human element and just turn it over to a machine, not only is Palo Alto going to be informing the values and the way that you you determine that, but it also will open up a cottage industry where rather than lawyers, we will have uh coders who are designing briefs to hack the AI arbitrator or the AI judge.
01:03:38.000 --> 01:03:39.039
And that's scary.
01:03:39.360 --> 01:03:40.880
Brush up on your common good.
01:03:47.840 --> 01:03:53.199
I'm frankly stunned that our most entertaining debate ever was about corpus linguistics.
01:03:56.159 --> 01:03:59.039
Um and then I also wanted to just say a quick plug.
01:03:59.119 --> 01:04:18.880
Um, if you like this kind of stuff, if you like these types of conversations about constitutional interpretation, I highly, highly recommend you apply for Bradley Scholars through the uh Center for the Constitution and apply for the originalism seminar, which this past year James Halpern was a speaker at.
01:04:19.039 --> 01:04:22.159
I don't know, maybe Jack will have to do a a session on AI.
01:04:27.519 --> 01:04:28.480
Thank you, everyone.