1
00:00:09,786 --> 00:00:14,330
Hello, this is Anthony and Diana from Reed Smith and welcome to Tech Law Talks.
2
00:00:14,330 --> 00:00:18,853
Today we are continuing a podcast series on AI-enabled e-discovery.
3
00:00:18,853 --> 00:00:28,842
This podcast series will focus on practical and legal issues when considering using
AI-enabled e-discovery with a focus on actual use cases, not just theoretical.
4
00:00:28,842 --> 00:00:33,065
Today, joining me are Therese Caparo and Marcian Krieger.
5
00:00:33,065 --> 00:00:37,977
And today we're gonna be discussing the use of AI-enabled discovery
6
00:00:37,977 --> 00:00:40,417
for doc and review and QC.
7
00:00:40,417 --> 00:00:42,657
So just the general topic of doc and review.
8
00:00:42,657 --> 00:00:45,218
So let's just start with Therese.
9
00:00:45,218 --> 00:00:47,198
What is using AI?
10
00:00:47,198 --> 00:00:51,520
What does it look like to use AI for doc and review?
11
00:00:52,183 --> 00:01:02,689
Well, I think one of the challenges with answering this question is there's so many things
that we do in document review that we could probably have several podcasts talking about
12
00:01:02,689 --> 00:01:08,272
all the different ways that you can use GenAI within your document review process
generally.
13
00:01:08,272 --> 00:01:21,305
But what I'm going to focus on today is really just using GenAI for first level, second
level QC reviews for data that you want to produce and what we're seeing
14
00:01:21,305 --> 00:01:25,358
as the opportunities for using gen AI in those processes at a high level.
15
00:01:25,358 --> 00:01:30,521
So I think that, you know, at a high level, there's probably three big bucket categories.
16
00:01:30,521 --> 00:01:34,565
One is just identifying categories of relevant documents.
17
00:01:34,565 --> 00:01:44,070
We know, we think, at the outset of a case, we have a complaint and we have, you know,
requests for production and we know generally what may be relevant.
18
00:01:44,070 --> 00:01:48,463
And we're looking to put together, we're putting together our review protocols and what we
think is going to be relevant.
19
00:01:48,463 --> 00:01:56,547
And a lot of times you find out later on that there are additional categories or things
that you didn't think of and things like that, which is the natural progression in any
20
00:01:56,547 --> 00:01:58,478
legal matter and any document review.
21
00:01:58,478 --> 00:02:09,244
But I think one of the things that GenAI can do is to help us surface those categories of
relevant documents sooner so that the outset of our document reviews, we're in a better
22
00:02:09,244 --> 00:02:11,425
position to say what may be relevant.
23
00:02:11,425 --> 00:02:20,819
what kind of categories out there, things we may not have thought of to help to structure
our review for the next big category of where GenAI can be used, which is in first level
24
00:02:20,819 --> 00:02:22,059
review, right?
25
00:02:22,059 --> 00:02:29,513
Using GenAI to review, so to speak, tag pre-identified documents that are likely to be
relevant, right?
26
00:02:29,513 --> 00:02:34,585
That can be, you know, for your productions and the like, and using that within those
processes.
27
00:02:34,585 --> 00:02:40,465
And then I think, you know, the other third big category is around QC.
28
00:02:40,465 --> 00:02:42,246
of your document review, right?
29
00:02:42,246 --> 00:02:47,690
Whether that is at second level to evaluate our documents being coded consistently, right?
30
00:02:47,690 --> 00:02:48,691
Are there gaps in there?
31
00:02:48,691 --> 00:02:50,512
Are there disagreements between the reviewers?
32
00:02:50,512 --> 00:02:55,846
Are there gaps in the production that we're sending out where we are missing categories of
things, right?
33
00:02:55,846 --> 00:03:06,684
Are things being mis-categorized and things like that to make sure that the productions
after our first level, whether that is human or computer, is complete and accurate and
34
00:03:06,684 --> 00:03:08,947
there's nothing that's being missed?
35
00:03:08,947 --> 00:03:21,045
So I think, you know, really at a super high level, those are three categories that we're
seeing to be able to use GenAI again, specifically for what do I need to produce and
36
00:03:21,045 --> 00:03:24,820
making sure that those productions are complete and accurate.
37
00:03:24,821 --> 00:03:29,125
And Therese, when you're talking about first level review, right?
38
00:03:29,125 --> 00:03:36,180
I think my understanding is that like TAR and others, that they're going to score each
document, like highly relevant, less relevant, whatever.
39
00:03:36,180 --> 00:03:37,311
It's going to be a score.
40
00:03:37,311 --> 00:03:46,349
And I guess one of the issues that people are going to have to think through is how much
you, is it, when we say, when we're saying we're going to use it in first level review,
41
00:03:46,349 --> 00:03:49,121
does that mean we're not going to look at the documents?
42
00:03:49,121 --> 00:03:51,423
And if it's highly relevant, we're just going to produce it?
43
00:03:51,423 --> 00:03:54,025
Or do we think that it's going to be
44
00:03:54,276 --> 00:03:59,148
People are going to still review it, but they're going to see what the scoring is.
45
00:03:59,148 --> 00:04:03,508
Like, what is your view in terms of how this is going to probably play out?
46
00:04:03,881 --> 00:04:05,932
Look, I think it depends on who you talk to.
47
00:04:05,932 --> 00:04:08,055
And I think it's also a matter of timing.
48
00:04:08,055 --> 00:04:14,640
We're at a pretty early stage and using GenAI tools for document review.
49
00:04:14,640 --> 00:04:17,823
There's definitely great use cases out there and people are using it.
50
00:04:17,823 --> 00:04:19,004
So we're at early stages.
51
00:04:19,004 --> 00:04:27,070
I think there are people who will tell you it is so good or will get so good that we will
really be able to forego a first level review.
52
00:04:27,070 --> 00:04:29,111
So there are definitely people who believe that.
53
00:04:29,111 --> 00:04:31,646
I think most lawyers are going to be for now.
54
00:04:31,646 --> 00:04:33,967
a lot more cautious than that.
55
00:04:33,967 --> 00:04:43,212
There will probably still be some level of review of the documents to make sure that
things going out are not just relevant, but not privileged.
56
00:04:43,212 --> 00:04:48,505
We've talked about privilege before in these podcasts to make sure that it's accurate.
57
00:04:48,505 --> 00:04:58,765
You're going to still see some level of review I expect for a while to human review, to
make sure that what is happening, that anything that goes out the door.
58
00:04:58,765 --> 00:05:02,987
is accurately categorized and is not privileged and we're not missing anything.
59
00:05:02,987 --> 00:05:16,122
I think the ideal goal would be to get to a point where we eliminate a lot of human
physically looking at each document and more human time is spent on the prompts and making
60
00:05:16,122 --> 00:05:20,274
sure the process is accurate and validation and things like that.
61
00:05:20,274 --> 00:05:27,519
And using those tools to make sure that the productions are accurate and correct and are
relevant and not, you know
62
00:05:27,519 --> 00:05:29,611
including privileged documents and the like.
63
00:05:29,611 --> 00:05:39,699
But I think as a starting point, we are still going to be seeing some level of human
review as people, you know, we don't quite trust the technology that much, but I do think
64
00:05:39,699 --> 00:05:50,408
the long-term goal, you know, should the technology get us there would be to, you know,
eliminate, largely eliminate manual document by document review.
65
00:05:50,408 --> 00:05:56,052
And instead you're basically having more of a level of technology review to make sure that
the substance is correct.
66
00:05:56,052 --> 00:05:56,466
But,
67
00:05:56,466 --> 00:05:57,784
We're not there yet.
68
00:05:57,785 --> 00:06:04,225
And so, Marcin why don't you explain a little bit of like, how would you actually do this?
69
00:06:04,225 --> 00:06:14,905
Like, how would you actually do the three stages that Therese said if you're using some
type of AI tool, particularly GenAI tool?
70
00:06:14,967 --> 00:06:23,845
All right, so generative AI tools in document review work drastically differently than the
types of technology that we're used to using, which is TAR.
71
00:06:23,845 --> 00:06:39,166
In generative AI, you have to imagine that you have a computer or a virtual room that has
a person that looks at only one document, and you have to write to that one person a set
72
00:06:39,166 --> 00:06:40,357
of instructions
73
00:06:40,357 --> 00:06:48,037
that they can read your protocol and look at that one document and hopefully be 100 %
accurate every time.
74
00:06:48,277 --> 00:06:51,337
What that means is there's no cross document intelligence.
75
00:06:51,337 --> 00:06:59,057
If you have a million documents in your document review population, imagine if you had to
write a document that you will hand to a million people and all they can do is read that
76
00:06:59,057 --> 00:07:05,857
document once and then they look at one document and tell you, it responsive or not to
that memo?
77
00:07:05,857 --> 00:07:09,097
Same thing for privilege, same thing for every issue code.
78
00:07:09,253 --> 00:07:11,694
So all of the work has to happen upfront.
79
00:07:11,694 --> 00:07:21,358
Now, that doesn't mean that you spend a month writing one crazy instruction and then you
just run it and you walk away because that would be incredibly expensive and wasteful.
80
00:07:21,358 --> 00:07:30,222
The way that we set up a generative AI review is that you start by selecting a small group
of documents that you're going to test against.
81
00:07:30,222 --> 00:07:37,465
Now, my best advice on this is you want to have about 30 documents that you absolutely
know are relevant.
82
00:07:37,572 --> 00:07:42,005
You want to have about 30 documents that you absolutely know are not relevant.
83
00:07:42,005 --> 00:07:46,869
And then just to make it rounded out an easy 40 randomly selected documents from your
population.
84
00:07:46,869 --> 00:07:50,633
What you're going to do is you're going to then draft your prompt instruction.
85
00:07:50,633 --> 00:07:56,197
Now, attorneys write and think very differently than generative AI does.
86
00:07:56,197 --> 00:08:01,421
Generative AI is very literal and you also have a limit to the amount of text that you can
put in.
87
00:08:01,421 --> 00:08:03,488
So when we used to write
88
00:08:03,488 --> 00:08:09,300
document review protocols that were 20 pages long and continued, contained every single
nuance in detail.
89
00:08:09,300 --> 00:08:16,882
You have to really distill that drafting process down to clear, near binary instructions.
90
00:08:16,882 --> 00:08:19,542
Here is a description of the case.
91
00:08:19,542 --> 00:08:22,323
Here is a description of what has been requested.
92
00:08:22,323 --> 00:08:24,084
This is what is relevant.
93
00:08:24,084 --> 00:08:29,225
And then it's always best to have a little bit of instruction on this is what is not
relevant.
94
00:08:29,225 --> 00:08:33,110
And it's a lot easier to do it when you're just doing the relevant, not relevant.
95
00:08:33,110 --> 00:08:37,654
There are tools out there that let you do, for example, up to 15 different issue codes.
96
00:08:37,654 --> 00:08:41,186
Each issue code needs to have that same thought process applied.
97
00:08:41,186 --> 00:08:43,760
And I actually have a tip about how to use those issue codes for later.
98
00:08:43,760 --> 00:08:46,582
But you start with your 100 and you test it.
99
00:08:46,582 --> 00:08:50,265
You run the GenAI against this 100 document population.
100
00:08:50,265 --> 00:08:57,801
And what you are looking for is not only that the AI agrees with what you expect to have
happen, but these GenAI tools give you rationales.
101
00:08:57,801 --> 00:09:01,474
They tell you why they think a document's relevant, why they think a document is not
relevant.
102
00:09:01,474 --> 00:09:11,173
You need to be looking at the rationales that AI is generating to make sure it's not
giving you a correct score, but it's rationales completely out of left field because that
103
00:09:11,173 --> 00:09:12,404
means you just got lucky.
104
00:09:12,404 --> 00:09:18,009
You want to make sure that not only is the document relevant, but it's the rationale is
grounded in the prompt that you generated.
105
00:09:18,009 --> 00:09:24,027
If you have to make small adjustments, you do until you get to a point where you say, Hey,
I feel pretty good here.
106
00:09:24,027 --> 00:09:27,487
my 30 documents that were relevant, it says they're relevant and I agree.
107
00:09:27,487 --> 00:09:33,767
The 30 that I self-selected that are not relevant are in fact being said not relevant by
the gen AI and its rationale is accurate.
108
00:09:33,767 --> 00:09:37,947
And within that random sample of 40, I agree with everything it did.
109
00:09:37,947 --> 00:09:41,287
You go from 100 documents to a thousand documents.
110
00:09:41,547 --> 00:09:46,267
Now, do you take the time to validate all 1000 documents?
111
00:09:46,667 --> 00:09:47,327
Probably yes.
112
00:09:47,327 --> 00:09:48,907
Maybe at this point, it's not you and I, Anthony.
113
00:09:48,907 --> 00:09:52,107
At this point, maybe you're using one of your senior eDiscovery attorneys.
114
00:09:52,187 --> 00:10:00,771
And what they are going through is they're not just confirming whether or not the
documents are aren't responsive, but they're identifying why they think that the prompt
115
00:10:00,771 --> 00:10:01,342
got it wrong.
116
00:10:01,342 --> 00:10:04,513
Cause remember the prompt controls everything here.
117
00:10:04,513 --> 00:10:08,896
So somebody might come back and say, Hey, this document is relevant to the gen AI got it
wrong.
118
00:10:08,896 --> 00:10:19,582
That's because in your prompt, you forgot to mention that documents about blue Buffalo dog
food is not the same as conversations about the wild Buffalo in the Plains, right?
119
00:10:19,582 --> 00:10:21,011
whatever the case may be.
120
00:10:21,011 --> 00:10:22,872
You have to go back to the drawing board.
121
00:10:22,872 --> 00:10:25,535
You have to retool your prompt.
122
00:10:25,535 --> 00:10:28,638
And unfortunately, you got to run it against the whole thousand again.
123
00:10:28,638 --> 00:10:35,463
You do this a couple of times until you get to a point where you feel comfortable that
your margin of error is small enough, and then you run it against the whole.
124
00:10:35,463 --> 00:10:38,475
That's how we kick off a GenAI document review.
125
00:10:38,475 --> 00:10:40,366
Now documents are going to be scored.
126
00:10:40,366 --> 00:10:47,771
Most of these GenAI tools, unlike TAR, which uses a scale of 100 down to zero, only have
basically five scores.
127
00:10:47,771 --> 00:10:48,878
And those are usually
128
00:10:48,878 --> 00:10:55,804
very likely relevant, relevant, maybe relevant, not relevant, and often error or junk or
some other flag.
129
00:10:55,804 --> 00:10:57,576
Like I couldn't look at the document.
130
00:10:57,576 --> 00:11:03,061
What you really want to do is make sure you have almost no documents that are in the maybe
relevant.
131
00:11:03,061 --> 00:11:08,646
These are the most problematic and costly because documents that your AI says are not
relevant.
132
00:11:08,646 --> 00:11:15,578
At this point, you should be able to set aside and just do some validation sampling
documents that the AI says are highly relevant.
133
00:11:15,578 --> 00:11:17,750
should be going to your senior attorneys.
134
00:11:17,750 --> 00:11:19,011
These are your hot documents.
135
00:11:19,011 --> 00:11:20,572
These are your key documents.
136
00:11:20,572 --> 00:11:29,780
These are not only likely fast track to your production or privilege queue, but also are
going to your case team for things like chronologies.
137
00:11:29,780 --> 00:11:33,765
And you have your human review team looking at the documents that the AI said are
relevant.
138
00:11:33,765 --> 00:11:39,350
And again, ideally you have very few documents that are the maybe relevant because that's
where you get your cost bloat.
139
00:11:39,350 --> 00:11:41,231
And I think that if you
140
00:11:41,470 --> 00:11:47,965
stratify your review in that way, and then you employ traditional validation techniques,
you can get to a defensible document review.
141
00:11:47,965 --> 00:11:57,210
But the important part is, especially in those early prompting iterations, you don't have
any, or you have almost zero documents in that maybe responsive, that unsure world.
142
00:11:57,210 --> 00:12:06,214
You really got to get your prompts to a point where the AI can with certainty look at what
you wrote and say, yes, this document is relevant or no, this document is not.
143
00:12:06,214 --> 00:12:09,602
And if you have too much of the, don't know early on,
144
00:12:09,602 --> 00:12:14,568
you're going to be saddled with a whole lot of review when you turn that on a million
document population.
145
00:12:14,987 --> 00:12:22,942
And so, again, I've done some of this and it's very difficult to figure out how to revise
the prompt.
146
00:12:22,942 --> 00:12:30,337
what happened, like, Marcy and obviously in a simple case, it's probably relatively
straightforward to say, I can describe the case, whatever.
147
00:12:30,337 --> 00:12:33,828
But obviously a lot of cases are incredibly complex, right?
148
00:12:33,828 --> 00:12:38,701
Really complex cases, lots of different parties, lots of like, they're messy.
149
00:12:38,701 --> 00:12:42,283
Hard, really hard to know.
150
00:12:42,471 --> 00:12:51,871
What are all the issues and what is relevant when you're starting to say, okay, tell me
everything you know about the case on day one.
151
00:12:51,871 --> 00:12:57,911
When we all know by day, you know, 30 or whatever, our knowledge is going to change.
152
00:12:57,911 --> 00:13:00,352
We're going to have, we're going to find a new fact.
153
00:13:00,412 --> 00:13:09,372
How, how do you deal with that with if you're starting with a prompt and it's all scored
and now you just did you know, custodian interviews or whatever and you found out, oh,
154
00:13:09,372 --> 00:13:11,108
here's some new issues.
155
00:13:11,108 --> 00:13:14,782
How does that work with GenAI prompts and the like?
156
00:13:15,151 --> 00:13:27,103
Yeah, this is definitely an emerging area of workflow development because with gen AI
prompting, you kind of have to consider if I have to retool my gen AI prompt, do I have an
157
00:13:27,103 --> 00:13:29,005
ethical obligation to rerun it on the whole?
158
00:13:29,005 --> 00:13:33,921
Or do I have an ethical obligation to just rerun it on the stuff that the AI already
missed?
159
00:13:33,921 --> 00:13:36,143
Also, how do you get ahead of that?
160
00:13:36,143 --> 00:13:36,703
Right.
161
00:13:36,703 --> 00:13:38,510
Rather than thinking.
162
00:13:38,510 --> 00:13:45,032
Well, how do I adapt if I've already done all this prompt engineering and then a week
later I find a new facts?
163
00:13:45,032 --> 00:13:52,654
Maybe a different question is how do I use GenAI to find those facts before I go to that
crazy process of writing these prompts?
164
00:13:52,654 --> 00:13:57,996
So the other things that GenAI tools create for you are what we call early case insights.
165
00:13:57,996 --> 00:14:04,384
You can leverage more basic prompts like, hey, so here's an actual thing that I did.
166
00:14:04,384 --> 00:14:10,006
So tools like Relativity Air let you upload a complaint and then it writes your prompt
framework based off that complaint.
167
00:14:10,006 --> 00:14:15,409
But we already know that complaints themselves are biased because they are in favor of the
person writing it.
168
00:14:15,409 --> 00:14:20,301
So you can also feed it things like the complaint and the answer and try to get it to
write a prompt.
169
00:14:20,301 --> 00:14:29,556
So what I did recently is I took complaint, answer, in-house interview notes, and document
requests.
170
00:14:29,556 --> 00:14:32,293
put them into a generative AI tool called Harvey.
171
00:14:32,293 --> 00:14:40,233
And I asked Harvey to write for me a 2000 word prompt that summarized just a factual
statement to the issues in the case.
172
00:14:40,233 --> 00:14:45,713
I then took our entire document population pre-review.
173
00:14:45,833 --> 00:14:54,953
I exported it by printing parent and attachments as a single document so that GenAI can
see the whole document as one document.
174
00:14:54,953 --> 00:14:58,833
I put all of that into Harvey as its own mini database.
175
00:14:58,833 --> 00:15:00,613
And then I took that
176
00:15:00,613 --> 00:15:06,773
2000 word summary and after I vetted it and made sure it was actually accurate and I put
it into Harvey and I said, roll.
177
00:15:06,773 --> 00:15:11,493
You are a senior attorney for this client task.
178
00:15:11,493 --> 00:15:17,973
You've been asked to review these documents to look for information that supports or
refutes these allegations.
179
00:15:18,073 --> 00:15:19,093
Please do it.
180
00:15:19,093 --> 00:15:21,973
And then write me a memo about it and cite for me some documents.
181
00:15:21,973 --> 00:15:22,733
It did it.
182
00:15:22,733 --> 00:15:26,353
It found me two, 300 out of 10,000 documents.
183
00:15:26,353 --> 00:15:27,482
I reviewed those.
184
00:15:27,482 --> 00:15:32,046
The knowledge from those documents helped inform the conversations that we had with the
client.
185
00:15:32,046 --> 00:15:33,767
discovered some new custodians.
186
00:15:33,767 --> 00:15:36,709
We discovered some new facts that we had no awareness of.
187
00:15:36,709 --> 00:15:41,933
We also found some great documents that we were able to take directly to opposing counsel
and wave under their noses.
188
00:15:41,933 --> 00:15:51,581
But having that early knowledge helps us get ahead of surprises later because you don't
want to have those kinds of surprises in a gen AI review.
189
00:15:51,581 --> 00:15:57,359
Traditional tar reviews are more tolerant of changing course halfway through tar reviews
that we've been doing for 10 years.
190
00:15:57,359 --> 00:15:58,599
Those algorithms adapt.
191
00:15:58,599 --> 00:16:04,192
could be two, three days, even two, three weeks into your review and you can steer the
ship in a new direction.
192
00:16:04,192 --> 00:16:05,662
You can adjust your scoring.
193
00:16:05,662 --> 00:16:08,276
You can do sampling to fix earlier errors.
194
00:16:08,276 --> 00:16:13,438
But usually with these TAR scores or these TAR models, they're pushing what it thinks is
irrelevant down.
195
00:16:13,438 --> 00:16:18,740
So if you discover a new fact that is relevant, you haven't missed your opportunity to
bring those documents up.
196
00:16:18,740 --> 00:16:25,941
But with GenAI, you've already invested 40, 50, 60 hours of senior attorney time to
197
00:16:25,941 --> 00:16:36,623
write these prompts, be smart about it, use GenAI to first find these weird facts, and
then inform yourself before you write your prompt structures for the primary document.
198
00:16:36,624 --> 00:16:37,054
That's fair.
199
00:16:37,054 --> 00:16:37,785
right.
200
00:16:37,785 --> 00:16:39,606
So, Therese, what do we think?
201
00:16:39,606 --> 00:16:43,449
mean, obviously, that's wildly complex in a lot of ways.
202
00:16:43,449 --> 00:16:48,453
Helpful, but it's going to take some time for people to learn this and figure out how to
do it.
203
00:16:48,453 --> 00:16:57,038
What do we think sort of courts, regulators, plaintiffs and the like are going to be
thinking about when you just heard what Marcin did?
204
00:16:57,038 --> 00:16:58,719
What are the challenges there?
205
00:16:58,970 --> 00:17:05,171
I think that a lot of the challenges that we're going to see are going to be similar to
what we saw with TAR.
206
00:17:05,271 --> 00:17:13,291
And yet also on the flip side, I think some of this might actually be easier given the
climate that we are in today.
207
00:17:13,291 --> 00:17:21,851
I think when TAR started to become a thing that we were using in eDiscovery, there was a
lot of the, what is this technology?
208
00:17:21,851 --> 00:17:23,248
Who's using it?
209
00:17:23,248 --> 00:17:24,569
Like what's going on?
210
00:17:24,569 --> 00:17:26,940
What is this machine learning thing?
211
00:17:26,940 --> 00:17:33,317
And I think that generated a lot of resistance, fear from people saying, how do I know
that it works?
212
00:17:33,317 --> 00:17:35,218
Are we sure that we should be using this?
213
00:17:35,218 --> 00:17:36,460
Maybe we don't need it.
214
00:17:36,460 --> 00:17:42,746
Interestingly, I think that in the era that we're in today, GenAI is so prolific.
215
00:17:42,746 --> 00:17:44,447
Everybody's using it.
216
00:17:44,447 --> 00:17:47,027
Everybody's using it for everything.
217
00:17:47,027 --> 00:17:47,317
Right.
218
00:17:47,317 --> 00:17:52,932
People are using it online for, you know, to do searches, to ask therapy questions.
219
00:17:52,932 --> 00:18:00,548
mean, people are using it so prolifically that there is also a general acceptance of it,
that this is a thing we should use and everyone's going to be use it.
220
00:18:00,548 --> 00:18:03,171
And it has a great value in many different ways.
221
00:18:03,171 --> 00:18:13,719
And so I think on the one hand, and we're seeing this, honestly, we are seeing litigation
teams who historically might've been worried about using TAR saying, how can I use GenAI
222
00:18:13,719 --> 00:18:14,820
in my review?
223
00:18:14,820 --> 00:18:16,081
How can we?
224
00:18:16,081 --> 00:18:19,304
leverage this to do some great things on my case.
225
00:18:19,304 --> 00:18:24,448
And so I think on the one hand, there's more of an expectation that it will be used.
226
00:18:24,448 --> 00:18:28,331
There's more an acceptance of the use of generative AI.
227
00:18:28,331 --> 00:18:37,708
I think you're going to see a corresponding more general acceptance from even from
opposing counsel in courts and regulators that are saying, of course, people are going to
228
00:18:37,708 --> 00:18:38,719
be using this.
229
00:18:38,719 --> 00:18:40,881
Maybe you'd be crazy not to use it.
230
00:18:40,881 --> 00:18:46,255
So I do think we're going to see a little bit less resistance, which may give us some
traction to use it.
231
00:18:46,473 --> 00:18:58,640
On the flip side, some of the challenges we had with TAR that caused litigators to be
reluctant to use it is the requests from opposing counsel and sometimes from regulators
232
00:18:58,640 --> 00:19:01,561
and the like to say, well, I want to see your seed set.
233
00:19:01,561 --> 00:19:08,206
I want to be involved in making determinate how you're making determinations about what
the tool is going to identify as relevant.
234
00:19:08,206 --> 00:19:10,987
want to know the details of your process.
235
00:19:10,987 --> 00:19:14,332
And then of course, there are arguments, had lots of arguments about
236
00:19:14,332 --> 00:19:24,586
What's privileged, what's not, to what degree should you have to disclose your use of TAR,
to what degree should the other side have any say or review of what you're doing?
237
00:19:24,586 --> 00:19:33,449
I think we've got a lot of cases that helped in the sense of the idea that, it's the
producing party's responsibility and it is on you and there are some privileged
238
00:19:33,449 --> 00:19:42,382
information in terms of the use of it and work product, things that may be protected by
the work product protection and the like, but there still was a lot of disclosure.
239
00:19:42,382 --> 00:19:48,424
particularly early on and negotiating and discovery on discovery about what you're doing.
240
00:19:48,484 --> 00:19:52,385
I think we're still going to see some of that surfacing.
241
00:19:52,385 --> 00:19:54,266
What are the prompts that you're using?
242
00:19:54,266 --> 00:19:58,266
Should the other side be able to have a say in your prompts?
243
00:19:58,266 --> 00:20:02,929
Should they be able to review your prompts to decide if they are relevant?
244
00:20:02,929 --> 00:20:06,650
I'd like to think that we are far enough down the road where
245
00:20:06,838 --> 00:20:14,443
Really, if you're being questioned, you're just talking about the high level process and
what the protections are to make sure that it's accurate and things like validation to
246
00:20:14,443 --> 00:20:18,396
prove that your review met the standards and things like that.
247
00:20:18,396 --> 00:20:27,972
I am certain that there will be some, you know, disputes over the level of involvement or
disclosure relating to the use of Gen.
248
00:20:27,972 --> 00:20:28,139
AI.
249
00:20:28,139 --> 00:20:34,814
Again, it'll be interesting to see if the recognition that everybody's using it, everybody
should use it.
250
00:20:35,262 --> 00:20:40,617
maybe lowers the likelihood that we have fights about what prompts people are using or how
they're doing it.
251
00:20:40,617 --> 00:20:47,072
I somehow think that that is at least going to be an issue for some period of time until
we get past it.
252
00:20:47,072 --> 00:20:48,593
But I do think there's worries about that.
253
00:20:48,593 --> 00:20:49,994
There will be concerns.
254
00:20:49,994 --> 00:20:57,870
And look, the reality is in most cases, we don't want to spend time on discovery on
discovery.
255
00:20:57,870 --> 00:21:01,382
We don't want to spend time, more time than we need to.
256
00:21:01,525 --> 00:21:03,336
fighting over prompts and things.
257
00:21:03,336 --> 00:21:06,618
We just want to get to the review and produce the documents and get to the merits.
258
00:21:06,618 --> 00:21:09,541
But we all know that discovery disputes are inevitable.
259
00:21:09,541 --> 00:21:18,687
And I think that as we've already seen, when we're looking at these things, a lot of the
times it's not the technology, it's the people who are using it.
260
00:21:18,687 --> 00:21:24,405
And we are going to see situations just like we've already seen publicly with people.
261
00:21:24,405 --> 00:21:32,039
using GenAI to do briefs and then not reviewing them and submitting them to the court and
getting in trouble because you submit fall, know, hallucinate briefs with hallucinations
262
00:21:32,039 --> 00:21:33,720
in them cases that don't exist.
263
00:21:33,720 --> 00:21:43,776
We are almost inevitably going to see a case that comes out where somebody used GenAI and
perhaps either they did not use it properly because they weren't properly changed.
264
00:21:43,776 --> 00:21:46,937
They didn't have the proper experts who know how to use the tools.
265
00:21:46,937 --> 00:21:48,788
They weren't trained properly and using it.
266
00:21:48,788 --> 00:21:51,009
They assume that I put in a prop that it all must be right.
267
00:21:51,009 --> 00:21:52,730
And I produced the data right.
268
00:21:52,730 --> 00:21:54,027
That will happen.
269
00:21:54,027 --> 00:22:01,873
it will end up in sanctions and it will create fear in people and worry about using these
tools and things like that.
270
00:22:01,873 --> 00:22:05,354
think, you we all know it's sort of a little trite.
271
00:22:05,375 --> 00:22:13,901
I think by now we all know we have an ethical obligation to if we're using technology to
understand it, to understand how it works, to educate ourselves so that we are using it
272
00:22:13,901 --> 00:22:14,502
properly.
273
00:22:14,502 --> 00:22:17,544
But, you know, people don't always do that.
274
00:22:17,544 --> 00:22:21,671
And I think that's going to be one of the big risks is people who don't properly
275
00:22:21,671 --> 00:22:25,403
educate themselves, do not get the proper experts involved in using things.
276
00:22:25,403 --> 00:22:30,135
As as Marcine said, these tools can be great, but it is complex.
277
00:22:30,135 --> 00:22:34,397
It requires a very specific skillset and it's not a skillset that we all already have.
278
00:22:34,397 --> 00:22:36,467
It is a skillset we all need to develop.
279
00:22:36,467 --> 00:22:38,488
It is not instinctive, right?
280
00:22:38,488 --> 00:22:40,809
It is not, you know, that easy to figure out.
281
00:22:40,809 --> 00:22:41,720
It is a skillset.
282
00:22:41,720 --> 00:22:42,610
You need to learn it.
283
00:22:42,610 --> 00:22:48,883
You need to make sure you're doing that correctly and that you take the time to learn how
to do it correctly.
284
00:22:49,028 --> 00:22:50,299
And that's going to be our challenges.
285
00:22:50,299 --> 00:22:53,491
think there's a big learning curve, much bigger learning curve here than for TAR.
286
00:22:53,491 --> 00:22:56,312
There will inevitably be discovery disputes.
287
00:22:56,693 --> 00:23:02,588
Someone will not educate themselves and do it poorly, and then it's going to raise
questions about the use of it generally.
288
00:23:02,588 --> 00:23:06,067
I don't know that that's different than a lot of other situations.
289
00:23:06,067 --> 00:23:13,652
We've seen that happen with people using just electronic review tools and making mistakes
and it becoming an issue, right?
290
00:23:13,652 --> 00:23:15,863
And having disciplinary hearings and the like.
291
00:23:15,863 --> 00:23:18,438
But I think it's something we all need to be aware of.
292
00:23:18,438 --> 00:23:26,972
so that we are educating ourselves properly to be able to represent our clients properly
and to leverage the technology properly.
293
00:23:26,972 --> 00:23:32,415
Cause it's also our ethical obligation to use technology where it will benefit our
clients.
294
00:23:32,415 --> 00:23:37,517
And I think we need to all make sure that we're just doing that and preparing ourselves
for what's to come.
295
00:23:37,768 --> 00:23:47,382
Thanks, uh Therese and Marcin I think we're gonna see a lot of activity in the coming
year, both in terms of court cases, best practice development, all that.
296
00:23:47,382 --> 00:23:53,184
I think we are on a schedule that it's probably next year we'll have a lot more clarity on
how this is gonna work.
297
00:23:53,184 --> 00:23:55,204
in the meantime, hope this was helpful.
298
00:23:55,204 --> 00:23:56,285
Thanks for listening in.
299
00:23:56,285 --> 00:23:57,986
Thanks for listening to Tech Law Talks.
300
00:23:57,986 --> 00:24:00,007
be on the lookout for more podcasts.
301
00:24:00,007 --> 00:24:01,087
Talk to you later.
00:00:09,786 --> 00:00:14,330
Hello, this is Anthony and Diana from Reed Smith and welcome to Tech Law Talks.
2
00:00:14,330 --> 00:00:18,853
Today we are continuing a podcast series on AI-enabled e-discovery.
3
00:00:18,853 --> 00:00:28,842
This podcast series will focus on practical and legal issues when considering using
AI-enabled e-discovery with a focus on actual use cases, not just theoretical.
4
00:00:28,842 --> 00:00:33,065
Today, joining me are Therese Caparo and Marcian Krieger.
5
00:00:33,065 --> 00:00:37,977
And today we're gonna be discussing the use of AI-enabled discovery
6
00:00:37,977 --> 00:00:40,417
for doc and review and QC.
7
00:00:40,417 --> 00:00:42,657
So just the general topic of doc and review.
8
00:00:42,657 --> 00:00:45,218
So let's just start with Therese.
9
00:00:45,218 --> 00:00:47,198
What is using AI?
10
00:00:47,198 --> 00:00:51,520
What does it look like to use AI for doc and review?
11
00:00:52,183 --> 00:01:02,689
Well, I think one of the challenges with answering this question is there's so many things
that we do in document review that we could probably have several podcasts talking about
12
00:01:02,689 --> 00:01:08,272
all the different ways that you can use GenAI within your document review process
generally.
13
00:01:08,272 --> 00:01:21,305
But what I'm going to focus on today is really just using GenAI for first level, second
level QC reviews for data that you want to produce and what we're seeing
14
00:01:21,305 --> 00:01:25,358
as the opportunities for using gen AI in those processes at a high level.
15
00:01:25,358 --> 00:01:30,521
So I think that, you know, at a high level, there's probably three big bucket categories.
16
00:01:30,521 --> 00:01:34,565
One is just identifying categories of relevant documents.
17
00:01:34,565 --> 00:01:44,070
We know, we think, at the outset of a case, we have a complaint and we have, you know,
requests for production and we know generally what may be relevant.
18
00:01:44,070 --> 00:01:48,463
And we're looking to put together, we're putting together our review protocols and what we
think is going to be relevant.
19
00:01:48,463 --> 00:01:56,547
And a lot of times you find out later on that there are additional categories or things
that you didn't think of and things like that, which is the natural progression in any
20
00:01:56,547 --> 00:01:58,478
legal matter and any document review.
21
00:01:58,478 --> 00:02:09,244
But I think one of the things that GenAI can do is to help us surface those categories of
relevant documents sooner so that the outset of our document reviews, we're in a better
22
00:02:09,244 --> 00:02:11,425
position to say what may be relevant.
23
00:02:11,425 --> 00:02:20,819
what kind of categories out there, things we may not have thought of to help to structure
our review for the next big category of where GenAI can be used, which is in first level
24
00:02:20,819 --> 00:02:22,059
review, right?
25
00:02:22,059 --> 00:02:29,513
Using GenAI to review, so to speak, tag pre-identified documents that are likely to be
relevant, right?
26
00:02:29,513 --> 00:02:34,585
That can be, you know, for your productions and the like, and using that within those
processes.
27
00:02:34,585 --> 00:02:40,465
And then I think, you know, the other third big category is around QC.
28
00:02:40,465 --> 00:02:42,246
of your document review, right?
29
00:02:42,246 --> 00:02:47,690
Whether that is at second level to evaluate our documents being coded consistently, right?
30
00:02:47,690 --> 00:02:48,691
Are there gaps in there?
31
00:02:48,691 --> 00:02:50,512
Are there disagreements between the reviewers?
32
00:02:50,512 --> 00:02:55,846
Are there gaps in the production that we're sending out where we are missing categories of
things, right?
33
00:02:55,846 --> 00:03:06,684
Are things being mis-categorized and things like that to make sure that the productions
after our first level, whether that is human or computer, is complete and accurate and
34
00:03:06,684 --> 00:03:08,947
there's nothing that's being missed?
35
00:03:08,947 --> 00:03:21,045
So I think, you know, really at a super high level, those are three categories that we're
seeing to be able to use GenAI again, specifically for what do I need to produce and
36
00:03:21,045 --> 00:03:24,820
making sure that those productions are complete and accurate.
37
00:03:24,821 --> 00:03:29,125
And Therese, when you're talking about first level review, right?
38
00:03:29,125 --> 00:03:36,180
I think my understanding is that like TAR and others, that they're going to score each
document, like highly relevant, less relevant, whatever.
39
00:03:36,180 --> 00:03:37,311
It's going to be a score.
40
00:03:37,311 --> 00:03:46,349
And I guess one of the issues that people are going to have to think through is how much
you, is it, when we say, when we're saying we're going to use it in first level review,
41
00:03:46,349 --> 00:03:49,121
does that mean we're not going to look at the documents?
42
00:03:49,121 --> 00:03:51,423
And if it's highly relevant, we're just going to produce it?
43
00:03:51,423 --> 00:03:54,025
Or do we think that it's going to be
44
00:03:54,276 --> 00:03:59,148
People are going to still review it, but they're going to see what the scoring is.
45
00:03:59,148 --> 00:04:03,508
Like, what is your view in terms of how this is going to probably play out?
46
00:04:03,881 --> 00:04:05,932
Look, I think it depends on who you talk to.
47
00:04:05,932 --> 00:04:08,055
And I think it's also a matter of timing.
48
00:04:08,055 --> 00:04:14,640
We're at a pretty early stage and using GenAI tools for document review.
49
00:04:14,640 --> 00:04:17,823
There's definitely great use cases out there and people are using it.
50
00:04:17,823 --> 00:04:19,004
So we're at early stages.
51
00:04:19,004 --> 00:04:27,070
I think there are people who will tell you it is so good or will get so good that we will
really be able to forego a first level review.
52
00:04:27,070 --> 00:04:29,111
So there are definitely people who believe that.
53
00:04:29,111 --> 00:04:31,646
I think most lawyers are going to be for now.
54
00:04:31,646 --> 00:04:33,967
a lot more cautious than that.
55
00:04:33,967 --> 00:04:43,212
There will probably still be some level of review of the documents to make sure that
things going out are not just relevant, but not privileged.
56
00:04:43,212 --> 00:04:48,505
We've talked about privilege before in these podcasts to make sure that it's accurate.
57
00:04:48,505 --> 00:04:58,765
You're going to still see some level of review I expect for a while to human review, to
make sure that what is happening, that anything that goes out the door.
58
00:04:58,765 --> 00:05:02,987
is accurately categorized and is not privileged and we're not missing anything.
59
00:05:02,987 --> 00:05:16,122
I think the ideal goal would be to get to a point where we eliminate a lot of human
physically looking at each document and more human time is spent on the prompts and making
60
00:05:16,122 --> 00:05:20,274
sure the process is accurate and validation and things like that.
61
00:05:20,274 --> 00:05:27,519
And using those tools to make sure that the productions are accurate and correct and are
relevant and not, you know
62
00:05:27,519 --> 00:05:29,611
including privileged documents and the like.
63
00:05:29,611 --> 00:05:39,699
But I think as a starting point, we are still going to be seeing some level of human
review as people, you know, we don't quite trust the technology that much, but I do think
64
00:05:39,699 --> 00:05:50,408
the long-term goal, you know, should the technology get us there would be to, you know,
eliminate, largely eliminate manual document by document review.
65
00:05:50,408 --> 00:05:56,052
And instead you're basically having more of a level of technology review to make sure that
the substance is correct.
66
00:05:56,052 --> 00:05:56,466
But,
67
00:05:56,466 --> 00:05:57,784
We're not there yet.
68
00:05:57,785 --> 00:06:04,225
And so, Marcin why don't you explain a little bit of like, how would you actually do this?
69
00:06:04,225 --> 00:06:14,905
Like, how would you actually do the three stages that Therese said if you're using some
type of AI tool, particularly GenAI tool?
70
00:06:14,967 --> 00:06:23,845
All right, so generative AI tools in document review work drastically differently than the
types of technology that we're used to using, which is TAR.
71
00:06:23,845 --> 00:06:39,166
In generative AI, you have to imagine that you have a computer or a virtual room that has
a person that looks at only one document, and you have to write to that one person a set
72
00:06:39,166 --> 00:06:40,357
of instructions
73
00:06:40,357 --> 00:06:48,037
that they can read your protocol and look at that one document and hopefully be 100 %
accurate every time.
74
00:06:48,277 --> 00:06:51,337
What that means is there's no cross document intelligence.
75
00:06:51,337 --> 00:06:59,057
If you have a million documents in your document review population, imagine if you had to
write a document that you will hand to a million people and all they can do is read that
76
00:06:59,057 --> 00:07:05,857
document once and then they look at one document and tell you, it responsive or not to
that memo?
77
00:07:05,857 --> 00:07:09,097
Same thing for privilege, same thing for every issue code.
78
00:07:09,253 --> 00:07:11,694
So all of the work has to happen upfront.
79
00:07:11,694 --> 00:07:21,358
Now, that doesn't mean that you spend a month writing one crazy instruction and then you
just run it and you walk away because that would be incredibly expensive and wasteful.
80
00:07:21,358 --> 00:07:30,222
The way that we set up a generative AI review is that you start by selecting a small group
of documents that you're going to test against.
81
00:07:30,222 --> 00:07:37,465
Now, my best advice on this is you want to have about 30 documents that you absolutely
know are relevant.
82
00:07:37,572 --> 00:07:42,005
You want to have about 30 documents that you absolutely know are not relevant.
83
00:07:42,005 --> 00:07:46,869
And then just to make it rounded out an easy 40 randomly selected documents from your
population.
84
00:07:46,869 --> 00:07:50,633
What you're going to do is you're going to then draft your prompt instruction.
85
00:07:50,633 --> 00:07:56,197
Now, attorneys write and think very differently than generative AI does.
86
00:07:56,197 --> 00:08:01,421
Generative AI is very literal and you also have a limit to the amount of text that you can
put in.
87
00:08:01,421 --> 00:08:03,488
So when we used to write
88
00:08:03,488 --> 00:08:09,300
document review protocols that were 20 pages long and continued, contained every single
nuance in detail.
89
00:08:09,300 --> 00:08:16,882
You have to really distill that drafting process down to clear, near binary instructions.
90
00:08:16,882 --> 00:08:19,542
Here is a description of the case.
91
00:08:19,542 --> 00:08:22,323
Here is a description of what has been requested.
92
00:08:22,323 --> 00:08:24,084
This is what is relevant.
93
00:08:24,084 --> 00:08:29,225
And then it's always best to have a little bit of instruction on this is what is not
relevant.
94
00:08:29,225 --> 00:08:33,110
And it's a lot easier to do it when you're just doing the relevant, not relevant.
95
00:08:33,110 --> 00:08:37,654
There are tools out there that let you do, for example, up to 15 different issue codes.
96
00:08:37,654 --> 00:08:41,186
Each issue code needs to have that same thought process applied.
97
00:08:41,186 --> 00:08:43,760
And I actually have a tip about how to use those issue codes for later.
98
00:08:43,760 --> 00:08:46,582
But you start with your 100 and you test it.
99
00:08:46,582 --> 00:08:50,265
You run the GenAI against this 100 document population.
100
00:08:50,265 --> 00:08:57,801
And what you are looking for is not only that the AI agrees with what you expect to have
happen, but these GenAI tools give you rationales.
101
00:08:57,801 --> 00:09:01,474
They tell you why they think a document's relevant, why they think a document is not
relevant.
102
00:09:01,474 --> 00:09:11,173
You need to be looking at the rationales that AI is generating to make sure it's not
giving you a correct score, but it's rationales completely out of left field because that
103
00:09:11,173 --> 00:09:12,404
means you just got lucky.
104
00:09:12,404 --> 00:09:18,009
You want to make sure that not only is the document relevant, but it's the rationale is
grounded in the prompt that you generated.
105
00:09:18,009 --> 00:09:24,027
If you have to make small adjustments, you do until you get to a point where you say, Hey,
I feel pretty good here.
106
00:09:24,027 --> 00:09:27,487
my 30 documents that were relevant, it says they're relevant and I agree.
107
00:09:27,487 --> 00:09:33,767
The 30 that I self-selected that are not relevant are in fact being said not relevant by
the gen AI and its rationale is accurate.
108
00:09:33,767 --> 00:09:37,947
And within that random sample of 40, I agree with everything it did.
109
00:09:37,947 --> 00:09:41,287
You go from 100 documents to a thousand documents.
110
00:09:41,547 --> 00:09:46,267
Now, do you take the time to validate all 1000 documents?
111
00:09:46,667 --> 00:09:47,327
Probably yes.
112
00:09:47,327 --> 00:09:48,907
Maybe at this point, it's not you and I, Anthony.
113
00:09:48,907 --> 00:09:52,107
At this point, maybe you're using one of your senior eDiscovery attorneys.
114
00:09:52,187 --> 00:10:00,771
And what they are going through is they're not just confirming whether or not the
documents are aren't responsive, but they're identifying why they think that the prompt
115
00:10:00,771 --> 00:10:01,342
got it wrong.
116
00:10:01,342 --> 00:10:04,513
Cause remember the prompt controls everything here.
117
00:10:04,513 --> 00:10:08,896
So somebody might come back and say, Hey, this document is relevant to the gen AI got it
wrong.
118
00:10:08,896 --> 00:10:19,582
That's because in your prompt, you forgot to mention that documents about blue Buffalo dog
food is not the same as conversations about the wild Buffalo in the Plains, right?
119
00:10:19,582 --> 00:10:21,011
whatever the case may be.
120
00:10:21,011 --> 00:10:22,872
You have to go back to the drawing board.
121
00:10:22,872 --> 00:10:25,535
You have to retool your prompt.
122
00:10:25,535 --> 00:10:28,638
And unfortunately, you got to run it against the whole thousand again.
123
00:10:28,638 --> 00:10:35,463
You do this a couple of times until you get to a point where you feel comfortable that
your margin of error is small enough, and then you run it against the whole.
124
00:10:35,463 --> 00:10:38,475
That's how we kick off a GenAI document review.
125
00:10:38,475 --> 00:10:40,366
Now documents are going to be scored.
126
00:10:40,366 --> 00:10:47,771
Most of these GenAI tools, unlike TAR, which uses a scale of 100 down to zero, only have
basically five scores.
127
00:10:47,771 --> 00:10:48,878
And those are usually
128
00:10:48,878 --> 00:10:55,804
very likely relevant, relevant, maybe relevant, not relevant, and often error or junk or
some other flag.
129
00:10:55,804 --> 00:10:57,576
Like I couldn't look at the document.
130
00:10:57,576 --> 00:11:03,061
What you really want to do is make sure you have almost no documents that are in the maybe
relevant.
131
00:11:03,061 --> 00:11:08,646
These are the most problematic and costly because documents that your AI says are not
relevant.
132
00:11:08,646 --> 00:11:15,578
At this point, you should be able to set aside and just do some validation sampling
documents that the AI says are highly relevant.
133
00:11:15,578 --> 00:11:17,750
should be going to your senior attorneys.
134
00:11:17,750 --> 00:11:19,011
These are your hot documents.
135
00:11:19,011 --> 00:11:20,572
These are your key documents.
136
00:11:20,572 --> 00:11:29,780
These are not only likely fast track to your production or privilege queue, but also are
going to your case team for things like chronologies.
137
00:11:29,780 --> 00:11:33,765
And you have your human review team looking at the documents that the AI said are
relevant.
138
00:11:33,765 --> 00:11:39,350
And again, ideally you have very few documents that are the maybe relevant because that's
where you get your cost bloat.
139
00:11:39,350 --> 00:11:41,231
And I think that if you
140
00:11:41,470 --> 00:11:47,965
stratify your review in that way, and then you employ traditional validation techniques,
you can get to a defensible document review.
141
00:11:47,965 --> 00:11:57,210
But the important part is, especially in those early prompting iterations, you don't have
any, or you have almost zero documents in that maybe responsive, that unsure world.
142
00:11:57,210 --> 00:12:06,214
You really got to get your prompts to a point where the AI can with certainty look at what
you wrote and say, yes, this document is relevant or no, this document is not.
143
00:12:06,214 --> 00:12:09,602
And if you have too much of the, don't know early on,
144
00:12:09,602 --> 00:12:14,568
you're going to be saddled with a whole lot of review when you turn that on a million
document population.
145
00:12:14,987 --> 00:12:22,942
And so, again, I've done some of this and it's very difficult to figure out how to revise
the prompt.
146
00:12:22,942 --> 00:12:30,337
what happened, like, Marcy and obviously in a simple case, it's probably relatively
straightforward to say, I can describe the case, whatever.
147
00:12:30,337 --> 00:12:33,828
But obviously a lot of cases are incredibly complex, right?
148
00:12:33,828 --> 00:12:38,701
Really complex cases, lots of different parties, lots of like, they're messy.
149
00:12:38,701 --> 00:12:42,283
Hard, really hard to know.
150
00:12:42,471 --> 00:12:51,871
What are all the issues and what is relevant when you're starting to say, okay, tell me
everything you know about the case on day one.
151
00:12:51,871 --> 00:12:57,911
When we all know by day, you know, 30 or whatever, our knowledge is going to change.
152
00:12:57,911 --> 00:13:00,352
We're going to have, we're going to find a new fact.
153
00:13:00,412 --> 00:13:09,372
How, how do you deal with that with if you're starting with a prompt and it's all scored
and now you just did you know, custodian interviews or whatever and you found out, oh,
154
00:13:09,372 --> 00:13:11,108
here's some new issues.
155
00:13:11,108 --> 00:13:14,782
How does that work with GenAI prompts and the like?
156
00:13:15,151 --> 00:13:27,103
Yeah, this is definitely an emerging area of workflow development because with gen AI
prompting, you kind of have to consider if I have to retool my gen AI prompt, do I have an
157
00:13:27,103 --> 00:13:29,005
ethical obligation to rerun it on the whole?
158
00:13:29,005 --> 00:13:33,921
Or do I have an ethical obligation to just rerun it on the stuff that the AI already
missed?
159
00:13:33,921 --> 00:13:36,143
Also, how do you get ahead of that?
160
00:13:36,143 --> 00:13:36,703
Right.
161
00:13:36,703 --> 00:13:38,510
Rather than thinking.
162
00:13:38,510 --> 00:13:45,032
Well, how do I adapt if I've already done all this prompt engineering and then a week
later I find a new facts?
163
00:13:45,032 --> 00:13:52,654
Maybe a different question is how do I use GenAI to find those facts before I go to that
crazy process of writing these prompts?
164
00:13:52,654 --> 00:13:57,996
So the other things that GenAI tools create for you are what we call early case insights.
165
00:13:57,996 --> 00:14:04,384
You can leverage more basic prompts like, hey, so here's an actual thing that I did.
166
00:14:04,384 --> 00:14:10,006
So tools like Relativity Air let you upload a complaint and then it writes your prompt
framework based off that complaint.
167
00:14:10,006 --> 00:14:15,409
But we already know that complaints themselves are biased because they are in favor of the
person writing it.
168
00:14:15,409 --> 00:14:20,301
So you can also feed it things like the complaint and the answer and try to get it to
write a prompt.
169
00:14:20,301 --> 00:14:29,556
So what I did recently is I took complaint, answer, in-house interview notes, and document
requests.
170
00:14:29,556 --> 00:14:32,293
put them into a generative AI tool called Harvey.
171
00:14:32,293 --> 00:14:40,233
And I asked Harvey to write for me a 2000 word prompt that summarized just a factual
statement to the issues in the case.
172
00:14:40,233 --> 00:14:45,713
I then took our entire document population pre-review.
173
00:14:45,833 --> 00:14:54,953
I exported it by printing parent and attachments as a single document so that GenAI can
see the whole document as one document.
174
00:14:54,953 --> 00:14:58,833
I put all of that into Harvey as its own mini database.
175
00:14:58,833 --> 00:15:00,613
And then I took that
176
00:15:00,613 --> 00:15:06,773
2000 word summary and after I vetted it and made sure it was actually accurate and I put
it into Harvey and I said, roll.
177
00:15:06,773 --> 00:15:11,493
You are a senior attorney for this client task.
178
00:15:11,493 --> 00:15:17,973
You've been asked to review these documents to look for information that supports or
refutes these allegations.
179
00:15:18,073 --> 00:15:19,093
Please do it.
180
00:15:19,093 --> 00:15:21,973
And then write me a memo about it and cite for me some documents.
181
00:15:21,973 --> 00:15:22,733
It did it.
182
00:15:22,733 --> 00:15:26,353
It found me two, 300 out of 10,000 documents.
183
00:15:26,353 --> 00:15:27,482
I reviewed those.
184
00:15:27,482 --> 00:15:32,046
The knowledge from those documents helped inform the conversations that we had with the
client.
185
00:15:32,046 --> 00:15:33,767
discovered some new custodians.
186
00:15:33,767 --> 00:15:36,709
We discovered some new facts that we had no awareness of.
187
00:15:36,709 --> 00:15:41,933
We also found some great documents that we were able to take directly to opposing counsel
and wave under their noses.
188
00:15:41,933 --> 00:15:51,581
But having that early knowledge helps us get ahead of surprises later because you don't
want to have those kinds of surprises in a gen AI review.
189
00:15:51,581 --> 00:15:57,359
Traditional tar reviews are more tolerant of changing course halfway through tar reviews
that we've been doing for 10 years.
190
00:15:57,359 --> 00:15:58,599
Those algorithms adapt.
191
00:15:58,599 --> 00:16:04,192
could be two, three days, even two, three weeks into your review and you can steer the
ship in a new direction.
192
00:16:04,192 --> 00:16:05,662
You can adjust your scoring.
193
00:16:05,662 --> 00:16:08,276
You can do sampling to fix earlier errors.
194
00:16:08,276 --> 00:16:13,438
But usually with these TAR scores or these TAR models, they're pushing what it thinks is
irrelevant down.
195
00:16:13,438 --> 00:16:18,740
So if you discover a new fact that is relevant, you haven't missed your opportunity to
bring those documents up.
196
00:16:18,740 --> 00:16:25,941
But with GenAI, you've already invested 40, 50, 60 hours of senior attorney time to
197
00:16:25,941 --> 00:16:36,623
write these prompts, be smart about it, use GenAI to first find these weird facts, and
then inform yourself before you write your prompt structures for the primary document.
198
00:16:36,624 --> 00:16:37,054
That's fair.
199
00:16:37,054 --> 00:16:37,785
right.
200
00:16:37,785 --> 00:16:39,606
So, Therese, what do we think?
201
00:16:39,606 --> 00:16:43,449
mean, obviously, that's wildly complex in a lot of ways.
202
00:16:43,449 --> 00:16:48,453
Helpful, but it's going to take some time for people to learn this and figure out how to
do it.
203
00:16:48,453 --> 00:16:57,038
What do we think sort of courts, regulators, plaintiffs and the like are going to be
thinking about when you just heard what Marcin did?
204
00:16:57,038 --> 00:16:58,719
What are the challenges there?
205
00:16:58,970 --> 00:17:05,171
I think that a lot of the challenges that we're going to see are going to be similar to
what we saw with TAR.
206
00:17:05,271 --> 00:17:13,291
And yet also on the flip side, I think some of this might actually be easier given the
climate that we are in today.
207
00:17:13,291 --> 00:17:21,851
I think when TAR started to become a thing that we were using in eDiscovery, there was a
lot of the, what is this technology?
208
00:17:21,851 --> 00:17:23,248
Who's using it?
209
00:17:23,248 --> 00:17:24,569
Like what's going on?
210
00:17:24,569 --> 00:17:26,940
What is this machine learning thing?
211
00:17:26,940 --> 00:17:33,317
And I think that generated a lot of resistance, fear from people saying, how do I know
that it works?
212
00:17:33,317 --> 00:17:35,218
Are we sure that we should be using this?
213
00:17:35,218 --> 00:17:36,460
Maybe we don't need it.
214
00:17:36,460 --> 00:17:42,746
Interestingly, I think that in the era that we're in today, GenAI is so prolific.
215
00:17:42,746 --> 00:17:44,447
Everybody's using it.
216
00:17:44,447 --> 00:17:47,027
Everybody's using it for everything.
217
00:17:47,027 --> 00:17:47,317
Right.
218
00:17:47,317 --> 00:17:52,932
People are using it online for, you know, to do searches, to ask therapy questions.
219
00:17:52,932 --> 00:18:00,548
mean, people are using it so prolifically that there is also a general acceptance of it,
that this is a thing we should use and everyone's going to be use it.
220
00:18:00,548 --> 00:18:03,171
And it has a great value in many different ways.
221
00:18:03,171 --> 00:18:13,719
And so I think on the one hand, and we're seeing this, honestly, we are seeing litigation
teams who historically might've been worried about using TAR saying, how can I use GenAI
222
00:18:13,719 --> 00:18:14,820
in my review?
223
00:18:14,820 --> 00:18:16,081
How can we?
224
00:18:16,081 --> 00:18:19,304
leverage this to do some great things on my case.
225
00:18:19,304 --> 00:18:24,448
And so I think on the one hand, there's more of an expectation that it will be used.
226
00:18:24,448 --> 00:18:28,331
There's more an acceptance of the use of generative AI.
227
00:18:28,331 --> 00:18:37,708
I think you're going to see a corresponding more general acceptance from even from
opposing counsel in courts and regulators that are saying, of course, people are going to
228
00:18:37,708 --> 00:18:38,719
be using this.
229
00:18:38,719 --> 00:18:40,881
Maybe you'd be crazy not to use it.
230
00:18:40,881 --> 00:18:46,255
So I do think we're going to see a little bit less resistance, which may give us some
traction to use it.
231
00:18:46,473 --> 00:18:58,640
On the flip side, some of the challenges we had with TAR that caused litigators to be
reluctant to use it is the requests from opposing counsel and sometimes from regulators
232
00:18:58,640 --> 00:19:01,561
and the like to say, well, I want to see your seed set.
233
00:19:01,561 --> 00:19:08,206
I want to be involved in making determinate how you're making determinations about what
the tool is going to identify as relevant.
234
00:19:08,206 --> 00:19:10,987
want to know the details of your process.
235
00:19:10,987 --> 00:19:14,332
And then of course, there are arguments, had lots of arguments about
236
00:19:14,332 --> 00:19:24,586
What's privileged, what's not, to what degree should you have to disclose your use of TAR,
to what degree should the other side have any say or review of what you're doing?
237
00:19:24,586 --> 00:19:33,449
I think we've got a lot of cases that helped in the sense of the idea that, it's the
producing party's responsibility and it is on you and there are some privileged
238
00:19:33,449 --> 00:19:42,382
information in terms of the use of it and work product, things that may be protected by
the work product protection and the like, but there still was a lot of disclosure.
239
00:19:42,382 --> 00:19:48,424
particularly early on and negotiating and discovery on discovery about what you're doing.
240
00:19:48,484 --> 00:19:52,385
I think we're still going to see some of that surfacing.
241
00:19:52,385 --> 00:19:54,266
What are the prompts that you're using?
242
00:19:54,266 --> 00:19:58,266
Should the other side be able to have a say in your prompts?
243
00:19:58,266 --> 00:20:02,929
Should they be able to review your prompts to decide if they are relevant?
244
00:20:02,929 --> 00:20:06,650
I'd like to think that we are far enough down the road where
245
00:20:06,838 --> 00:20:14,443
Really, if you're being questioned, you're just talking about the high level process and
what the protections are to make sure that it's accurate and things like validation to
246
00:20:14,443 --> 00:20:18,396
prove that your review met the standards and things like that.
247
00:20:18,396 --> 00:20:27,972
I am certain that there will be some, you know, disputes over the level of involvement or
disclosure relating to the use of Gen.
248
00:20:27,972 --> 00:20:28,139
AI.
249
00:20:28,139 --> 00:20:34,814
Again, it'll be interesting to see if the recognition that everybody's using it, everybody
should use it.
250
00:20:35,262 --> 00:20:40,617
maybe lowers the likelihood that we have fights about what prompts people are using or how
they're doing it.
251
00:20:40,617 --> 00:20:47,072
I somehow think that that is at least going to be an issue for some period of time until
we get past it.
252
00:20:47,072 --> 00:20:48,593
But I do think there's worries about that.
253
00:20:48,593 --> 00:20:49,994
There will be concerns.
254
00:20:49,994 --> 00:20:57,870
And look, the reality is in most cases, we don't want to spend time on discovery on
discovery.
255
00:20:57,870 --> 00:21:01,382
We don't want to spend time, more time than we need to.
256
00:21:01,525 --> 00:21:03,336
fighting over prompts and things.
257
00:21:03,336 --> 00:21:06,618
We just want to get to the review and produce the documents and get to the merits.
258
00:21:06,618 --> 00:21:09,541
But we all know that discovery disputes are inevitable.
259
00:21:09,541 --> 00:21:18,687
And I think that as we've already seen, when we're looking at these things, a lot of the
times it's not the technology, it's the people who are using it.
260
00:21:18,687 --> 00:21:24,405
And we are going to see situations just like we've already seen publicly with people.
261
00:21:24,405 --> 00:21:32,039
using GenAI to do briefs and then not reviewing them and submitting them to the court and
getting in trouble because you submit fall, know, hallucinate briefs with hallucinations
262
00:21:32,039 --> 00:21:33,720
in them cases that don't exist.
263
00:21:33,720 --> 00:21:43,776
We are almost inevitably going to see a case that comes out where somebody used GenAI and
perhaps either they did not use it properly because they weren't properly changed.
264
00:21:43,776 --> 00:21:46,937
They didn't have the proper experts who know how to use the tools.
265
00:21:46,937 --> 00:21:48,788
They weren't trained properly and using it.
266
00:21:48,788 --> 00:21:51,009
They assume that I put in a prop that it all must be right.
267
00:21:51,009 --> 00:21:52,730
And I produced the data right.
268
00:21:52,730 --> 00:21:54,027
That will happen.
269
00:21:54,027 --> 00:22:01,873
it will end up in sanctions and it will create fear in people and worry about using these
tools and things like that.
270
00:22:01,873 --> 00:22:05,354
think, you we all know it's sort of a little trite.
271
00:22:05,375 --> 00:22:13,901
I think by now we all know we have an ethical obligation to if we're using technology to
understand it, to understand how it works, to educate ourselves so that we are using it
272
00:22:13,901 --> 00:22:14,502
properly.
273
00:22:14,502 --> 00:22:17,544
But, you know, people don't always do that.
274
00:22:17,544 --> 00:22:21,671
And I think that's going to be one of the big risks is people who don't properly
275
00:22:21,671 --> 00:22:25,403
educate themselves, do not get the proper experts involved in using things.
276
00:22:25,403 --> 00:22:30,135
As as Marcine said, these tools can be great, but it is complex.
277
00:22:30,135 --> 00:22:34,397
It requires a very specific skillset and it's not a skillset that we all already have.
278
00:22:34,397 --> 00:22:36,467
It is a skillset we all need to develop.
279
00:22:36,467 --> 00:22:38,488
It is not instinctive, right?
280
00:22:38,488 --> 00:22:40,809
It is not, you know, that easy to figure out.
281
00:22:40,809 --> 00:22:41,720
It is a skillset.
282
00:22:41,720 --> 00:22:42,610
You need to learn it.
283
00:22:42,610 --> 00:22:48,883
You need to make sure you're doing that correctly and that you take the time to learn how
to do it correctly.
284
00:22:49,028 --> 00:22:50,299
And that's going to be our challenges.
285
00:22:50,299 --> 00:22:53,491
think there's a big learning curve, much bigger learning curve here than for TAR.
286
00:22:53,491 --> 00:22:56,312
There will inevitably be discovery disputes.
287
00:22:56,693 --> 00:23:02,588
Someone will not educate themselves and do it poorly, and then it's going to raise
questions about the use of it generally.
288
00:23:02,588 --> 00:23:06,067
I don't know that that's different than a lot of other situations.
289
00:23:06,067 --> 00:23:13,652
We've seen that happen with people using just electronic review tools and making mistakes
and it becoming an issue, right?
290
00:23:13,652 --> 00:23:15,863
And having disciplinary hearings and the like.
291
00:23:15,863 --> 00:23:18,438
But I think it's something we all need to be aware of.
292
00:23:18,438 --> 00:23:26,972
so that we are educating ourselves properly to be able to represent our clients properly
and to leverage the technology properly.
293
00:23:26,972 --> 00:23:32,415
Cause it's also our ethical obligation to use technology where it will benefit our
clients.
294
00:23:32,415 --> 00:23:37,517
And I think we need to all make sure that we're just doing that and preparing ourselves
for what's to come.
295
00:23:37,768 --> 00:23:47,382
Thanks, uh Therese and Marcin I think we're gonna see a lot of activity in the coming
year, both in terms of court cases, best practice development, all that.
296
00:23:47,382 --> 00:23:53,184
I think we are on a schedule that it's probably next year we'll have a lot more clarity on
how this is gonna work.
297
00:23:53,184 --> 00:23:55,204
in the meantime, hope this was helpful.
298
00:23:55,204 --> 00:23:56,285
Thanks for listening in.
299
00:23:56,285 --> 00:23:57,986
Thanks for listening to Tech Law Talks.
300
00:23:57,986 --> 00:24:00,007
be on the lookout for more podcasts.
301
00:24:00,007 --> 00:24:01,087
Talk to you later.
