Automating privilege: How TAR and GenAI transform e-discovery review and logs

WEBVTT
00:00:13.173 --> 00:00:16.813
Intro: Hello, and welcome to Tech Law Talks, a podcast brought to you by Reed Smith's Emerging Technologies Group. In each episode
00:00:16.813 --> 00:00:21.273
of this podcast, we will discuss cutting-edge issues on technology, data, and the law.
00:00:21.533 --> 00:00:26.333
We will provide practical observations on a wide variety of technology and data
00:00:26.333 --> 00:00:31.313
topics to give you quick and actionable tips to address the issues you are dealing with every day.
00:00:33.033 --> 00:00:37.613
Anthony: Hello, this is Anthony Diana from Reed Smith, and welcome to Tech Law Talks.
00:00:37.753 --> 00:00:43.433
Today, we are continuing our podcast series on AI-enabled e-discovery.
00:00:43.713 --> 00:00:48.053
The podcast series will focus on practical and legal issues when considering
00:00:48.053 --> 00:00:52.853
using AI-enabled e-discovery with a focus on actual use cases.
00:00:53.153 --> 00:00:59.953
So joining me today are Therese Craparo and Samantha Walsh from the New York office of Reed Smith.
00:01:00.293 --> 00:01:03.793
Welcome, guys. So now let's jump in and get started.
00:01:04.073 --> 00:01:09.853
Today, we're going to be talking about privilege and the use cases around using
00:01:09.853 --> 00:01:14.053
AI-enabled e-discovery for privileged issues.
00:01:14.413 --> 00:01:20.913
So let's just start with the problem that everyone has, which is privilege reviews
00:01:20.913 --> 00:01:25.453
and privilege logging represent one of the highest costs of e-discovery.
00:01:25.693 --> 00:01:28.753
And it has been a challenge, I think, you know,
00:01:29.413 --> 00:01:34.793
over the past, you know, 20 years, as everything has gotten better in terms
00:01:34.793 --> 00:01:39.293
of operationalizing and squeezing it, squeezing out as much efficiency as possible,
00:01:39.473 --> 00:01:43.973
privilege continues to be a challenge that, frankly, hasn't really been solved.
00:01:44.253 --> 00:01:50.373
So the question is, can GenAI or other types of artificial intelligence,
00:01:50.873 --> 00:01:56.833
can it be effectively used to start bringing down these costs?
00:01:57.563 --> 00:02:02.723
Samantha: Yeah, I mean, I agree. Dealing with privilege is a huge challenge and a huge
00:02:02.723 --> 00:02:05.803
cost because it's so important to get it right.
00:02:06.063 --> 00:02:11.683
And getting it right is a pretty heavy lift. So we've definitely seen success
00:02:11.683 --> 00:02:16.743
using AI for handling privilege in e-discovery.
00:02:16.743 --> 00:02:23.723
I think in particular with privilege review and QC of privilege review,
00:02:23.983 --> 00:02:29.423
but also just in generally identification of potentially privileged material
00:02:29.423 --> 00:02:32.803
and in generation of privilege logs,
00:02:32.903 --> 00:02:39.423
which I think is one of the more resource intensive aspects of the privilege. privilege, endeavor.
00:02:41.035 --> 00:02:47.595
Anthony: And so, Therese, how do we see AI currently being used in the privilege review
00:02:47.595 --> 00:02:51.775
process? And how does that differ from sort of traditional manual review?
00:02:52.215 --> 00:02:57.895
Therese: The main area where we've seen a lot of success with the use of AI with privilege
00:02:57.895 --> 00:03:04.735
review has been with the technology-assisted review models in sort of directing
00:03:04.735 --> 00:03:07.915
and focusing privilege review.
00:03:08.335 --> 00:03:13.635
And really, you know, it's like any kind of TAR review, it's augmenting human
00:03:13.635 --> 00:03:17.835
review, right, and helping to make the human review more efficient and more effective.
00:03:18.055 --> 00:03:21.895
So just in terms of talking about it in real terms, we had a case,
00:03:22.055 --> 00:03:27.975
we had hundreds of thousands of privileged documents, tens of millions of documents
00:03:27.975 --> 00:03:30.215
to be reviewed. So really a massive case.
00:03:30.395 --> 00:03:35.155
And as Sam already highlighted, right, identifying privilege is critical, right?
00:03:35.215 --> 00:03:38.375
You have to be precise, you have to find the information. There was a lot of
00:03:38.375 --> 00:03:41.295
concern with the speed with which the review needed to be done because,
00:03:41.435 --> 00:03:45.295
of course, we're always under the gun and under deadlines to produce documents
00:03:45.295 --> 00:03:50.155
of getting it done, but also getting it done precisely and making sure that
00:03:50.155 --> 00:03:54.535
privileged documents weren't produced and other types of confidential information wasn't produced.
00:03:55.095 --> 00:03:58.875
And so effectively, what we did is use a TAR model, train the model.
00:03:59.015 --> 00:04:04.135
What we had already known were privileged documents so that we could score the
00:04:04.135 --> 00:04:07.135
privileged, likely-to-be-privileged documents in the process.
00:04:07.695 --> 00:04:12.695
In doing that, what we elected to do to streamline the review was anything that
00:04:12.695 --> 00:04:16.435
had a certain high score, so after reviewing it and QC-ing the model,
00:04:16.835 --> 00:04:22.575
highly likely-to-be-privileged got put into a privilege-slash-responsive review
00:04:22.575 --> 00:04:26.415
queue, like direct into the queue, so that we weren't going through several
00:04:26.415 --> 00:04:28.695
levels of review where that may not be necessary.
00:04:28.855 --> 00:04:31.015
Highly likely-to-be-privileged goes to the privileged review.
00:04:31.529 --> 00:04:35.669
But then even with the reviewers who are doing the review of the remaining documents,
00:04:35.869 --> 00:04:39.609
they could still see the score of the documents, if it was likely to be privileged,
00:04:39.929 --> 00:04:42.829
which gave them a little bit of insight, maybe pay a little bit more attention
00:04:42.829 --> 00:04:48.429
here because this may be a privileged document, in order just to help everyone
00:04:48.429 --> 00:04:51.249
focus on things likely to be privileged.
00:04:51.249 --> 00:04:55.529
And then on the flip side, for the QC purposes, use the model to say,
00:04:55.669 --> 00:05:02.729
is there anything in the production sets that have a higher score as privilege? So did we miss anything?
00:05:03.029 --> 00:05:06.829
Did something slip through the cracks? Can we take a look at that just to make
00:05:06.829 --> 00:05:11.529
sure that we were doing that extra check, not just relying on search terms and
00:05:11.529 --> 00:05:13.809
things like that to try to find the information?
00:05:13.809 --> 00:05:19.249
So it's not like it made the review magically, you know, done with privilege,
00:05:19.469 --> 00:05:22.009
but it made it much more efficient, much more effective.
00:05:22.229 --> 00:05:26.989
The time that went into it to get the model ready saved so much time on the
00:05:26.989 --> 00:05:32.629
back end in terms of the ability to, you know, get the documents through the door,
00:05:32.849 --> 00:05:37.089
out in the production, but feeling confident that we had been able to identify
00:05:37.089 --> 00:05:38.249
the privilege documents.
00:05:38.249 --> 00:05:43.749
And so I think that there's a real, you know, live use case really was effective.
00:05:44.309 --> 00:05:46.909
And I think it's something people really should be thinking about.
00:05:47.089 --> 00:05:52.049
Even where people are still, for reasons that escape me, nervous about using TAR in review,
00:05:52.509 --> 00:05:56.669
you know, using it to identify privilege for things that are going through the
00:05:56.669 --> 00:06:02.109
review to make your review better and to help people to identify privilege information,
00:06:02.109 --> 00:06:04.809
it really just seems like a no-brainer to me.
00:06:05.900 --> 00:06:10.680
Anthony: And I think, again, I think people get, and we've heard this from lots of clients
00:06:10.680 --> 00:06:15.880
and outside counsel, whatever, they're very nervous about using machine learning
00:06:15.880 --> 00:06:18.220
TAR in this, and it hasn't been used that much.
00:06:18.320 --> 00:06:24.500
I mean, the reality is that very, very, very few people and organizations are
00:06:24.500 --> 00:06:27.640
actually using even something like TAR, which has been around forever.
00:06:27.820 --> 00:06:33.040
Therese: Anthony, you mean not everyone is as enthusiastic about AI and e-discovery as you are?
00:06:34.120 --> 00:06:37.440
Anthony: It should be done. It should be done. We used it. But again,
00:06:37.600 --> 00:06:40.820
I think a few concepts here, and I think this is important, Therese,
00:06:40.900 --> 00:06:47.680
as you mentioned, is the way we used it wasn't, oh, AI made the privilege call, right? It's a tool.
00:06:47.820 --> 00:06:51.940
And I think we found it effective, right? What you said is it's one,
00:06:52.240 --> 00:06:53.860
establishing workflow, right?
00:06:53.980 --> 00:06:57.200
Which is we're scoring it. And people don't understand it's scored.
00:06:57.200 --> 00:07:02.000
So it could be a nine or whatever, and it's scored. And nine means that it's
00:07:02.000 --> 00:07:06.220
very likely to be privileged and zero is, you know, it's not going to be privileged at all.
00:07:06.320 --> 00:07:09.720
And obviously, you know, you can develop workflows based on that.
00:07:09.800 --> 00:07:12.300
And I think a lot of people that have used it have tried to use this.
00:07:12.380 --> 00:07:14.780
And we're going to be seeing that also, I think, with GenAI.
00:07:15.080 --> 00:07:17.860
So that's that concept of scoring the document on whether it's privileged.
00:07:18.200 --> 00:07:21.940
But again, if you're just using it to, you know, get a workflow,
00:07:21.940 --> 00:07:26.320
so you have people that are really focused on privilege, looking at that is helpful.
00:07:26.520 --> 00:07:30.200
And then, like you said, the QC, I mean, there's two areas where there's not
00:07:30.200 --> 00:07:31.640
a lot of risk associated with that.
00:07:31.700 --> 00:07:36.160
It's a tool that helps, but you're not saying, oh, I relied on AI to determine privilege.
00:07:36.400 --> 00:07:41.080
No, it's just helping your teams, whether it's workflow or QC,
00:07:41.280 --> 00:07:45.800
that to me is very low risk, which I agree with is why people should be using it.
00:07:46.000 --> 00:07:50.240
So Sam, in terms of, we talked a little bit about that example,
00:07:50.400 --> 00:07:53.480
which is helpful, which is historically, but.
00:07:54.098 --> 00:07:57.898
Sort of in the future, as we start thinking about what types of AI models or
00:07:57.898 --> 00:08:04.078
approaches are sort of, you know, going to be commonly used to identify privileged content, right?
00:08:04.158 --> 00:08:07.158
Is it, we talked about rules-based, machine learning, GenAI,
00:08:07.538 --> 00:08:11.558
what are your thoughts on, you know, sort of what the approaches are going to be?
00:08:12.278 --> 00:08:16.838
Samantha: Yeah, well, I mean, as we've discussed, machine learning is pretty tried and
00:08:16.838 --> 00:08:23.538
true in e-discovery and, you know, it does work quite well in,
00:08:23.898 --> 00:08:29.418
you know, when trained properly in identifying, you know, things that are likely to be privileged.
00:08:30.098 --> 00:08:37.038
I think what is sort of interesting or exciting about GenAI in this area is
00:08:37.038 --> 00:08:42.658
that instead of maybe providing a score, a GenAI can provide a rationale, right?
00:08:43.038 --> 00:08:46.298
This document is privileged because this.
00:08:46.818 --> 00:08:50.098
And, you know, I think that just sort of advances the,
00:08:50.098 --> 00:08:56.838
the ball a little bit in terms of what we're getting out of AI for privilege,
00:08:56.838 --> 00:09:04.278
possibly reducing the amount of human effort required in terms of training.
00:09:05.358 --> 00:09:11.278
When you're using TAR, there's still a fair amount of human review that has to happen.
00:09:11.438 --> 00:09:17.258
And ultimately, after the model runs through the data set, you still have to review everything.
00:09:17.478 --> 00:09:22.578
So I think the potential to maybe, you know, lighten the load even more and
00:09:22.578 --> 00:09:25.758
a little bit maybe more reliably is there with GenAI.
00:09:26.538 --> 00:09:30.098
Therese: Yeah. And I also think, and I think this is one of the areas,
00:09:30.418 --> 00:09:36.318
a little bit of ECA, but also for privilege purposes, one of the challenges
00:09:36.318 --> 00:09:38.398
with privilege is you don't know what you don't know.
00:09:39.091 --> 00:09:44.671
Right? So you don't know what privileged communications were happening necessarily.
00:09:45.011 --> 00:09:50.191
You don't know what topics were out there. And the number of times we've done a review,
00:09:50.351 --> 00:09:53.711
you've done everything you can, you give people all the information you can
00:09:53.711 --> 00:09:57.691
give them to what to look for for privilege, and then something gets produced
00:09:57.691 --> 00:10:01.171
and you're like, we didn't know that communication was happening and that should
00:10:01.171 --> 00:10:03.051
have been on our privilege list, right?
00:10:03.251 --> 00:10:08.411
I think one of the other promising things about GenAI is that it's interrogating
00:10:08.411 --> 00:10:12.511
your data early on to say, what should be privileged?
00:10:12.811 --> 00:10:16.331
What kinds of communications are out there? What kind of topics are out there?
00:10:16.891 --> 00:10:20.371
How do I use this to surface to better prepare? Like Sam said,
00:10:20.531 --> 00:10:23.751
better prepare if you're going to use TAR, better train the model because I
00:10:23.751 --> 00:10:26.311
found the things ahead of time that it needs to know.
00:10:26.631 --> 00:10:30.031
And I think that that's something that TAR can't do, right? Machine learning
00:10:30.031 --> 00:10:32.791
doesn't do that for you because you have to tell it what to look for.
00:10:33.231 --> 00:10:36.631
GenAI can find things for you. If you're asking the right prompts,
00:10:36.871 --> 00:10:40.631
if you're doing the right interrogation of the data, it can surface things for
00:10:40.631 --> 00:10:46.631
you that you don't even know about and then help you to better prepare for your privilege reviews.
00:10:46.851 --> 00:10:51.751
And I think that's an area that just really could make a material difference
00:10:51.751 --> 00:10:58.911
in helping with making sure that privilege reviews themselves have all the information
00:10:58.911 --> 00:11:03.291
they need to find the privileged information as you go through the review.
00:11:04.028 --> 00:11:09.148
Anthony: Yeah. And I know like some of the vendors out there are, are sort of doing a
00:11:09.148 --> 00:11:10.528
combination of, you know,
00:11:10.828 --> 00:11:14.808
our search terms, everything, and that's sort of their normal workflow,
00:11:14.808 --> 00:11:20.128
which is they have already developed a specific sort of identifier,
00:11:20.308 --> 00:11:24.268
a long language, large language model that is, that is geared for privilege,
00:11:24.268 --> 00:11:26.128
which is sort of your base.
00:11:26.308 --> 00:11:31.028
And Therese, your point, And it may get general stuff. And then you layer that
00:11:31.028 --> 00:11:34.528
and sort of hone the tool using TAR.
00:11:34.768 --> 00:11:38.448
So if you have a general model that's based on privilege, which could help identify
00:11:38.448 --> 00:11:42.068
privilege information, again, never rely on that, but it's a nice base.
00:11:42.208 --> 00:11:46.428
But then the concept of now I'm going to develop a privilege tool that's unique
00:11:46.428 --> 00:11:54.108
for this business line, this case, this organization, you can use samples, right?
00:11:54.208 --> 00:12:01.348
Like we do for TAR to train that tool better. so it's even more precise and more relevant.
00:12:01.708 --> 00:12:03.968
And obviously you would assume that the performance is going to be better.
00:12:04.408 --> 00:12:07.628
And then there's, I think, in terms of rule-based and stuff like that,
00:12:07.808 --> 00:12:12.628
you can start thinking about using search terms for when privilege may break, right?
00:12:12.768 --> 00:12:17.088
So if it's sent to a third party and those types of things, you can sort of
00:12:17.088 --> 00:12:18.568
model it and put it all together.
00:12:18.808 --> 00:12:22.888
But that means you sort of have to become an expert on how these tools work
00:12:22.888 --> 00:12:24.848
and the like, which is definitely part of it.
00:12:24.848 --> 00:12:29.608
But I think this is something I'm super excited about, because I do think this
00:12:29.608 --> 00:12:35.508
is an opportunity for us to really move the needle on how people conduct privilege.
00:12:36.028 --> 00:12:39.268
So, Therese, in terms of, obviously, we just talked about this,
00:12:39.508 --> 00:12:45.928
is obviously one of the concerns is how good are these tools in actually getting the nuance, right?
00:12:45.928 --> 00:12:51.488
And we know that's going to be the pushback from outside counsel and everybody
00:12:51.488 --> 00:12:56.928
is, it's too nuanced for these tools to work. What is the response to that?
00:12:57.970 --> 00:13:05.770
Therese: It depends on how you're using them and the oversight that's provided when you're using them, right?
00:13:05.970 --> 00:13:12.750
I mean, right now, are the AI tools great at making nuanced evaluations between
00:13:12.750 --> 00:13:17.070
attorney-client privilege or work product or things like that?
00:13:17.330 --> 00:13:21.670
No, but you're not relying on them for that. And I think that's the important thing.
00:13:21.850 --> 00:13:25.890
And I think they'll get better. Frankly, we've seen such incredible advances in AI.
00:13:26.190 --> 00:13:30.170
I think we'll see that move along pretty quickly. But I think it's important
00:13:30.170 --> 00:13:34.510
to remember, again, you're not asking the AI to make a decision for you.
00:13:34.650 --> 00:13:39.290
You are asking the AI to surface things for you to make so that a decision can
00:13:39.290 --> 00:13:42.530
be made and to find the things that are most likely to fit the criteria,
00:13:43.010 --> 00:13:45.210
right? And then you can do the review.
00:13:45.470 --> 00:13:51.750
So we're not asking AI right now to be able to be super precise in the type of privilege.
00:13:51.910 --> 00:13:55.070
As Sam already mentioned, There are some of these where it's actually helpful
00:13:55.070 --> 00:13:57.770
because it can give you a reason why it thinks.
00:13:58.170 --> 00:14:02.630
Not that it thinks. The reason why the algorithm is telling it that it may be privileged.
00:14:02.910 --> 00:14:06.790
So it can help to give you a background to make a decision.
00:14:07.070 --> 00:14:12.830
But the real goal of these tools is how do I find the things that are likely
00:14:12.830 --> 00:14:17.090
to be similar so I can make faster decisions? And they do that very well.
00:14:17.610 --> 00:14:22.030
So I think that it's always a nuance is that people, if you think you're using
00:14:22.030 --> 00:14:27.570
AI as a magic bullet that will give you answers and alleviate us from having
00:14:27.570 --> 00:14:29.550
to do any work, it will not do that.
00:14:29.650 --> 00:14:34.110
And we don't want it to do that right now, right? We want there to be a human element.
00:14:34.250 --> 00:14:37.870
Goodness gracious, it's hard when people are looking at documents and have debates
00:14:37.870 --> 00:14:41.710
over whether or not it should be considered privileged, right? It isn't an easy answer.
00:14:41.830 --> 00:14:46.350
But I think they are very good at surfacing categories, finding things that
00:14:46.350 --> 00:14:52.190
look alike, so that you can move, go better, faster, quicker in your reviews.
00:14:52.190 --> 00:14:54.130
And that's really what we're trying to do here.
00:14:55.710 --> 00:15:03.030
Anthony: Yeah, so, Sam, where do you see the use of AI where it could be helpful in terms of the privilege process?
00:15:03.570 --> 00:15:09.410
Samantha: What I would love to see is fully automated privilege logging.
00:15:12.430 --> 00:15:17.170
Privilege logs are the worst. They're totally a necessary evil.
00:15:17.610 --> 00:15:24.990
And they are very expensive and often not even used.
00:15:25.710 --> 00:15:31.390
I do actually think that generative AI could probably produce a pretty good privilege log.
00:15:31.630 --> 00:15:38.010
And I think, you know, as far as we've come streamlining the privilege logging
00:15:38.010 --> 00:15:40.190
process, it's still a slog.
00:15:40.410 --> 00:15:45.450
And I think to get any kind of meaningful privilege log is a slog.
00:15:45.630 --> 00:15:50.930
You can give a metadata log pretty easily. And I do think that large language
00:15:50.930 --> 00:15:56.290
model would probably make pretty quick work of summarizing very briefly,
00:15:56.290 --> 00:16:00.410
you know, 10,000 privileged documents for your adversary.
00:16:01.150 --> 00:16:05.230
Anthony: Yeah, no, and I think, look, I think there's cost issues associated with it.
00:16:05.310 --> 00:16:07.270
I think the cost will come down like everything else.
00:16:07.550 --> 00:16:12.490
I tend to think that this is probably an area where AI is going to dominate
00:16:12.490 --> 00:16:16.350
once it gets accepted and stuff and the price point comes down because as you
00:16:16.350 --> 00:16:18.690
said, it costs a ton of money,
00:16:19.150 --> 00:16:23.590
not much use either, but it's going to be AI tools talking to AI tools, right?
00:16:25.050 --> 00:16:27.910
You're going to submit your privilege log that was created by an AI tool with
00:16:27.910 --> 00:16:32.350
QC and the like, and then the other side is going to use their AI tool to analyze
00:16:32.350 --> 00:16:34.430
it and highlight things or whatever.
00:16:34.630 --> 00:16:38.030
So it'll be interesting, but I think there's an area where it definitely is
00:16:38.030 --> 00:16:41.330
going to work as soon as the cost comes down, because I think it's relatively
00:16:41.330 --> 00:16:44.130
expensive right now, but it is also awful.
00:16:44.370 --> 00:16:51.410
The other thing that I think is interesting is one of the challenges we have is that
00:16:51.983 --> 00:16:55.043
documents are tag privilege, but shouldn't be privileged.
00:16:55.263 --> 00:16:58.763
And then you don't find it until you do the privilege log. It's a nice way if
00:16:58.763 --> 00:17:03.203
you do it with, you know, if you're doing a GenAI tool, it's going to have
00:17:03.203 --> 00:17:05.803
a problem describing why it's privileged, right?
00:17:05.943 --> 00:17:09.103
So you can sort of see it and say, well, it doesn't know why it's privileged, right?
00:17:09.223 --> 00:17:13.583
So, well, that's maybe one that we should throw back and say it's, we can produce it.
00:17:13.683 --> 00:17:20.043
So it's almost like a QC of your privilege calls, because if the AI tool can't
00:17:20.043 --> 00:17:25.343
describe it as to why it's privileged, then maybe it shouldn't be privileged at all.
00:17:25.483 --> 00:17:31.763
So I do think that's an area where it's certainly ripe for people to start using.
00:17:31.943 --> 00:17:36.043
It'll be interesting to see if that becomes standard, that we start using these
00:17:36.043 --> 00:17:39.423
as a standard process if the price point is right, for all the reasons,
00:17:39.563 --> 00:17:40.383
Sam, that you mentioned.
00:17:40.583 --> 00:17:43.503
So exciting times. So thank you both.
00:17:43.823 --> 00:17:46.463
Again, this is on privilege. We may have others on privilege,
00:17:46.463 --> 00:17:52.183
But we'll be doing more of these podcasts on AI-enabled e-discovery, among various things.
00:17:52.323 --> 00:17:54.643
Hopefully you enjoyed it. If you have any questions, obviously you can reach
00:17:54.643 --> 00:17:55.883
out to Reed Smith. Thanks.
00:17:58.143 --> 00:18:03.143
Outro: Tech Law Talks is a Reed Smith production. Our producers are Ali McCardell and Shannon Ryan.
00:18:03.603 --> 00:18:07.123
For more information about Reed Smith's emerging technologies practice,
00:18:07.503 --> 00:18:09.963
please email techlawtalks@reedsmith.com.
00:18:10.463 --> 00:18:14.843
You can find our podcast on Spotify, Apple Podcasts, Google Podcasts,
00:18:15.003 --> 00:18:17.423
reedsmith.com, and our social media accounts.
00:18:18.103 --> 00:18:22.303
Disclaimer: This podcast is provided for educational purposes. It does not constitute legal
00:18:22.303 --> 00:18:25.583
advice and is not intended to establish an attorney-client relationship,
00:18:25.583 --> 00:18:29.723
nor is it intended to suggest or establish standards of care applicable to particular
00:18:29.723 --> 00:18:31.463
lawyers in any given situation.
00:18:31.843 --> 00:18:34.463
Prior results do not guarantee a similar outcome.
00:18:34.883 --> 00:18:38.903
Any views, opinions, or comments made by any external guest speaker are not
00:18:38.903 --> 00:18:43.543
to be attributed to Reed Smith LLP or its individual lawyers. All rights reserved.