Relativity aiR: Real-world results – accelerating review and case strategy with GenAI

WEBVTT
00:00:13.173 --> 00:00:16.813
Intro: Hello and welcome to Tech Law Talks, a podcast brought to you by Reed Smith's Emerging Technologies Group. In each episode
00:00:16.813 --> 00:00:21.293
of this podcast, we will discuss cutting-edge issues on technology, data, and the law.
00:00:21.533 --> 00:00:26.333
We will provide practical observations on a wide variety of technology and data
00:00:26.333 --> 00:00:31.313
topics to give you quick and actionable tips to address the issues you are dealing with every day.
00:00:33.383 --> 00:00:37.183
Anthony: Hello, this is Anthony and Diana from Reed Smith's Emerging Technologies Group.
00:00:37.183 --> 00:00:39.823
I'm a partner here in the New York office of Reed Smith.
00:00:40.203 --> 00:00:46.703
And today, we are doing another of our podcast series on AI-enabled e-discovery,
00:00:46.883 --> 00:00:52.923
which hopefully will help legal departments determine when and if to use AI-enabled e-discovery.
00:00:53.363 --> 00:00:58.103
Our podcast series is really focused on practical and legal issues when considering
00:00:58.103 --> 00:01:00.703
using AI-enabled e-discovery.
00:01:00.903 --> 00:01:05.143
And we are going to focus on actual use cases and not just on the theoretical.
00:01:05.803 --> 00:01:10.123
And with that in mind, today we're going to be focusing on Relativity Air,
00:01:10.323 --> 00:01:15.083
obviously one of the big AI tools out there that was available to a lot of people.
00:01:15.363 --> 00:01:19.823
And joining me today are Marcin Krieger, who's counsel here at Reed Smith,
00:01:20.223 --> 00:01:23.643
part of the RED team and Emerging Technologies Group, and Sharri Wilner,
00:01:23.843 --> 00:01:28.023
who's part of Gravity Stack, our technology sub of Reed Smith.
00:01:28.303 --> 00:01:31.863
So thank you guys for joining me. Marcin: Yeah, thanks for having us, Anthony.
00:01:32.483 --> 00:01:36.763
Kind of cool thing to have Sharri here with us. You know, we are now part of
00:01:36.763 --> 00:01:42.803
Reed Smith Legal Solutions, which is really combining the technology and the legal side of things.
00:01:42.983 --> 00:01:48.523
And so the two of us are pretty excited to provide some perspectives on Relativity Air and our use of it.
00:01:49.123 --> 00:01:51.783
Sharri: Yeah, I'm excited to join you guys today. Thanks for having me.
00:01:52.663 --> 00:01:56.223
Anthony: Terrific. So Sharri, let's start with you. Since you're the technologist here,
00:01:56.423 --> 00:02:00.183
what is Relativity Air? Everybody hears about it. What is it? What does it do?
00:02:00.983 --> 00:02:07.483
Sharri: Yeah, absolutely. Relativity Air is Relativity's generative AI framework.
00:02:07.963 --> 00:02:15.663
It's built directly into Relativity One, and it brings the power of these large
00:02:15.663 --> 00:02:21.563
language models into the e-discovery and legal workflow environment.
00:02:22.896 --> 00:02:30.756
But because it's within Relativity's secure framework, it helps that it has
00:02:30.756 --> 00:02:35.076
that more defensible air to it, if you will,
00:02:35.336 --> 00:02:38.016
since it's in that particular cloud.
00:02:38.376 --> 00:02:47.916
But what makes it special or unique is that it's not a plugin or a third-party add-on.
00:02:47.956 --> 00:02:53.356
It's embedded right into the platform. So this means teams can work in there
00:02:53.356 --> 00:03:01.816
faster and hopefully smarter without sacrificing any control or concerns of compliance issues.
00:03:02.866 --> 00:03:05.866
Anthony: And what does it do? Like, how do you use it?
00:03:06.626 --> 00:03:11.306
Marcin: Yeah, so what's really cool about Relativity Air is that it doesn't really rely
00:03:11.306 --> 00:03:15.706
on the traditional technologies that we use in discovery.
00:03:16.106 --> 00:03:19.306
Unlike traditional technology-assisted review,
00:03:19.706 --> 00:03:25.226
which is sometimes called TAR or CAL, Relativity Air is leveraging the latest
00:03:25.226 --> 00:03:30.746
and greatest generative AI technologies, primarily but but not entirely built
00:03:30.746 --> 00:03:34.266
on top of OpenAI's GPT protocols.
00:03:34.566 --> 00:03:38.206
So this is a really new way of approaching discovery.
00:03:38.866 --> 00:03:42.486
It's a completely novel approach.
00:03:42.666 --> 00:03:46.546
Now it's being adopted broadly in the industry, and there are a lot of products
00:03:46.546 --> 00:03:49.726
out there that are leveraging the same technology, but it's novel in the sense
00:03:49.726 --> 00:03:54.426
that this is doing new things in new ways that we've never been able to do before.
00:03:54.866 --> 00:04:00.686
Sharri: And there are two key components to it right now. There's error for review,
00:04:01.046 --> 00:04:06.806
which assists in the document reviewers by suggesting coding decisions.
00:04:07.206 --> 00:04:13.106
And then there's error for case strategy, which is focused on deposition transcripts
00:04:13.106 --> 00:04:20.706
and summarizing testimony, you know, extracting issues and building factual narratives faster.
00:04:20.946 --> 00:04:25.526
Anthony: And then let's just focus on the relativity error of the review part.
00:04:25.526 --> 00:04:31.646
You mentioned that it suggests basically, I'm assuming it suggests whether it's responsive or not.
00:04:31.946 --> 00:04:37.066
Is that fair? Is that what the concept is or relevant to the litigation?
00:04:37.939 --> 00:04:42.279
Marcin: Yeah, it's interesting. So with things like Cal technology, when documents are
00:04:42.279 --> 00:04:46.519
reviewed, the algorithms look at other documents in the review universe, they assign a score.
00:04:46.719 --> 00:04:51.239
So you always hear about like a TAR or a Cal score of like zero to 100 or some
00:04:51.239 --> 00:04:57.319
other integer value, and it pulls similar documents based upon what human reviewers are doing.
00:04:57.319 --> 00:05:00.459
But what air does is
00:05:00.459 --> 00:05:03.879
completely different what air does is you tell
00:05:03.879 --> 00:05:06.599
air what makes a document responsive it's almost
00:05:06.599 --> 00:05:09.459
like you are giving it a document review protocol like what
00:05:09.459 --> 00:05:13.079
you would give to a junior associate or an e-discovery attorney aiR
00:05:13.079 --> 00:05:16.579
then reads that protocol and it looks at each document independently
00:05:16.579 --> 00:05:19.479
and it says based upon what I read up in this
00:05:19.479 --> 00:05:22.279
document i tell you it's highly likely to
00:05:22.279 --> 00:05:25.319
be relevant it's relevant it's maybe relevant it's
00:05:25.319 --> 00:05:28.699
not relevant or you know maybe there's an error or something else but
00:05:28.699 --> 00:05:32.219
rather than scoring documents zero to a hundred based
00:05:32.219 --> 00:05:38.319
upon how they are similar to other documents each document is looked at independently
00:05:38.319 --> 00:05:43.539
on its own based entirely upon the instructions that are provided to it through
00:05:43.539 --> 00:05:48.419
the the prompting that you know that like e-discovery protocol thing I provide
00:05:48.419 --> 00:05:51.399
but essentially it's a very complicated prompted.
00:05:51.559 --> 00:05:57.799
You hear the term prompt writing. This is prompt writing, God-tier level difficulty.
00:05:58.159 --> 00:06:02.439
You write really amazing prompts and you get really amazing results from the AI.
00:06:04.059 --> 00:06:07.739
Anthony: We're going to talk a little bit more on that because that seems like a challenge to me.
00:06:08.139 --> 00:06:12.459
But first, before we get there, based on your experience, and this is something
00:06:12.459 --> 00:06:15.859
that we talk about in some of the other podcasts where you have to look at a
00:06:15.859 --> 00:06:19.459
tool and evaluate it and determine what the best use cases are because they're
00:06:19.459 --> 00:06:20.379
not going to be good for everything.
00:06:20.519 --> 00:06:25.719
So for a relativity aiR review, what are the use cases that you're seeing
00:06:25.719 --> 00:06:27.359
where it's most effective?
00:06:28.153 --> 00:06:31.433
Sharri: Yeah, there are a few standout areas, Marcin, if you want to.
00:06:31.793 --> 00:06:36.933
Yeah, well, so I'll give a couple. Sharri has a lot more experience because
00:06:36.933 --> 00:06:39.933
the work that she does touches a lot of different practice groups.
00:06:40.193 --> 00:06:46.653
But myself being in litigation, where we see aiR being used the most frequently
00:06:46.653 --> 00:06:52.913
right now is in accelerated document review, where you have a fairly large data
00:06:52.913 --> 00:06:54.793
footprint, maybe not massive.
00:06:55.453 --> 00:07:00.853
But like, you know, data footprints, 50,000 to 100,000 documents where the cost
00:07:00.853 --> 00:07:03.273
of a human review makes it unfeasible,
00:07:03.513 --> 00:07:07.853
usually based upon things like the amounts of controversy of being able to get
00:07:07.853 --> 00:07:12.253
human reviewers to get through 100,000 documents may not be financially feasible.
00:07:12.473 --> 00:07:17.213
These tools are lower cost. And with a small investment of time up front,
00:07:17.413 --> 00:07:20.953
we can then use this technology to go through these documents,
00:07:21.353 --> 00:07:24.373
identify the ones that are responsive or relevant.
00:07:24.593 --> 00:07:28.593
But also, interestingly, aiR allows you to do issue coding.
00:07:28.973 --> 00:07:34.333
So another place that we see this being used is in environments where we already
00:07:34.333 --> 00:07:40.673
know the documents are highly likely to be relevant. Let's say it's an opposing party's production.
00:07:41.093 --> 00:07:46.293
And rather than reviewing them for relevance, because they're presumptively
00:07:46.293 --> 00:07:49.493
relevant because they're in response to our discovery requests,
00:07:49.733 --> 00:07:54.473
we can use aiR to identify which requests they are responsive to.
00:07:54.753 --> 00:07:57.973
So this would be used really well for categorization.
00:07:58.273 --> 00:08:02.733
In that way, let's say an opposing party produces 20,000 documents,
00:08:02.933 --> 00:08:04.993
presumptively all of them are going to be relevant.
00:08:05.333 --> 00:08:09.133
So traditional TAR technology doesn't really apply because TAR is going to say
00:08:09.133 --> 00:08:09.973
all of these are relevant.
00:08:10.464 --> 00:08:14.624
Human review team might go through these and categorize these documents into
00:08:14.624 --> 00:08:15.904
five different buckets.
00:08:16.124 --> 00:08:20.484
But then you have to take one of those five buckets, get an associate or a senior
00:08:20.484 --> 00:08:21.724
counsel to review that bucket.
00:08:21.904 --> 00:08:25.644
Now we can use aiR to categorize documents. And we tell aiR,
00:08:25.724 --> 00:08:30.084
here are our five buckets and our prompting defines them.
00:08:30.544 --> 00:08:34.864
That way aiR does that first pass. And then you could say, okay, great.
00:08:35.164 --> 00:08:40.384
Documents in bucket one go to our damages expert. Documents in bucket two go to senior counsel.
00:08:41.324 --> 00:08:46.344
Three might go to a junior associate that's just going to confirm that nothing was overlooked.
00:08:46.604 --> 00:08:53.544
But using aiR to as a culling method for very rich document populations is something
00:08:53.544 --> 00:08:56.884
that you couldn't do with traditional TAR without basically having to review
00:08:56.884 --> 00:09:00.344
the whole universe. Now we can use aiR to really cut things down.
00:09:00.844 --> 00:09:04.684
Another useful place, and Sharri, I'll let you provide some insights that you've
00:09:04.684 --> 00:09:07.284
worked with aiR on, are things like internal investigations.
00:09:07.284 --> 00:09:12.544
So we're outside of discovery, now we're dealing with very narrow issues,
00:09:12.544 --> 00:09:17.124
but very broad populations of documents, sort of the inverse.
00:09:17.424 --> 00:09:20.864
In my last example, we were talking about a universe where everything was relevant.
00:09:21.064 --> 00:09:25.204
Here, we're talking about a universe where less than 1% of documents are relevant.
00:09:25.324 --> 00:09:29.744
If we were using traditional TAR algorithms, we probably would still have to
00:09:29.744 --> 00:09:35.824
review 25% of the population and then still have to go through a second or a
00:09:35.824 --> 00:09:38.544
third round to really get to those super key documents,
00:09:38.764 --> 00:09:45.224
with appropriate prompting, we can conduct an internal investigation in hours instead of months.
00:09:45.504 --> 00:09:51.864
And we can identify the 100 most important documents out of a population of tens of thousands.
00:09:52.304 --> 00:09:56.624
Instead of the traditional method of tens of thousands, you still have to review
00:09:56.624 --> 00:10:00.264
10,000, which leaves you with 5,000.
00:10:00.524 --> 00:10:03.244
And then an associate has to get those down to those couple hundred.
00:10:03.444 --> 00:10:07.524
Now aiR can really get you to those couple hundred documents,
00:10:07.524 --> 00:10:12.364
not literally instantaneously, but in terms of the timescale of a traditional
00:10:12.364 --> 00:10:13.904
internal investigation,
00:10:14.344 --> 00:10:16.484
the equivalent of nearly instantaneously.
00:10:16.764 --> 00:10:20.104
Anthony: Yeah. The speed to knowledge is the big call.
00:10:20.484 --> 00:10:22.684
Marcin: Exactly. Anthony: Internal investigations and the use of AI.
00:10:23.184 --> 00:10:26.504
So Sharri, what else are you thinking in terms of use cases that you've seen
00:10:26.504 --> 00:10:27.964
that it's been particularly helpful?
00:10:28.658 --> 00:10:32.398
Sharri: Transcript summarization and deposition preparation.
00:10:33.138 --> 00:10:37.578
So using error for case strategy application
00:10:37.578 --> 00:10:46.458
to help summarize key points and align particular issues within transcripts
00:10:46.458 --> 00:10:54.798
to help prepare for additional depositions and or really just intelligence driving.
00:10:54.798 --> 00:11:01.958
So instead of worrying about taking a set of data to just do review,
00:11:02.318 --> 00:11:07.078
now instead you're driving the intelligence piece of it.
00:11:07.258 --> 00:11:12.818
So reviewing those, having it help you review those transcripts to make smarter
00:11:12.818 --> 00:11:19.978
questioning for future depositions and, you know, certainly upcoming trials.
00:11:19.978 --> 00:11:25.518
So it's that knowledge capture piece of it, I think, is really where we're seeing good use cases on that.
00:11:25.738 --> 00:11:29.838
I mean, Marcin, you've used it, I think, a few times in that particular respect.
00:11:30.278 --> 00:11:36.718
And soon what we're hoping to be able to do with aiR Assist is being able to
00:11:36.718 --> 00:11:39.398
surface those ‘hot’ documents,
00:11:39.638 --> 00:11:42.738
if you will, Marcin, the ones that you're referring to in investigations,
00:11:42.738 --> 00:11:48.978
to bring those really up to the surface quicker with respect to,
00:11:48.998 --> 00:11:49.778
you know, understanding
00:11:49.978 --> 00:11:55.738
very particular issues and themes within a investigation.
00:11:56.907 --> 00:12:00.187
Marcin: Yeah, that's right. And Sharri, you mentioned something pretty important,
00:12:00.347 --> 00:12:01.847
which maybe we didn't clue in on.
00:12:02.227 --> 00:12:06.287
Within aiR, we mentioned that there is aiR for review and aiR for case strategy,
00:12:06.447 --> 00:12:11.567
but Relativity just recently this month at Relativity Fest announced that they
00:12:11.567 --> 00:12:14.867
are opening an entire new suite of aiR products.
00:12:14.867 --> 00:12:19.927
One of them being, they call it aiR Assist, which is taking these workflows,
00:12:20.447 --> 00:12:25.487
this, Anthony, you call it speed to knowledge, our quest to find key documents
00:12:25.487 --> 00:12:29.087
faster, and they are making new tools around it.
00:12:29.087 --> 00:12:33.227
So I described how we could use aiR for review to find key documents,
00:12:33.467 --> 00:12:38.207
but Relativity is really listening to its clients, and they're now coming out
00:12:38.207 --> 00:12:42.407
with completely different feature sets on how to do early case analysis,
00:12:42.807 --> 00:12:46.067
how to do early key document and issue spotting,
00:12:46.187 --> 00:12:51.747
and taking a lot of those early case assessment tools that already may have
00:12:51.747 --> 00:12:56.367
existed on the market from other providers, but approaching them in a new way
00:12:56.367 --> 00:12:59.007
using generative AI instead of traditional,
00:12:59.727 --> 00:13:03.107
algorithms and programmatic tools for
00:13:03.107 --> 00:13:07.207
things like identifying the right custodians identifying the right date ranges
00:13:07.207 --> 00:13:11.367
identifying the right hot spots in your data set. Anthony: All right then that's great.
00:13:11.367 --> 00:13:15.487
Let's talk a little bit about the challenges and obviously every use case
00:13:15.487 --> 00:13:18.687
is going to be you know there's going to be certain challenges so Sharri why
00:13:18.687 --> 00:13:22.127
don't we talk first with you in terms of some of the technical challenges.
00:13:22.969 --> 00:13:29.189
Sharri: Yeah, absolutely. From a technical standpoint, some of the challenges that I'm
00:13:29.189 --> 00:13:34.809
seeing for reviewers happens to be something as easy as change management, right?
00:13:35.289 --> 00:13:44.489
Understanding that we are now otherwise needing reviewers to more clearly communicate
00:13:44.489 --> 00:13:49.349
the background and issues within the case, right?
00:13:49.729 --> 00:13:57.689
And being able to get them to think about those at the forefront before any review starts,
00:13:57.689 --> 00:14:05.889
which also then brings to one of the most challenging from a technical standpoint that we see,
00:14:05.969 --> 00:14:08.129
and we alluded to earlier,
00:14:08.549 --> 00:14:10.069
is the prompting skills.
00:14:10.609 --> 00:14:17.569
aiR responds best to specific contextual prompts, as Marcin had indicated.
00:14:18.828 --> 00:14:24.768
Teams really need to practice phrasing questions that are going to get them
00:14:24.768 --> 00:14:27.208
the meaningful answers that they're looking for.
00:14:27.648 --> 00:14:30.888
Marcin: So building on that, you're absolutely right, Sharri. Some of the challenges
00:14:30.888 --> 00:14:34.488
that we see from the legal side, I'll start off with what you said,
00:14:34.848 --> 00:14:39.568
which is getting a case team to articulate what it needs out of the documents
00:14:39.568 --> 00:14:42.548
at the start of the review is a lot harder than people think.
00:14:43.008 --> 00:14:48.008
Most document review workflows are you start with your protocol with what you
00:14:48.008 --> 00:14:50.248
think your issues are, you start working through the documents,
00:14:50.408 --> 00:14:52.828
you're three days in, you're changing course. You're a week in,
00:14:52.928 --> 00:14:53.668
you're changing course.
00:14:53.868 --> 00:14:57.488
With aiR, changing course isn't really an option. You really have to have that
00:14:57.488 --> 00:15:01.288
roadmap set early on, and you have to know how to articulate that roadmap.
00:15:01.448 --> 00:15:02.888
Like Sharri said, prompt writing.
00:15:03.268 --> 00:15:07.308
From the legal perspective, we have a lot of novel challenges.
00:15:07.628 --> 00:15:13.348
We are over a decade past the cases that said that TAR is, you know,
00:15:13.508 --> 00:15:15.708
accepted, right? Judge Peck's early opinions.
00:15:16.088 --> 00:15:20.788
Now we are still fighting over the use of TAR, and now we're introducing a whole
00:15:20.788 --> 00:15:23.868
new technology, a whole new way of approaching document review.
00:15:24.188 --> 00:15:26.928
So we have challenges in terms of.
00:15:27.488 --> 00:15:32.928
Client buy-in, getting clients past their reluctance about the use of new technology.
00:15:33.468 --> 00:15:37.328
When we're not even in a litigation context, you have clients who just don't
00:15:37.328 --> 00:15:40.888
trust it yet, who are concerned that they're going to miss a needle in a haystack.
00:15:41.248 --> 00:15:45.208
And the challenge is effectively communicating to them that if you had humans
00:15:45.208 --> 00:15:50.148
review all of your documents, you're still going to have a high risk of losing
00:15:50.148 --> 00:15:52.928
the needle in a haystack. It's called a needle in a haystack for a reason.
00:15:53.488 --> 00:15:58.428
The idea that someone's using secret code words and a human review team looking
00:15:58.428 --> 00:16:02.668
at every document and spending a million dollars somehow is going to find that
00:16:02.668 --> 00:16:07.228
code word better than an AI which is designed to pick up on those things and
00:16:07.228 --> 00:16:08.468
say, oh, you're using code words.
00:16:08.588 --> 00:16:12.988
So that's our first challenge. Next challenge is in the actual litigation context.
00:16:13.248 --> 00:16:17.048
We don't yet have judicial acceptance of the use of generative AI technology.
00:16:17.288 --> 00:16:21.588
So if clients are deploying it in the litigation context, we have to make sure
00:16:21.588 --> 00:16:27.548
that what we are doing conforms to the existing rules, the existing case law.
00:16:27.708 --> 00:16:31.488
There is a framework of case law that allows for the use of these types of tools,
00:16:31.488 --> 00:16:33.328
even without judicial acceptance.
00:16:33.548 --> 00:16:37.748
But we have to make sure that we are conducting ourselves in an ethical way
00:16:37.748 --> 00:16:42.068
so that should questions arise, our work is defensible.
00:16:42.628 --> 00:16:46.728
And if it ever gets in front of the judge, we want to make sure that we are
00:16:46.728 --> 00:16:50.528
the law firm that gets a judge to say, I've looked at what you're doing and
00:16:50.528 --> 00:16:54.588
I agree, and not the law firm that says, I looked at what you're doing and I have concerns.
00:16:55.368 --> 00:16:59.428
Anthony: Yeah, Marcin, on that side, I'm obviously with TAR, there was a lot of discussion
00:16:59.428 --> 00:17:02.388
about validation, right? How do I know I'm getting everything?
00:17:02.848 --> 00:17:06.708
And there was lots of sampling after the fact and whatever. We all developed
00:17:06.708 --> 00:17:09.888
really good processes, which is why judicially acceptanced.
00:17:11.168 --> 00:17:16.508
Are we using the same type of thing for Relatively aiR in gen AI tools?
00:17:16.808 --> 00:17:20.068
I mean, it's not the same way of looking at it. So I'm just wondering,
00:17:20.648 --> 00:17:23.068
How do we validate? What is the validation?
00:17:23.806 --> 00:17:28.106
Marcin: Right. So validation is the name, you know, that's the real name of the game.
00:17:28.346 --> 00:17:31.826
It doesn't matter what technology you're using. If you're not validating the
00:17:31.826 --> 00:17:33.726
technology, you're not using it correctly.
00:17:34.246 --> 00:17:38.666
Another way of saying validating, validating is just a nice way of saying you're using it the right way.
00:17:39.086 --> 00:17:43.826
Now, the traditional way to validate a TAR review is through statistical sampling.
00:17:44.186 --> 00:17:48.426
Very simple, but very well accepted and understood mathematical principles.
00:17:48.746 --> 00:17:52.746
You have the universe of documents you've identified that are responsive.
00:17:52.986 --> 00:17:56.886
You have the universe of documents that the AI says are not responsive.
00:17:56.886 --> 00:18:00.906
It doesn't matter if you're using TAR or Gen AI or something completely different.
00:18:01.166 --> 00:18:05.766
There is a very basic way that you can validate, which is called illusion testing,
00:18:05.766 --> 00:18:07.326
and that's a whole separate podcast.
00:18:07.626 --> 00:18:12.126
But you basically take statistically significant samples out of the stuff that
00:18:12.126 --> 00:18:16.826
the AI says are not relevant, and you can calculate how many relevant documents did you miss?
00:18:17.226 --> 00:18:22.266
The challenge with generative AI workflows is that with TAR,
00:18:22.486 --> 00:18:27.006
we knew that the algorithms pulled together documents that are similar to the
00:18:27.006 --> 00:18:28.206
ones you've coded responsive.
00:18:28.446 --> 00:18:32.586
So if you code a document that talks about combs as responsive,
00:18:32.806 --> 00:18:34.326
other docs about combs get pulled in.
00:18:34.486 --> 00:18:38.246
And so there's like a conceptual understanding that, yeah, like this is all
00:18:38.246 --> 00:18:41.006
sort of like homogenized and separated and we can sample.
00:18:41.286 --> 00:18:44.966
With validating generative AI, that validation
00:18:44.966 --> 00:18:48.186
process starts on day one it starts
00:18:48.186 --> 00:18:51.186
with validating the prompts themselves you have
00:18:51.186 --> 00:18:55.386
to make sure that you start your document review very carefully and slowly with
00:18:55.386 --> 00:19:01.686
a very small universe of documents let's say 50 to 100 you test aka validate
00:19:01.686 --> 00:19:06.466
your prompts against them you make sure that the results are incredibly correct
00:19:06.466 --> 00:19:10.406
very accurate as in the 50 responsive docs you put into your seed set,
00:19:10.546 --> 00:19:15.486
the AI correctly identified them as responsive and the rationale it gave makes sense.
00:19:15.746 --> 00:19:18.706
And the documents that you put in there that are not responsive also make sense.
00:19:18.846 --> 00:19:21.566
Then you expand that to 500 documents, then a thousand.
00:19:22.231 --> 00:19:25.331
Then you can turn it on on your entire document review population.
00:19:25.331 --> 00:19:29.191
So you're validating for errors before you run it on the whole.
00:19:29.651 --> 00:19:33.931
Now after you've run it on the whole, I think the same workflows that you would
00:19:33.931 --> 00:19:38.391
implement in a human review kick in, you do things like look for file names
00:19:38.391 --> 00:19:41.351
that should be relevant, look for concepts that should be relevant.
00:19:41.591 --> 00:19:44.431
But these are validation steps that you would take in a human review as well
00:19:44.431 --> 00:19:46.451
to make sure that your human reviewers aren't missing things.
00:19:46.631 --> 00:19:49.891
And as the last step, you can do that random sampling
00:19:49.891 --> 00:19:52.911
and you can do that illusion test but the main
00:19:52.911 --> 00:19:57.871
difference is we spent a decade with TAR and CAL making algorithms that are
00:19:57.871 --> 00:20:03.271
resistant to errors early in the process so you can validate later generative
00:20:03.271 --> 00:20:08.651
AI has no tolerance for errors at the start of the process a lot of the validation
00:20:08.651 --> 00:20:11.491
work happens before you even kick off the full project,
00:20:12.302 --> 00:20:16.302
Anthony: And that leads us to sort of the best practices, which is like,
00:20:16.482 --> 00:20:18.242
that seems like a lot of work. We'll see.
00:20:18.382 --> 00:20:23.722
And obviously, I think as you alluded to, Sharri, is you have to know what you're doing, right?
00:20:23.922 --> 00:20:28.822
So I think best practices, don't do this on your own without having someone
00:20:28.822 --> 00:20:30.222
who knows what they're doing.
00:20:30.362 --> 00:20:33.362
Because the way you explained it, I couldn't do this. If I started,
00:20:33.502 --> 00:20:34.422
I wouldn't be able to do this.
00:20:34.702 --> 00:20:40.422
I assume I would learn every time. But Sharri, what other best practices are you seeing?
00:20:40.562 --> 00:20:43.642
Because we have to close up here. But what other best practices besides what
00:20:43.642 --> 00:20:44.722
Marcin said in validation?
00:20:45.722 --> 00:20:49.262
Sharri: It actually sums up everything that Marcin was saying. Start small,
00:20:49.842 --> 00:20:57.262
pilot an error in one workspace with very specific issues, validate often,
00:20:58.242 --> 00:21:03.422
so you're comparing your AI results even with human review samples,
00:21:04.042 --> 00:21:12.382
document the process so that you are, you know, documenting what your findings are.
00:21:12.382 --> 00:21:17.102
So you're keeping records of your prompts, your validation results,
00:21:17.622 --> 00:21:23.222
your workflows, and all that's going to help strengthen your defensibility.
00:21:23.742 --> 00:21:27.982
Train your team, right? Make sure that, you know,
00:21:28.202 --> 00:21:36.522
your technical reviewers are on standby with you to help you craft those good
00:21:36.522 --> 00:21:39.402
prompts and how to interpret the results.
00:21:39.802 --> 00:21:42.182
And then iterate continuously.
00:21:42.822 --> 00:21:48.942
Treat error like it's a teammate. Refine your process and, you know,
00:21:49.222 --> 00:21:56.882
capture your lessons learned so that you can take these lessons onto larger, more complex use cases.
00:21:57.818 --> 00:22:02.098
Marcin: So my best practice, going back to what you said, Anthony, don't try to do it
00:22:02.098 --> 00:22:05.818
on your own and partner with people who know what they're doing.
00:22:06.118 --> 00:22:10.258
Partner with people like Sharri, who understand the technology and the process,
00:22:10.258 --> 00:22:15.818
who can detect when things are going wrong before things really go wrong.
00:22:16.338 --> 00:22:20.158
Have the right experts in the room with you, and then you don't have to be an
00:22:20.158 --> 00:22:23.258
expert, but trust your experts and listen to them.
00:22:23.698 --> 00:22:28.018
Anthony: And as we know, Marcin, you have an ethical obligation to do that as well.
00:22:28.318 --> 00:22:32.138
And I think, look, I think there is certainly risk here, and I'll conclude on this.
00:22:32.298 --> 00:22:35.538
So I think making sure you know what you're doing, you have an ethical obligation
00:22:35.538 --> 00:22:37.298
to know what you're doing when using these tools.
00:22:37.638 --> 00:22:41.458
I think this is the wave of the future, but I think it's going to take some time.
00:22:41.618 --> 00:22:47.238
And my fear is that we're going to have that bad case that leads to disaster, right?
00:22:47.358 --> 00:22:49.658
Where someone doesn't know what they're doing and the court's like,
00:22:49.778 --> 00:22:52.898
why are you using this tool at all? So hopefully that won't happen.
00:22:53.118 --> 00:22:57.018
So best practice, make sure you have good people who know what they're doing.
00:22:57.238 --> 00:23:00.418
We're going to have many other podcasts to talk about this.
00:23:00.558 --> 00:23:04.078
We may have another one just on Relativity aiR if we have other use cases that
00:23:04.078 --> 00:23:06.078
we can think of. But thanks, everybody, for joining.
00:23:06.338 --> 00:23:09.338
And thank you both. Marcin: Thank you. Sharri: Thanks again, Anthony.
00:23:11.818 --> 00:23:16.598
Outro: Tech Law Talks is a Reed Smith production. Our producers are Ali McCardell and Shannon Ryan.
00:23:17.038 --> 00:23:20.578
For more information about Reed Smith's emerging technologies practice,
00:23:20.978 --> 00:23:23.418
please email techlawtalks@reedsmith.com.
00:23:24.338 --> 00:23:28.298
You can find our podcast on Spotify, Apple Podcasts, Google Podcasts,
00:23:28.478 --> 00:23:30.858
ReedSmith.com, and our social media accounts.
00:23:31.618 --> 00:23:35.778
Disclaimer: This podcast is provided for educational purposes. It does not constitute legal
00:23:35.778 --> 00:23:39.058
advice and is not intended to establish an attorney-client relationship,
00:23:39.058 --> 00:23:43.218
nor is it intended to suggest or establish standards of care applicable to particular
00:23:43.218 --> 00:23:44.898
lawyers in any given situation.
00:23:45.318 --> 00:23:49.078
Prior results do not guarantee a similar outcome. Any views,
00:23:49.198 --> 00:23:53.018
opinions, or comments made by any external guest speaker are not to be attributed
00:23:53.018 --> 00:23:56.998
to Reed Smith LLP or its individual lawyers. All rights reserved.