TRANSKRIPTIO 🔗
speaker-0: With TA, legal teams defended the results with statistics. With Gen AI, we still do. The standard never changed. What changed is that attorneys can now read the reasoning behind every call, document by document, the way they were trained to defend their own work. That shift moved adoption further in a few years than TAR managed to in a decade. So the question in front of us now is a better one. How far upstream should this go? And also, when the proof is already on the table, why do legal teams often fall back on approaches that they've used in the past? Today, on the orchestration layer, we're looking back at a conversation Sashi recorded with Susan Stone, Director Legal Discovery at ATT, and Robert Keeling, e-Discovery Counsel and partner at Redgrave LLP. One sees adoption from inside the company, the other sits between the client and the trial team. I'm Brandon Jessup, Chief Delivery Officer at Simplify, and I'm joined by Sashi Vallavalla, SVP of Applied Legal AI at Simplify. Together, we're going to walk through the four moments that stuck with us most from that conversation with Susan and Robert. Sashi, welcome.
speaker-1: Thank you, Brandon.
speaker-0: So let's start with the question we hear more than any other. Why did generative AI catch on so much faster than tar? And I think Susan has an answer that has nothing to do with the technology getting better. You mentioned tar early on, and that you has always had a
speaker-1: Very slow burn adoption rate.
speaker-0: and I think a lot of what we're seeing with generative ability
speaker-1: B.
speaker-0: for the system to actually show its work. create some confidence in the attorneys that
speaker-1: It is
speaker-0: performing at the task on which it it was asked to do. whereas in the past where tar they had a just blind faith trust that what they were doing based on the information that's applied. So I think by having those visual cues and being able to see their work product, which is something I think a lot of attorneys are taught in law school, you know, and having product and and being able to claims for that work product, generative AI was a really tool and I love this point, Sashi. It's a really good one. I I think for you and I, we've talked a lot about TAR and this transition now for our industry as we move forward and and we get to Gin AI in instead of tar or in addition to tar. And for the two of us, I know we've had all these conversations about, well, why is it different? How do you justify it? And it always comes back to the statistics, right? We go, well, it's justified by the stats. We're running the same validations. This is how you can defend it, right? And we're real really we're talking about defensibility. And I I think sometimes We lose track of the fact that part of what we're trying to do, in addition to allowing attorneys to defend this process, is to get them to trust in the process, to believe that what's happening is valid, that they can, you know, sign
Load full transcript
their name on the the dotted line and go, Yeah, I'm I'm bought in here. I believe that this is a valid process. And the big difference that we're seeing, as Susan points out, with Gen AI solutions is that. They are giving you the reasoning document by document. So if you go in and you ask, why did this document get marked as responsive? It's going to tell you there's a blurb there that says this is why or not responsive. There's a blurb that tells you this is why. So I do think that's a really important point for us to make. So I I'm just curious, any thoughts on that? Obviously, you were in the room there with Susan and Robert.
speaker-1: Yeah, no, that that that was the real unlock. I mean, it's not the accuracy, it's not the auditability, it's about, you know, show your work, right? Like this is what what every lawyer is taught in the law school to say that, you know, provide a reasoning, provide citations, and then being able to bring those re rationale and and the citations to the forefront on why a document was classified. I think that's the turning point in in terms of having comfort in knowing that there is a reason why a document got classified in a certain way. It it feeds into the defensibility narrative, but much more so at a human level. It it provides a comfort to say that yes, this is the right document and this is why this document got classified. So in terms of when we are building this review workflows, we are showing our work, the Gen AI tools that that we advocate for and then prevalent in the industry. these are some of the tools that are giving the lawyers the visibility into it com compared to the TAR. TAR was purely a machine learning. I mean it it's really bad right.
speaker-0: The difference, right?
speaker-1: It's a black box. And not knowing why a got document got scored in a certain way always was the hindrance saying, like, you know, how do I know this is the best document? I know it it did get a high score. And now we are able to back that up with citations with the reasoning. And counter reasoning if if something was not to be in in the in the right classification zone. it does provide the Gen AI g gives you a little bit more insight.
speaker-0: Sashi, in many ways it's better than human review in the sense that you can go in and look and see why was this why was this made responsive or not responsive. Whereas, you know, having dealt with human review is as we both have on some level over the years, it can be a real mystery sometimes why a human reviewer made a call.
speaker-1: Yeah, exactly. Especially on the documents that that are in the gray zone that could, you know, depending on, you know, which side of the argument you take, it could fall into the responsive or the not responsive categories. And and it was always hard to reason on an attorney decision on why a document was coded in a particular way. So this is is like basically it comes down to I'm showing my work and here is every everything you need. And somebody can go look through those and and and agree. And if you don't agree, and that's okay. If you don't agree, you get to redo it. And then because the speed with which the generative AI classifies a document, you can go back, adjust your prompt, and then rerun it to to elicit a better classification. And this is all coming back to the iterative work product, right? Like you're asking the the the prompt. You're up you're asking for a classification using the prompt, and it can be iterated.
speaker-0: So i I I mentioned a moment ago the you know this conversation around the defensibility and and validation. And so Robert touches on that a little bit. So let's listen to a clip where he dives in a little bit on the dif defensibility side of things.
speaker-1: It's just a different way. just
speaker-0: of going about it. it
speaker-1: so happens that
speaker-0: it typically is outperforming search
speaker-1: More traditional ways such as
speaker-0: terms or manual, you know, human linear review or anything like that. It it's it's just a better mousetrap in that regard. And
speaker-1: But it just because it's new
speaker-0: two doesn't mean it should be treated differently. you can validate it the same way as you can either with search terms or with tar approach.
speaker-1: Cheers.
speaker-0: And there's really no difference in that regard on how it can be how the results can
speaker-1: Be validated.
speaker-0: I I mean, this goes back to, you know, I I I mentioned our conversations that we've had in the past about validation and that not a lot has changed here on that side of things. The defensibility is essentially the same. I I think sometimes people raise questions and they go, Well, how do we defend this? And and we simply say, Well, you go back to the statistics and you go back to the same types of approaches that we've taken with tar. Sashi, your thoughts on that? What has changed, if anything?
speaker-1: I don't think anything has changed. I mean, I agree with Robert's assessment here. Like, you know, the the burden of proof was never zero for manual reviews, right? We have been used to doing QCs even in the manual review stuff phases. And there are multiple levels of QCs and then there's and there are overturns in the in those QCs. That part of of saying that, you know, what you're obligated to produce and then making sure that you produce your productions are complete, that part did not change. And that part did not change with Todd. That part did not change in the manual review. They that part is not changing now with the generative AI. That burden of proof is still there. And and the sampling technologies, sorry, techniques that we have used, they still apply today. You are making sure that you're not leaving behind good documents and you do testing and you do statistical validation, you know, whether you reached a certain target recall, which is to ensure that your production is complete. So all of those are still the same. So a
hundred percent agree with with Robert's assessment there.
speaker-0: And I I as you were talking through that, Sashi, I have to say I I was thinking to myself, thank goodness that perfection was not the bar for human review, because we certainly weren't perfect. And your point is a good one that a lot of the validation that we talk about, even for tar, we were taking similar steps during human review where we had to go in and do some sampling and make sure that we felt like the quality was where it needed to be and that people were making consistent decisions. So not a lot has changed in some ways. So As we talk about AI solutions and how they fit into our workflows, we tend to think about AI as an alternative to tar. You know, we think of it in the context of determining responsiveness, for instance. But there are a lot more use cases. And Susan talked a little bit about moving further left of the EDRM and using AI in other contexts, including legal hold. So why don't we listen to that? So to Robert's question too, I would like to even go further left, right? As a corporate client,
speaker-1: How
speaker-0: you know, I would love to also see it maybe also implemented on the legal fold side. You know, as it stands now, legal bolts are evergreen
speaker-1: Typically ever
speaker-0: and or hold everything in place, right? But may
speaker-1: The b.
speaker-0: not be relevant to the matter, you know, maybe ten percent of what we're holding is maybe relevant or even one peristically.
speaker-1: Percent mobile.
speaker-0: so maybe even applying Gen AI to the legal fold process to fully fold the data relevant to that specific matter and allowing the other items to roll off based on your RIM policies. I mean, I think
speaker-1: that I mean would be a huge cost savings to to
speaker-0: to corporate clients or or any clients really. One percent, Sashi. So one percent of the data that a lot of our our corporate clients may be hanging on to and sitting on and have under legal hold may actually be applicable to the case. And so obviously it it seems like this is also an area where AI can help. So curious your thoughts here.
speaker-1: Yeah, no, this is this is more or less a governance play rather than a discovery play. Like you know, the b the farther left you're moving, now you're in the information governance realm. And and that's where some of the the AI will make a bigger difference over there because you know, if you can put a hold on what is actually necessary and what is important for a matter, I think that will be a huge cost savings for the clients, but but also this is a a risk management. the the less you hold on for the longer duration of the time, there is less risk in in in terms of, you know, overcollections. And then when you are overcollecting, could that be used in another litigation? And then could somebody compel you to produce because you have an overcollection. So this is definitely a pr under preservation is a harder sell than defensible reviews. So this is one of those information governance caveats that has to be abridged. And the real question is like, you know, who owns that? Is that legal or is that IT? I mean that that's that's where I I believe the industry will have to to move towards answering that question.
speaker-0: And it's probably a a partnership there with different stakeholders across the business and including both of those. You know, you made a really good point on the risk front, which I I think when companies are looking at how might you use AI when it comes to legal hold, you know, going back to this thought on 1%, there's mixed risk here, right? On the one hand, you have risk when you hang on to additional documents because they may end up being relevant to a future litigation. now you've held them for longer than your retention policies may have otherwise required. And so you're opening yourself up to risk. But on the flip side, and I I think you touched on this a bit, there is this, there's risk in not preserving enough. And, you know, probably courts are going to be less lenient if you say, you know, hey, we were really aggressive in what we actually held. There's probably going to be a higher bar there to hold on to this, to have a A bit more scope in terms of your legal hold than there would be, for instance, if you're going through and doing a review and saying, hey, we only produced the stuff we thought was relevant. Okay, well, let's let's keep moving here. I think that's a terrific conversation and and possibly for another time that we'll dig into more. But I want to go back to Robert. Again, you know, he is working with merits counsel. And in some instances, they're falling back on not just. Tar, but actually going back to leveraging search terms and and things like that. So let's hear from Robert. As Edith I think we are risk
speaker-1: Discovery Council, I think we're on the risk.
speaker-0: prefer side of the equation than
speaker-1: Then say typical
speaker-0: merits council, your your trial counsel or your white collar counsel.
speaker-1: but I think
speaker-0: We s part of our role is what we do is is act as a a little bit as a translator and
speaker-1: Between the client and
speaker-0: and outside merits council to them with
speaker-1: make them comfortable
speaker-0: AI.
speaker-1: and the application of AI in
speaker-0: And in particular
speaker-1: the application of AI
speaker-0: and nothing else. Like it typically our merits since
speaker-1: Council would want to actually use the same circle
speaker-0: terms or otherwise that they've always used because it makes them comfortable. But if you're
speaker-1: Some of
speaker-0: doing that that takes away of some of the advantages and efficiencies of AI and So for a little bit of background here, so you had asked the question where Robert and Susan typically disagree. And you know, the answer was they often don't. They're often fairly aligned and and both of them are obviously
steeped in in e-discovery expertise. They understand the tools, they understand the the risks. But Robert mentioned something that I I think is is worth fleshing out a little bit, which is Their role often and and Roberts in particular is serving as a translator, as he called it. So being this intermediary between these different stakeholders. Additional thoughts on that feedback from Robert.
speaker-1: Yeah, no, it's I mean that's the perspective they each bringing to the table. You know, Robert from a from a eDiscovery council and Susan from an in-house. I I think the to your point this they do in their own roles, they do they do not disagree very much. But this this point of translator, I think this is this is a key here. Running AI alongside the old search terms to make everybody comfortable is how the technology is gonna pay for itself, right? Like this is this is such a a comfortable place for the trial council to or the merits counsel to say, Yeah, I know what the search terms do. You get a hit, you get the br you bring the document in. And providing insights on top using Gen AI, I think that's the value. And this is a change management. I mean this is the orchestration and this is this is a change management I think that that Robert and in his capacity as an e discovery council is bringing to to the world and and bringing some organization around, you know, what what is the the true and trite tested methods for rediscovery in general. I mean look looking at it holistically all the way from the from the legal hold or or or more importantly, once it gets to the search terms and the review phase. And it's also a a place where you know there's it's not a blocker for by any means. And it's not again something that we we've always said, you know, Gen AI can be used complementary to other technologies. And this could be the layering of the technologies and that's the orchestration that that Robert is talking.
speaker-0: Sashi, I I really love it. You organically brought this full circle back to the orchestration layer, which of course, you know, is something that we're very invested in and thinking about how the human in the loop is is a key part of the equation here. And we go back a lot, I think, to this, this concept or this approach that that you have that we've talked about before, which is that this isn't about. AI replacing everything else. this is about you have another tool in your toolkit. You may mix and match your tools depending on on what the use case is. You may use one over another one. But if you have a team that's comfortable with search terms, it doesn't mean that it has to be search terms to the exclusion of AI. You may still be in a position where you're utilizing search terms, you're also utilizing AI in in some way. And of course, we're seeing different tools and applications now coming out with AI, you know, we we tend to think of there's there's AI for review, right? And and that now this is this alternative to traditional tar. But we have AI, you know, with assist in in relativity, for instance, where you can go in and and utilize that to gather information about your documents. And there are a lot of different use cases there. So Again, I think this comes back to that orchestration layer, having the folks who are knowledgeable about the tools that are available and who are working with all of the relevant stakeholders to make sure that they're comfortable with the process, they understand how to defend it and that it is defensible, and that they are getting that level of trust. And so that ability, you know, we talked about it in early on in this episode, that ability to trust what you're seeing. And then to defend it. So not missing the trust piece. Like we have to build that trust, being able to go in and see why has why has the AI said that this document is relevant or key or answers a specific question you're asking, or it's giving you a summary or an answer and it's pointing to those documents so you can go and see them and look at them. That really is a key element here in building that level of trust. So Any closing thoughts here, Sashi? I I know you had obviously a longer recording with these two. There were a lot of great things that you all discussed.
speaker-1: Yeah. I I think a adoption is a is a conversion problem. It's not a capability problem at this stage. where the advancements in the AI are pushing us more towards a level of comfort there. And that's that's what the cornerstone of, you know, where we will go in in the near future is going to be looking like. So it's a it's a conversion, it's a change management, and it is and they're all lead towards adoption.
speaker-0: Thank you, Sashi. Four clips, one thread. AI shows its work. It holds up to the same validation standards we have always applied. And it reaches well beyond review, all the way upstream to legal hold and beyond. The capability has arrived. What moves adoption now is the conversation between clients, eDiscovery council, and the trial teams who have to trust the results. If your organization Has adopted AI for review, but still runs the same search terms alongside it. That is not a technology gap. It is an opportunity to bring your trial team into the conversation earlier and capture the full value of tools you already have. Thank you to Susan and to Robert for a candid practical conversation. And Sashi, thank you to you for joining me. This is the orchestration layer where legal data, AI, and operations come together to create. Great outcomes that you can trust.