Understanding Active Learning Best Practices

100:00:00,000 --> 00:00:07,240Welcome to Reinventing Professionals, a podcast hosted by industry analyst Ari Kaplan,200:00:07,240 --> 00:00:12,640which shares ideas, guidance and perspectives from market leaders shaping the next generation300:00:12,640 --> 00:00:18,960of legal and professional services.400:00:18,960 --> 00:00:25,520This is Ari Kaplan, and I'm speaking today with James McGregor, the founder of Ethical500:00:25,520 --> 00:00:31,360E-Discovery, a boutique consulting firm that captures and reviews digital evidence and the600:00:31,360 --> 00:00:38,400chair of the litigation special interest group for the International Legal Technology Association.700:00:38,400 --> 00:00:44,000Hi James, how are you? Hi Ari, I'm good thanks. How are you? It's been a while since we did this.800:00:44,000 --> 00:00:50,800Yes, it's been about a year, so remind the listeners about your background and tell us about your900:00:50,800 --> 00:00:58,160work on Ilta's litigation special interest group. Yeah, sure, happy to. My name is James McGregor,1000:00:58,160 --> 00:01:04,160I founded Ethical E-Discovery and I've worked in the Discovery Industry for a long time,1100:01:04,160 --> 00:01:09,120I'm already at big consulting firms before I decided to go out and start my own venture,1200:01:09,120 --> 00:01:15,440and the reason that I wanted to do it is because I felt like the big firms do a great job of servicing1300:01:15,440 --> 00:01:21,600big clients, but increasingly with all the M&A activity in the space, those smaller boutique providers1400:01:21,600 --> 00:01:26,240have been gobbled up and formed and part of the larger conglomerates in this sphere.1500:01:26,240 --> 00:01:31,520And so I thought I'd start my own and service them at the mid market and the boutique market1600:01:31,520 --> 00:01:35,920that until now doesn't have the same choice that it used to have.1700:01:35,920 --> 00:01:42,000So part of what I've done in my professional career throughout is I've always volunteered1800:01:42,000 --> 00:01:47,440with various different organisations. I was president of ACES UK chapter for about seven years1900:01:47,440 --> 00:01:52,880until I joined Ilta and I chair their litigation special interest group here in the UK.2000:01:52,880 --> 00:01:59,520And we meet roughly once a quarter and we're a collection of heads of E-Discovery at all2100:01:59,520 --> 00:02:04,800burn major law firms that litigate in London, and we talk about a variety of different2200:02:04,800 --> 00:02:10,240issues and topics that are of interest to people that work inside a law firm department,2300:02:10,240 --> 00:02:16,320managing E-Discovery on the day to day. And what I wanted to speak to you about today is a project2400:02:16,320 --> 00:02:21,520that I've been working on for a long time. It's the combination of a huge amount of work from2500:02:21,520 --> 00:02:26,080a variety of different sources. Head to the Discovery ever shares at Redisport, a Chamberlain2600:02:26,080 --> 00:02:30,720that Alan's and over, a A&O Sherman, rather as it is now as it was of where we started this,2700:02:30,720 --> 00:02:36,320as well as Baker McKenzie, all the big law firms who have big internal E-Discovery departments,2800:02:36,320 --> 00:02:43,120their team members have been working with me on this best practices guide to using active learning.2900:02:43,120 --> 00:02:48,960And why you ask, we've all done something like this, so the reason being that3000:02:48,960 --> 00:02:56,000in the course of England of Wales, the practice direction states that for any disclosure3100:02:56,000 --> 00:03:01,440discovery exercise that parties are working on, where there are more than 50,000 documents3200:03:01,440 --> 00:03:07,360that need to be reviewed, professionals need to use some form of technology assisted review.3300:03:07,360 --> 00:03:14,400But what technology they use and how that technology is operated has never been written down3400:03:14,400 --> 00:03:18,480previously, in a dispute in England of Wales, when there were a document of3500:03:18,480 --> 00:03:23,360population of more than 50,000 and two sites would come to a dispute, on each occasion they would3600:03:23,360 --> 00:03:27,360have to negotiate for what technology are we going to use and how are we going to use it,3700:03:27,360 --> 00:03:32,800and effectively spend a fair amount of time negotiating in the early stages of the dispute3800:03:32,800 --> 00:03:36,960in order to decide what was a course of action that both parties were happy with.3900:03:36,960 --> 00:03:43,520And when in the sort of various filter meetings that we've been having, this problem arose4000:03:43,520 --> 00:03:47,920many times, we thought about collectively what we could do in order to solve it.4100:03:47,920 --> 00:03:53,680And the thinking was, if we were to develop this guide, this standard, and Ilters got some4200:03:53,680 --> 00:03:59,040precedent in this sphere, so priors of this they create something called the Documents Exchange Protocol,4300:03:59,040 --> 00:04:06,480which effectively lays out the mechanism whereby you will provide electronic disclosure to the other4400:04:06,480 --> 00:04:11,360side in a dispute in the course of England of Wales, just so that it effectively a kind of a4500:04:11,360 --> 00:04:16,160load file format to everybody knows what it is they're receiving. But building on that good work,4600:04:16,160 --> 00:04:20,480we thought why don't we develop a guide that talks to different law firms and their internal4700:04:20,480 --> 00:04:26,000clients and what technology to use in a dispute and how to use that technology so that if we can get4800:04:26,000 --> 00:04:31,840everybody to adhere to the same standards, it will save us a great deal of time and negotiation4900:04:31,840 --> 00:04:37,680at the start of the dispute and also at the encourage more people to use technology assisted5000:04:37,680 --> 00:04:43,120with you during a dispute, which is in everyone's best interest because it delivers for accurate5100:04:43,120 --> 00:04:49,040results and in a more cost effective manner. What is active learning and why should legal5200:04:49,040 --> 00:04:57,120professionals understand it? Active learning is a data scientist and ultimately it is the5300:04:57,120 --> 00:05:03,040updating of an algorithm. The easiest way to define it is that as you're going about and you're5400:05:03,040 --> 00:05:09,520conducting a document review, every time you tag a document as relevant, not relevant or you tag5500:05:09,520 --> 00:05:14,480it as relevant to a particular issue or not relevant, what you're doing is you're identifying to5600:05:14,480 --> 00:05:20,240the system that you are making a certain decision about that document and the system then will5700:05:20,240 --> 00:05:26,400automatically classify other documents that it believes are similar or not similar to that document5800:05:26,400 --> 00:05:32,160based on the decisions that you are putting in. So it's active learning in the sense that as you5900:05:32,160 --> 00:05:39,600actively code more documents, the system learns what it thinks you're going to code the next document6000:05:39,600 --> 00:05:47,200hats and so ultimately you're creating a brain in whatever software you're using and that becomes6100:05:47,200 --> 00:05:53,120like a reviewer in that it then looks at a document and will make a decision about whether it takes6200:05:53,120 --> 00:05:58,400its relevance or not relevant based on all the past decisions and it's also active because6300:05:58,400 --> 00:06:04,240as you're continuing to code more documents, your decision process may change and all the documents6400:06:04,240 --> 00:06:10,240are the documents that you have previously coded one way you may now code another way perhaps because6500:06:10,240 --> 00:06:15,840the case strategy has shifted and therefore the brain also adapts and the decisions that6600:06:15,840 --> 00:06:22,560we've made. I hope that answers your question. What are the benefits of applying standards to the use of6700:06:22,560 --> 00:06:30,960active learning? If you don't have standards it's a Wild West and so the situation that you end up in6800:06:30,960 --> 00:06:35,600is people are saying I'm going to use this technology and the other side is going to say you can't6900:06:35,600 --> 00:06:40,800use that technology we don't trust it we're not sure that what you're abiding by is a standard7000:06:40,800 --> 00:06:45,440if you think about the practice and this is the standard of using the reason to create a standard7100:06:52,000 --> 00:06:58,720So whenever your engagement disputes you will refer to precedent on either side of the arguments7200:06:58,720 --> 00:07:06,560in order to support your case and so when it comes to creating a standard in the absence of precedent7300:07:06,560 --> 00:07:12,240regarding particular technology what we're saying this consensus among the various7400:07:12,240 --> 00:07:18,240professionals at different law firms involved in this project is that we annoying to this process7500:07:18,240 --> 00:07:23,360we think this is the way to do things which means that when you're then engaged in a dispute7600:07:23,360 --> 00:07:28,640effectively you're saying we're going to follow the standard that everybody else follows7700:07:28,640 --> 00:07:35,360and in theory it shouldn't then be questioned or contradicted because it is an industry standard7800:07:35,360 --> 00:07:42,320it is a precedent if you like. Can you give us a sense of the outline of this best practices guide?7900:07:42,320 --> 00:07:48,400It's all available on the ill society you can see all of the contributors that are involved8000:07:48,400 --> 00:07:54,240it walks you through a definition of a variety of different types of terminology used in the8100:07:54,240 --> 00:08:01,120industry so people can understand exactly what learning means for example as well as a series of8200:08:01,120 --> 00:08:08,400other typical terms used in the process disclosure of discovery it then outlines various things to8300:08:08,400 --> 00:08:15,040consider when beginning an active learning project so it is a practical guide in a step-by-step sense8400:08:15,040 --> 00:08:19,920as to what you need to consider depending on the nature of the case depending on the types of8500:08:19,920 --> 00:08:25,280documents that you're dealing with and how to structure a project like this and then most importantly8600:08:25,280 --> 00:08:32,240there's a section on validation so after you use active learning on a project how can you then confirm8700:08:32,240 --> 00:08:38,400that the output is accurate and can you validate your results because that's important otherwise8800:08:38,400 --> 00:08:44,720there's no point in accepting that the machine has coded documents as a human reviewer would do8900:08:44,720 --> 00:08:49,680if you can't then validate that those results are accurate so there's a section of validation9000:08:49,680 --> 00:08:55,360and then finally there's a if you like a small synopsis that says and these are the best practices9100:08:55,360 --> 00:09:00,080based on everything you've read in the guide if you follow these core principles then9200:09:00,080 --> 00:09:04,960that is in place of reading the guide you should still go ahead and read the guide9300:09:04,960 --> 00:09:12,080and but those are the key takeaways how do you see the guide changing how law firms approach9400:09:12,080 --> 00:09:18,480disclosure we're doing a few things within the first thing that we've done is we've submitted it to9500:09:18,480 --> 00:09:24,720the master of the roles as this consensus documents with a view to hopefully it being incorporated9600:09:24,720 --> 00:09:31,600into English law so I mentioned that the practice direction PD57AD probably talks about the needs9700:09:31,600 --> 00:09:37,520to use technology as history review review on any project there's over 50,000 documents but it's not9800:09:37,520 --> 00:09:43,360prescriptive about how you do that our hope is that this guide will now be in a dendom to that part9900:09:43,360 --> 00:09:49,920of the practice direction which will in effect outline for people exactly how you can use technology10000:09:49,920 --> 00:09:55,920as history review that's the first step and by being incorporated law and there really is precedent10100:09:55,920 --> 00:10:03,120behind the work that's been done on this and it also opens the doorway for use of other artificial10200:10:03,120 --> 00:10:08,560intelligence so active learning is a form of artificial intelligence but obviously with the10300:10:08,560 --> 00:10:14,400advancement of generative AI and there are many other forms of artificial intelligence which are10400:10:14,400 --> 00:10:21,440not included in this best practice guide and as is challenging given the nature of how quickly this10500:10:21,440 --> 00:10:27,520artificial intelligence evolves you can't always be using the most cutting edge technology because10600:10:27,520 --> 00:10:33,920it has to be tested and but our hope is that by encouraging at greater use of active learning we're10700:10:33,920 --> 00:10:40,080opening the door for using more sophisticated artificial intelligence and hopefully there will be a10800:10:40,080 --> 00:10:46,640generative AI a dendom to this active learning best practices guide at some point in the future.10900:10:46,640 --> 00:10:52,320Where do you see disclosure headed in 2025?11000:10:52,320 --> 00:10:59,040That's a great question. Nobody knows ultimately do they and I would say that the11100:10:59,040 --> 00:11:04,320discovery industry and I know that your tangentially connected to this albeit you're not an active11200:11:04,320 --> 00:11:09,360practitioner yourself. There are many things that are optimistic about this space one being the11300:11:09,360 --> 00:11:15,040fact that the use of active learning hivots of this guide. I mean good example of this. It's11400:11:15,040 --> 00:11:21,040generally a field that is a pretty good adopter of technology and so as this technology quickly evolves11500:11:21,040 --> 00:11:27,680I think more and more people are adopting it and embracing it but that is balanced with the fact that11600:11:27,680 --> 00:11:34,160the law from the legal industries built on precedent and so whilst people are interested in using11700:11:34,160 --> 00:11:39,600this new technology they also don't want to open themselves up to exposure for having used something11800:11:39,600 --> 00:11:44,480which hasn't been tested and validated. So I think we're going to continue to move towards greater11900:11:44,480 --> 00:11:53,120adoption of advanced machine learning strategies to do the job that lawyers used to do but at the same12000:11:53,120 --> 00:11:59,600time it's not going to be a speed of light move so the boring answer is we're gradually going to12100:11:59,600 --> 00:12:06,560improve but improve we shall. You mentioned the availability of this resource where can listeners12200:12:06,560 --> 00:12:14,160find it? So the ill-toed guide is available on the ill-toed website and it's free to download. We love12300:12:14,160 --> 00:12:18,720as many people to you this is possible a lot of work went into this and that work will only be12400:12:18,720 --> 00:12:25,120worthwhile if lots of people read it and decide that they're going to follow it. This is Ari Kaplan12500:12:25,120 --> 00:12:31,040speaking with James McGregor the founder of Ethical E-Discovery or Boutique Consulting Firm that12600:12:31,040 --> 00:12:36,960captures and reviews digital evidence. He is also the chair of the litigation special interest group12700:12:36,960 --> 00:12:42,480for the International Legal Technology Association which has recently published a best practice12800:12:42,480 --> 00:12:49,200guide for using active learning. James thanks so very much. Thanks Ari really appreciate it.12900:12:49,200 --> 00:12:57,600Thank you for listening to the reinventing professionals podcast visit reinventingprofessionals.com13000:12:57,600 --> 00:13:02,240or areecaplandadvisors.com to learn more.