Voiceover (00:03):
Welcome to the Future Work Playbook. This podcast series features founders, investors, and legal talent that will help you embrace technology and transform your organization for a better future. This series is hosted by Natalie Pierce, the chair of Gunderson Dettmer's Labor and Employment Law practice. Natalie and her guests are committed to helping you develop new playbooks to elevate your game.
Natalie Pierce (00:29):
Hi, this is Natalie. Welcome to our final episode of season five, featuring experts and advisors in the field of generative AI. If you haven't already listened to our previous episodes, they cover how to get started with generative AI, privacy and security strategy, and shaping the law firm of the future. This episode will briefly touch on these topics, but we'll quickly dive into killer use cases for gen AI.
(01:04):
Today, we welcome two visionaries in the field of generative AI, Dr. Lance Eliot, a regular Forbes contributor and AI fellow at Stanford University, plus Dazza Greenwood, a renowned Fortune 100 legal tech advisor and the executive director of law.mit.edu. Both gentlemen have been leading the industry with their research and publications. They routinely counsel attorneys, CEOs, educators, and business leaders. They're both on the cutting edge of gen AI trends, developing roadmaps for legal teams, engineers, and executives to transform the future. Dazza and Dr. Eliot, thank you so much for joining us.
Dazza Greenwood (01:57):
Oh, thank you. It is such a pleasure to be on the show. I'm a longtime listener, first time caller.
Natalie Pierce (02:04):
I love that.
Dr. Lance Eliot (02:05):
This is Lance Eliot, and I also would like to say thank you so much for having me on the show. I'm very excited about the coverage that you'll be doing.
Natalie Pierce (02:13):
Well, it is such a true privilege to have you both joining us, so looking forward to this discussion. Okay, just to kick us off, wow, what a year it's been. Most of us never imagined that we'd have such a powerful tool at our disposal. Would you each mind sharing with our listeners initial reactions when ChatGPT first hit the scene?
Dazza Greenwood (02:43):
This is Dazza Greenwood and my initial reaction when ChatGPT first hit the scene was one of astonishment and delight because I wasn't expecting it. I was familiar with GPT 2 and GPT 3 and comparable models and had been actually in the midst of some projects pushing them for all they were worth for legal contexts. ChatGPT just appeared one day. Somebody emailed me about it. I started using it.
(03:13):
Well, I'm sure you're familiar with it, but it really works very astoundingly well. It was just cutting through legal use cases and challenges one after the other when I started. I honestly still haven't gotten over it. It's almost a year now, and I am still just delighted and astonished by what it can do. I know we'll get into the limits and the risks, but my first reaction, which continues to be my prominent reaction, is wow, this stuff is really good. It's a serious advance over what was available, broadly available, just really for all time prior to this.
Natalie Pierce (03:51):
Yeah, a lot of us having those OMG moments. I think I recall you saying, Dazza, at one point you said, "It's a tool for us," like you said, to a room full of lawyers and law students. It's a tool for all of us. How about you, Dr. Eliot?
Dr. Lance Eliot (04:11):
Yeah, so echo the same comments that were just made and would add a little bit of an additional item to it, which is that for many of the AI insiders, we knew that what OpenAI, the maker of ChatGPT, was doing is they were trying to overcome a prior big problem for generative AI systems that were released, which was that oftentimes, right away, they would get into trouble because they would emit offensive outputs or people would try to spur them to do that. The question was, would ChatGPT get caught up in that same aspect that right away it might become something that people would complain about because it was saying things that we would not expect in polite society to be said?
(04:59):
It was known that they were using a technique called RLHF, reinforcement learning from human feedback, trying to try to hone ChatGPT, so that it wouldn't get quite as caught up in that. When it came out then, what did surprise me was how effective that had been to make it that this was something that the public at large could use and not get all sorts of just wild, zany, and insulting comments coming out of it. Then of course, it took off after that.
Natalie Pierce (05:29):
Yeah, it took off in a very, very big way. Dr. Eliot, you wrote, I believe it was last month in force, something that I very much agreed with you when you said it. The reality is that we're on a conveyor belt that is going to keep pushing us along on advancing AI and leveraging AI into all quarters of society. I think the reason why we are seeing that now, why we'll continue to see that is because yeah, it's surprising how good it is, how good these new iterations. We have new language models and iterations coming out all the time, but it is truly astounding.
Dr. Lance Eliot (06:12):
Absolutely. For any listeners that somehow haven't played with generative AI, which you'd almost have to, I think, live in a cave not to have tried it in some way, but the thing I think that catches most people by surprise is the fluency of it because we're used to natural language processing types of systems in the past where it was very stilted, awkward to use. Many, when they get their first taste of using a modern day generative AI, whether it's ChatGPT or any of the others, finds themselves raising their eyebrows a bit surprised and pleasantly finding that they can almost carry on a fluent like conversation.
Dazza Greenwood (06:50):
Yeah, that's so true. If I could pick up on that a little bit.
Natalie Pierce (06:53):
Yeah.
Dazza Greenwood (06:54):
So Lance, you had just said a moment ago that part of what caught your eye when ChatGPT came out was that they had more or less successfully grappled with the alignment and guardrails problems to make sure that the toxic outputs didn't come out just because they were maybe statistically more likely from a training set, but you could take the transcript to polite societies as you said it. I think that's big, but there's two other functional things that we'd be remiss if we didn't bring up that are different. One of them gets to why it feels more fluent. So, in addition to the reinforcement learning with human feedback sanitize the content, the format's different too.
(07:39):
So, before this ChatGPT modality, it was a text completion. So, you can start writing words and it would auto complete an entire sentence or a paragraph or more. That's okay, but it's limited usefulness. What they did with the packaging around a chat interface could be more turn-based, which is fundamentally how we communicate as humans. So, we do a prompt of some kind, a question or an instruction, and then it provides its output. Then that brings me to the other thing, which is in addition to reinforcement learning, there's also InstructGPT, which is one of the other many innovations.
(08:18):
So, this instruction set let it basically understand what kind of form and answer it should take. So, is the form of a poem or is it the form of an answer to a factual question? Is it an interrogatory? Is it a memo? Is it an email? You could instruct it or explicitly or it could infer what the form is and then it could provide it in that shape. These are among the profound advances that have all come together to create this almost human-like experience of communication.
Dr. Lance Eliot (08:50):
Yeah, I absolutely agree with that. To your comments, the idea of taking turns, what I've discovered is newbies who've never used generative AI before who are perhaps familiar with let's say an Alexa or a Siri or something like that, they're used to the idea that you give very curt commands or direct instructions and that's about as far as you go. I think most people would say, "Yeah, you don't really have a conversation with those kinds of systems."
(09:16):
With generative AI, as you say, an important part for anyone who's savvy using it is that you have to think of this not simply as, "Oh, I enter something as a prompt and I get a result and then I walk away." It's more of I enter something, it says something back, I enter something else in reaction to that. It reacts to the me and you effectively carry on a conversation. As you say, taking turns doing this, which is again, quite a surprise for many people that have that genuine interactivity with an AI system.
Natalie Pierce (09:48):
Both of you have been so generous with your knowledge and expertise in terms of putting content out there and prompt engineering and legal prompt engineering is a thing. This truly is a technology for us, lawyers. Dr. Eliot, I know you're not a lawyer, but you get it. You've written so extensively about the application in our legal industry. So, I want to just dive right into gen AI killer use cases. If we're going to be discussing gen AI killer use cases, and I think we do need to set the foundation by defining a framework, if you will. Dazza, let's start with you. Can you provide some brief steps on a framework that you should follow to be successful with gen AI? You also mentioned being responsible as well.
Dazza Greenwood (10:44):
Yeah, certainly, Natalie. I think one essential step towards harnessing the power of generative AI responsibly in a legal context like for law practice and for legal processes is exactly that. It's creating a robust framework. I think the key steps, the way I look at it at a very high level are number one, to be able to formulate well-constructed prompts that provide the necessary context and other parameters in order to get the results that you're looking for. That opens this topic of prompt engineering that you just foreshadowed a little bit. I think the second one, the second I guess leg of the framework would be to get really good at reviewing and critiquing the outputs, using your judgment, your expertise, your skill. That's essential.
(11:41):
Then finally, to be able to iterate on that, to learn and to adapt, because the prompts that you used you may discover are creating outputs that are not optimal in various ways. You can go back. In fact, you should go back according to this third leg of iteration and try changing the prompts in different ways and see how that changes the output. The output is very sensitive to every little word in the prompt, and you'll see that. Then finally, establishing human oversight and controls for the overall process, so that you have a person that's accountable, that can stand behind the outputs of these models when those outputs are an input to law practice. So, I'd say those are at a very high level, the basic elements of a framework to get started with this.
Natalie Pierce (12:36):
Excellent, excellent. Dr. Eliot, you wrote about 50 crucial golden nuggets you need to know about AI and the law. From that list, which I would encourage all of our listeners to read, if you haven't already, from that list, what has surfaced recently would you say is the most critical things? I know that's a tough question, because to get to the 50, you really had to do a lot of whittling down of what was a list of... I think you said well over 100 golden nuggets.
Dr. Lance Eliot (13:09):
Well, first, yes, thank you for pointing out that indeed, when people first hear that there were 50 items, they say, "Well, you must be stretching to get to 50," and then the answer is no. It's more of I had to choose judiciously to get it down to 50. But to your question, I'll cover two that I think would be very valuable for the listeners to be cognizant of. The first one is for law firms and lawyers, one of the things in working with a lot of law firms and with lawyers individually, that they should be aware of the synergy between applying AI to the law.
(13:44):
We're using AI to assist you in some legal-oriented task, plus what I call the application of a law to AI in the sense of there's a lot going on about AI and generative AI related to, for example, intellectual property rights is a big issue these days, but there are many other issues as well that for example, generative AI that a client of a law firm, a lawyer might be using, could embed various biases that affect how they're using that AI in the act of producing their products or their services. They may have legal exposures that they're not even aware of.
(14:25):
So, what's happening right now is we're having both law firms and lawyers looking at and adopting AI for the performance of legal tasks, and in addition, augmenting, enhancing, or maybe even going into the area of legal services associated with clients of theirs or prospective clients that are using AI. Those two have a synergy between them, because generally, what happens is a law firm that adopts AI learns more about the nature of what generative AI is like and how it works, which further informs them when they're trying to advise their clients or prospective clients about that. So, that's the first item. Then the second one is that AI is coming into law practices whether lawyers or law firms want it or not.
(15:17):
There's some that I work with that say, "Well, I'll wait. It's a fad. It's going to just fade away. It's one of those things come and gone and I'll just ignore it until it's disappeared." But my view is, and there are many others that hold the same view, that no, we're seeing emerging disruption in the legal field as a result of generative AI. This is not going away. It's simply going to get more and more significant over time. The AI is going to get more advanced. Also, there's a line that everyone knows who's going to the inside in AI and law, and that is those that are equipped with and know how to ably use AI are going to be outdoing those lawyers and law firms that do not. So, those are the two items that I thought might be especially worthy out of the 50 to bring attention to the listeners.
Natalie Pierce (16:05):
Thank you. Could not possibly agree more. I think law firm leaders, there's no question that lawyers and everyone else who can on the planet will be using this incredibly powerful tool. Leaders at the end of the day should be striving to foster a culture of responsible AI usage by establishing an environment that really does balance lawyers' exploration and use of the technology while still adhering to all the ethical, legal, and professional obligations. So, I think that's a really, really important point.
(16:46):
So, listen, with that foundation, I'd like to focus the rest of the time on what is the headline for this episode and that's your research, findings on killer use cases. Starting with prompts and prompt engineering, you break down how you focused on prompts, because you've both been doing this quite a bit in an extremely helpful way, why you focused on prompts and why they're important. Dr. Eliot, let's start with you this time.
Dr. Lance Eliot (17:22):
Where I usually begin that conversation about prompts and prompt engineering, there's an old line in the computer field, and I think everyone pretty much knows it that's ever used computers. That's the garbage in, garbage out. When that's applied to the generative AI, when you enter a prompt, some believe it's simply something you do ad hoc, off the top of your head. The problem with that is that typically, the kind of output that you're going to get or the kind of conversation you're going to have with generative AI is bound to not be as productive, as useful for you, otherwise if you thought beforehand and tried to be a bit more systematic. So, prompt engineering is all about the idea.
(18:03):
Maybe there are good ways to craft prompts and carry on conversations that will be beneficial for the person who's interacting with the generative AI in the sense of that they give the generative AI a better sense of where to head and what kind of responses to produce, whether it's essays or narratives or just answering questions or solving problems. Just a quick example of that is one thing that research has shown is that in a prompt, if you say something simple such as think about this step-by-step or respond in that way or act in that way, what tends to happen is the generative AI usually tends to produce a better result in a way. It's not because it's sentient by the way. It's because the computational pattern matching goes into a mode where basically it can be more thorough.
(18:59):
So, a simple couple of words of how to word something as a prompt, prompt engineering gives various guidance and techniques that one, for example, is known as the chain of thought technique. So, if you know these techniques and they don't take that long to learn, though maybe to master just take a little while longer, you can really do a lot more with generative AI than otherwise just poking around and hoping that what you happen to enter will get you some good results.
Natalie Pierce (19:29):
Well, I have to say another one of your pieces in Forbes last month where you had links to so many different lessons for boosting your prompting skills, 20 and all. I dug in and I started educating myself, but the skills like determining when best to use the show me versus tell me prompting strategy, vagueness as a useful prompt engineering tool, and then the flipped interaction I thought was really explaining how it's a crucial prompt engineering technique that everyone should know. That's just to name a few. So, thank you for putting that out there with all of those links. Dazza, you had a guest on your podcast, I love this, who called gen AI sent our lawyering better, faster, and stronger.
(20:29):
To the point that Dr. Eliot just made, it said, "It's really important when you are using prompt engineering as a lawyer to try and be as descriptive as possible, articulate the desired output, speak to the language model clearly as you would, for example, to a first year associate in that way, in a sense, trying to teach a language model to react." I'm very aware, Dr. Eliot, that you don't like to associate sentient beings to the model, but in any event, really training to have a reaction like a lawyer. So, Dazza, what's your reaction?
Dazza Greenwood (21:13):
When it comes to prompting large language models like ChatGPT or Anthropic's Claude 2, Google's Bing, lawyers are in a great starting position, because all of our training and our skillset is focused on words and articulating ideas, on being able to take apart and evaluate and assess and put back together thoughts for different purposes. This fluency is perfectly well-matched for the current generation of generative AI. It's word based. It's language based. So, we're in a great starting position. It's like a hand in glove fit. The main guidance, I'd say, especially when getting started, is to keep the prompts simple and clear and to remember that you don't have to ask a question.
(22:13):
You can give an instruction of different types like instruct it to review a memo or to create questions that you can ask a client at intake or to help you with an email. Instruction is critical and you can iterate. Not only can you go deeper, as Lance was starting to mention earlier in a conversational set of inputs and outputs, you can also just restart the session and try again and try again. So, those three guiding lights. I think at a high level, prompt engineering is just a vital aspect of harnessing the power of generative AI effectively. What I've been doing at MIT is we're calling legal prompt engineering. It's a subset of prompt engineering that's particularly focused on common legal tasks and legal concepts and legal processes and the legal context.
(23:11):
So, you can see more of that tutorials, including the one that you just referenced earlier at law.mi.edu/ai and all of that is free and open source. I guess I'd be remiss if I didn't hit on some of the practical things. Summarization I think is maybe one of the killer apps in the law for this technology. Part of this is because when you give these models the content you wanted to summarize as part of the prompt, it puts all of its attention on that text and it is much less likely, virtually never starts to stray or hallucinate or confabulate or get off point. It just puts its attention on the text and does a good job summarizing it. That's a very common thing that we need help with in the law.
(24:10):
You can have it critique things. I've been doing that a lot with scientific papers lately and with PhD students, some hypotheses. It does a pretty good job doing a critique or finding holes or problems or things like that. You can have an extract information like extract all of the named parties in this affidavit or what have you or in this set of email or Slack or whatever it's trying to do and see the relationships. You can do that reasoning. It's great at reformatting things, translating to different languages as a start, not perfect, but you can transform completely from a bunch of emails into a memo or that thing. You can make completely new things. So, as you get better at instructing and having an idea of goal, if you can articulate it, and as I said, we can articulate things in the law.
(25:02):
If you can envision and imagine and articulate what you want it to do and then write that in a simple clear way in the prompt, it's very good at generating a first draft of things that have never existed before. So, I guess I'll just end by saying before we got on this podcast today, I've been working on an automated prompt engineering little script that makes it easier to basically put the same prompt into a model a bunch of times and measure the differences, make sure it'll be reliable and look at a distribution of that. Then the other thing it does is it can change the prompt in subtle ways and then run it a bunch of times to see what the full impacts of that are. I'm going to make that available at dazzagreenwood.com on a blog probably by the time people hear this podcast.
Natalie Pierce (25:53):
Oh, that's fantastic.
Dazza Greenwood (25:54):
I encourage people to try it manually, but also think about trying it in a more programmatic automated way. So, you can get a good sense of what you can rely on it for and what you can't rely on it for.
Natalie Pierce (26:05):
Fantastic. At Gunderson, we've been brainstorming the best ways to leverage ChatGD, which is our homegrown generative AI chat app. We represent thousands of startups. So, as the head of Gunderson's employment law practice, we're really looking with our innovation team at creating our own employment collection within ChatGD. We've started collecting our greatest examples and samples with the ultimate goal really being to be able to ask questions back and forth, to quickly get tailored employment information and generate templates that we can use as a starting base for our clients. Just because we have retrieval augmented generation and because of that, we will do things like feed a document that is a leave policy.
(27:02):
Now, again, these are startups, so they're not going to have an employment lawyer on their team. They're not even necessarily going to have a head of human resources or people or talent. So, saying, "Okay, take this policy, it's a new policy, and come up with talking points for managers, come up with questions and answers that employees might have," yeah, it's incredibly effective for so many things that we lawyers do. You mentioned a lot of them, and even just converting legalese into plain language as for example. So, the applications are truly endless. I would like to stick with you, Dazza, if we could. Can you give any examples, the applications for prompt that you would describe as killer use cases?
Dazza Greenwood (27:55):
Absolutely. Yeah. The first one right out the box, I mentioned a moment ago, but it's summarization and there's more and more research coming out that shows just how powerful these modern large language models are at it. There's something I just posted about on LinkedIn a few days ago. It was a small scale experiment, so you can't put too much weight on it, but still it's relevant and it's some evidence. It showed that the summarizations were overall better than PhD students. Then there was some evidence that the summarization actually hallucinated by a technical definition of that word. The LL one hallucinated less than the students did as they started bringing in analogies and real world knowledge.
(28:38):
We're all fallible. We get things wrong a little bit, but when you're looking at the first draft, it compared very favorably. Of course, it does the draft in like three seconds. So, wow. Summarization is something I do probably 50, 60 times a day to get started on things. I now have a lot more understanding of a lot more things that people send me than I ever had the opportunity to even scan, much less read all the way through in the past. But some other high value applications include more structured document reviews and there's aspects of legal research and analysis that are good. I was just working with a team... I don't want to bring back traumatic memories, but remember in law school when we did IRAC, issue, rule, application, et cetera.
(29:28):
So, there's ways you can provide a template not just for to think into, but for it to write into by formatting into the template. So, it tells it the right things to look at, to extract, to reason about, and then to generate into the right template, which is I think also similar to what you were just describing for some of the workflows at your firm. It did a great job on cases when we made cases up and it did a perfectly good job, so it never would've seen it in the training set. I think also it's pretty good, surprisingly good in my view at risk detection. So, looking at something and trying to identify how could this go wrong.
(30:09):
Hallucinating is bad when you want a factual case to cite that may or may not know about, but when you're trying to imagine hypothetical futures, things could happen, it is a powerful augmentation to be able to imagine and suppose such things. Oh, God, I can just go on and off. I think the main thing is it's still relatively new and I think the task at hand now for people listening to this podcast is to try things and evaluate critically. So, I think we're still only scratching the surface on how this could work. I'm on the American Bar Association's task force for AI. We had a meeting earlier today.
Natalie Pierce (30:48):
Fantastic.
Dr. Lance Eliot (30:49):
One of the things that really was clear is the conversation generally up to now has been frameworks for ethics and for risk mitigation and for safety, all of which is appropriate. I'm sure we'll talk about that in due course, but the theme that really emerged and which I'm absolutely an advocate of myself is let's look in the law, what this is good at and how we embrace it and how we can apply it and adapt and evolve the practice of law to supercharge our role and our capabilities and the value that we provide to clients and to the economy and society at large. I think this is a huge unlock for lawyers and the law.
Natalie Pierce (31:34):
If you are a lawyer out there and you have not started and there are so many, again, excellent examples. Dazza, you said something that reminded me of how you can be an expert, how you can be the PhD candidate for example and want to find ways to, yes, summarize some works, but also communicate, because sometimes when you're an expert, you just get so lost in the weeds, because you know so much. That brings me back, Dr. Eliot, to what you had really recommended in one of your articles about that flipped interaction is a really critical prompt engineering technique, because you as the expert can ask the generative AI to ask you questions. You gave a very funny example of the expert in underwater basket weaving.
(32:35):
I thought that was priceless and really the way that it worked. I want you to ask me questions about underwater basket weaving, so that I can share with you the key principles involved. Then after asking these questions and you limit the number of questions, it gives you this recap at the end, but I thought that the questions were asked and finding a way to really communicate that expertise, I thought that was just a fantastic example. But Dr. Eliot, I'll let you add any other examples you want to give for applications for prompt that you would describe as killer use cases.
Dr. Lance Eliot (33:14):
Absolutely. Everything I've heard so far echoes what I would definitely say as well. I oftentimes try to shape the use of generative AI into the legal realm. I'm doing a workshop, in fact, yesterday, today and tomorrow, where I'm walking lawyers through different aspects of killer use cases for AI and the law. Also, I have an article in the California Lawyers Association periodical that covers this. Just real quick, what I typically do is I try to do five key killer use cases that will sound somewhat similar, I believe, to you and Dazza have just described. I'll just quickly rattle them off.
(33:57):
The first one is carry on a legal dialogue involving legal arguments to see what you might be up against in court or against a legal adversary by having that flipped interaction, where you have the generative AI try to challenge you and see how you respond to that. Another aspect is we use generative AI to examine voluminous materials. I know Dazza mentioned that.
(34:23):
For example, during e-discovery, when you're trying to ferret out what's significant to a case and what makes this different than what you would conventionally do, you conventionally might look for a particular keyword, but then the problem with keywords typically is that you can get back all sorts of junk that really doesn't contextually make sense for what you're looking with generative AI because of the computational pattern matching involved that it's able to invoke. It's able to have a better semblance of context, bring back hits or what you're trying to find in voluminous material in a way that keywords can't. Then the other three are similar to what you said.
(35:00):
So, using generative AI to compose a first pass of a legal document and then doing a human lawyer review to see how it comes out. That's very important because one aspect mentioned so far here is the idea of hallucinations and what that refers to. It's a term I don't like because it implies a semblance of sentience. We think of humans having hallucinations, but it's taken hold and so we have to just roll with it. But what it refers to is the generative AI effectively making stuff up. The problem with that is that it makes stuff up, but it often has an air of confidence to it in the sense that it might make up an entire legal case that doesn't really exist, but give so many details about it that you would swear when you look at it, well, it must be true, but often times, it isn't.
(35:52):
You have to be very careful of that. The other two, use generative AI to review a lawyer's composed legal document and see what generative AI says about the weaknesses, strengths, issues, ambiguities in something that a human lawyer composed. Then the last one, convert a legal document to plain English or possibly sharing with your clients, but once again, only after an additional human lawyer review to make sure that it's reflective of what the legal document is.
(36:21):
So, those are the five ways that in my workshops and that I've written about that I find are very handy for lawyers and law firms that are trying to get into this. That or anything in that realm is a good way to get started, to get a feel for what this is like, what prompting is like, what prompting engineering is about, the kinds of techniques that you can use, and how far you can potentially take this in a legal realm.
Natalie Pierce (36:46):
Perfect, perfect. Dazza, there's a Sam Altman quote that I just so love. I'm not going to be able to get my fingers on it, but what Dr. Eliot just said definitely reminded me when he was talking about the hallucinations. Just that it's really good, but it can again, with that confidence, give as semblance of correctness. I don't have the quote in front of me, maybe you know which one I'm talking about.
Dazza Greenwood (37:14):
Yeah, well, there's so many to choose from.
Natalie Pierce (37:16):
True, true.
Dazza Greenwood (37:18):
It's well-known behavior and it's generally considered a flaw if what you're trying to do is something that's fact based. I agree with Lance, anthropomorphizing these things and attributing human cognition and sanctions to them is not helpful. It means people will reason about them wrong. You'll come to the wrong conclusions about what they're doing and what they're likely to output. So, we shouldn't do that. For hallucination, at least in my clients and with my research team, I just use the term fabricate at this point. Hallucination actually, it has like a DSM pathological definition that we don't want to use and we don't want to impugn humans by diminishing those conditions either. But the main thing is we're going to reason incorrectly by misunderstanding fundamentally what the capability is.
(38:12):
You can edit this out if you want to, but I feel like we should mention what the technology is, because Lance just asserted quite rightly that this does not operate like Boolean search, what we do in Westlaw or Lexus or the type of pattern that we would even put in a search engine or even like a SQL query to a database. What's happening here is it's not based on find the word or the sequence of words, rather this is essential to what the technology is. It takes all of this knowledge, all of all these documents and other writings as part of the training set and part of the training method. It atomizes them down into these tiny little constituent elements and expresses them as math basically on something called vectors in vector space.
(39:06):
So, there are all these little parameters, these little points on these vectors and sets of vectors and matrices, billions of them in the larger models. So, you can end up seeing the deep relationship of the word if, to, and, and the verb to the noun in this sentence to that sentence. So, it's almost like seeing into the shape of the concepts that are happening. So, when you want to ask it about something or to do something, you don't need to use the words, do the exact words, rather it looks at the idea and the concept. It's great at analogies and metaphors or translating knowledge from one domain to another, or as you said, just even doing plain language, you have to get the concepts and restate them in a different way. This is completely different from how computation used to work, and it is very, very powerful.
(39:58):
It's really good for the law when you want to look at a cause of action or a situation and look across all the bodies of law to see where that comes up. You have to get lucky by putting the right words in. In the old days or hope that somebody caught a headnote correctly, not anymore. Now, we can finally pare deeply into the essence and the nature and the shape of the law. We can see the law for the first time, and that means we can use it. We can understand it. We can apply it. So, the high dimensional vector space is good for us, and that's fundamentally what's happening here.
Natalie Pierce (40:35):
Thank you for that, Dazza. We're definitely going to weave that into the episode. That gen AI workshop that you did and I don't remember the name of your guest, but he just worked through, showing exactly how this can work. In that case, I believe it was writing and helping write an opposition to a motion for summary judgment.
Dazza Greenwood (40:58):
Exactly. Yeah, Damien Real, who's another great fellow traveler, prompt engineer at the forefront. He was also a very experienced litigator, which is even more well testimony as it were to how important it's to bring our expertise and to bring our knowledge and our skill and our wisdom to the task of articulating what we're trying to do. The more articulate we are because of the way these systems work, their inference engines are really good and they do a pretty good job of getting what we're trying to say and then putting it in the shape that we want and hitting a lot of the right points conceptually.
(41:41):
Again, it's not like a fact. It's like you have to feed it facts that you want it to stick to. That's what we did in that show, which is it was an Idea Flow episode at law.mit.edu/media if you want to find it. Basically, he took motion to dismiss, I believe, or for summary judgment in one of the IP cases on I think involving OpenAI. He basically took the table of contents of the motion and then he saw what an answer would be, basically a rebuttal to each one. He did that and then he unfolded the rebuttal into what would be the elements that we'd have to prove for each thing. Then he went down and down and down. You saw it yourself.
Natalie Pierce (42:24):
Yeah, I was pretty blown away.
Dr. Lance Eliot (42:27):
It's basically like magic. I mean, I do basically understand what's going on, but every time I see this, I am just floored by how powerful and how on point this technology is for law.
Natalie Pierce (42:40):
Absolutely. All right. Well, Dr. Eliot, before we head into our quick fire challenge, which you've both agreed to, I'm going to leave our listeners with a final thought or tip.
Dr. Lance Eliot (42:56):
I guess one of the final tips I might provide is in the news these days, there's a lot of banner headlines of an AI doomsday heading towards us. The idea behind it is that as sophisticated as this computational pattern matching underlying the AI seems to be, are we on the cusp of what some would call artificial general intelligence, AGI? Meaning a type of AI that essentially would be indistinguishable from how humans think, which of course, we still to this day don't really know exactly how humans think, but in any case, the AI would be so good that you could not differentiate it from that, which you might do when you interact with a human. There's a famous test called the Turing Test named after Alan Turing. It relates to that.
(43:49):
What I believe, and there's a lot of controversy over this, is that we are not on the verge of that AGI doomsday. I find it overshadows unfortunately and distracts from the reality of what AI and generative AI can do today and will be doing in the next several years. So, I don't want to downplay how these advances are going to really impact society will be tremendously so, including in areas such as autonomous weapons systems, the idea of using AI that can control weapons and not have any human in the loop. That's a very serious aspect. What I find happens is people become, I think, a little too preoccupied with the existential risk idea that AGI is coming and it's going to wipe us all out.
(44:38):
I tell people, "Well, look, you use conventional AI of what we know of today or in the next several years, and you use that in an autonomous weapons system that can create mass destruction." To me, that's something I'd be more worried about than the idea of AGI that suddenly enslaves us or decides with a wave of a computer system to wipe us all out. I guess just the tip is there's a lot of banner headlines about it. It's a very controversial topic. There are many that agree or disagree about how close or how far we are away from this. My view is focus on the reality of what AI can do today and what I believe will be happening in the next 5 to 10 years. Let's certainly talk about the existential risks, but let's not let us be blinded by that and overlook or ignore the realities of what AI can and cannot do today.
Natalie Pierce (45:34):
Hear, hear. There's so many potential applications of AI for good, and we're already seeing countless such applications. So, I will remain an optimist and we are going to head right into our quickfire challenge as we are almost at the hour here. So, we've talked so much about technology in the future. Dazza, we'll start with you. What was your favorite book or movie growing up as a young adult or child and why?
Dazza Greenwood (46:10):
I really liked The Foundation series and I really liked Lord of the Rings so much. It just did world building. It was creative. It sparked curiosity. It showed what courage and fear and relationships and honor and dedication and getting over adversity looked like. It showed a bigger picture and how it played out with individual actions. I just love them. Now, of course, they're all major Hollywood productions, which is even better. All the stuff from my childhood is big screen now, so I'm loving that.
Natalie Pierce (46:47):
Oh, I love it. How about you, Dr. Eliot?
Dr. Lance Eliot (46:50):
So between books and movies, I'd like to do books and places or place if I could. The book would be when I was in college as a computer science major, I read the Handbook of Artificial Intelligence by Feigenbaum and Barr. This now is a dated book unfortunately, but at the time, it had what was considered the state-of-the-art comprehensively about the field of AI. It's what then launched me into basically the rest of my career doing consulting work, being a Chief Information Officer, Chief Technology Officer using AI, a scholar in AI, and so on was that book back in that day. I think it's still worth reading, but of course, it's dated now.
(47:28):
So, you'd have to certainly augment it with more contemporary aspects, but it still had the fundamentals in it. The other would be, if I could just quickly mention, is in terms of place, so I was born and raised in Southern California. I'm very lucky in the sense that I lived near Disneyland and I went there all the time. I went there on weekends, evenings, summers, just all the time. The reason I bring that up, my focus at Disneyland was Tomorrowland. At Tomorrowland, it was a showcase of the future-
Natalie Pierce (47:54):
My favorite.
Dr. Lance Eliot (47:55):
... especially technology that we might one day have. Yes. So, as a child going there all the time, I would just be amazed at the technology that I saw, which of course today, we'd laugh at or something like that to think how crude it was at the time. But it inspired me to go down the computer and technology route with the dream in my mind, perhaps coming from Disney and himself in a sense that maybe I could do something about the future in terms of being able to develop this technology. I'll just add one other thing, one point. I actually worked at the Walt Disney Imagineering, WDI, that conceives and designs the theme parks. So, it turned full circle around for me.
Natalie Pierce (48:39):
Wow. Wow. Okay. Then one more question before I let you both go. Dazza, we are talking about being far from AGI. Hopefully, that is true. So, let's talk about humans. What is your favorite or least favorite human trait?
Dazza Greenwood (48:58):
Yes, there may be a bit of a theme with my answer to the last question about why those books.
Natalie Pierce (49:02):
I was thinking there might be.
Dazza Greenwood (49:06):
There's a lot of human traits I really like. I guess I go with curiosity and playful creativity. That never gets old and applying that with friends and relationships, but even at work, there's a place for that. When you're especially doing innovative things, that attitude can be very helpful to look beyond the horizon and to have the courage to try things. I guess I'm thinking about not just an individual person like in a capsule, isolated, being curious and playful, but with other people. When people in groups have the group dynamics of curiosity and playfulness, I love that. That's partly why I'm associated with MIT, because the Media Lab has got a culture of the intrepid hacker spirit where we get together and prototype things and do stuff like that.
(50:01):
I live in Oakland now and there's a lot of interesting cultural communities here where we build things, do stuff. So, I'd say that would be my favorite characteristic. I know we're getting away from the theme of the show, but I have to mention it is very well tuned for how to look at and address and have the courage to look in the face of the big changes that are happening now and to face them and to adapt to them and to make the best of it, so we can have a better tomorrow.
Natalie Pierce (50:32):
Great answer. Great answer. Dr. Eliot, how about you, favorite or least favorite human trait?
Dr. Lance Eliot (50:40):
Well, I guess, Dazza and I, we know each other. We're great colleagues and I guess we think alike in many ways. His comment about the idea of curiosity and so on, for me, the human trait that I really favor is the quest to figure out the unknown. That's why I went into AI. I wanted to try to figure out how is it that humans think and how is it that we can make technology that appears to do what humans do in terms of thinking, whether it actually replicates that or in some way simulates that.
(51:11):
That's an open question, but just the ability to get technology to behave in that same fashion is really what has drove me into the area and continues to keep me in that area today. To a comment that you made just a few moments ago, I do that also because I also strongly believe that if we can attain that kind of AI, that it will help us to do things like cure cancer or possibly deal with climate change or aid us in resolving a lot of major issues facing humankind that I believe AI as a beneficial tool can really help us with.
Natalie Pierce (51:49):
Wonderful. Well, I say let's all stay curious and creative and lifelong learners. I think we all have an optimistic view of what's possible and what the future holds. Unfortunately, we're out of time. So, I just want to thank you both so much for joining us today for our season finale and what was a great conversation on killer use cases for gen AI. I truly thank you both so much.
Dr. Lance Eliot (52:24):
Thank you so much.
Dazza Greenwood (52:26):
Thank you so much. Great speaking with you. It's always great to be in conversation with Lance again, so thanks so much.
Natalie Pierce (52:32):
Wonderful, wonderful. Thank you listeners for joining us. The successful future of the legal industry is intertwined with the ability of lawyers to responsibly embrace new technologies like generative AI. Bye-bye for now.
Voiceover (52:50):
You've just listened to the Future Work Playbook. This podcast series is brought to you by Gunderson Dettmer, the world's number one law firm representing venture capital funds and high growth companies. Join Natalie Pierce on our next episode as she and her guests help prepare your organization for the future. Please subscribe to the Future Work Playbook.
Welcome to the Future Work Playbook. This podcast series features founders, investors, and legal talent that will help you embrace technology and transform your organization for a better future. This series is hosted by Natalie Pierce, the chair of Gunderson Dettmer's Labor and Employment Law practice. Natalie and her guests are committed to helping you develop new playbooks to elevate your game.
Natalie Pierce (00:29):
Hi, this is Natalie. Welcome to our final episode of season five, featuring experts and advisors in the field of generative AI. If you haven't already listened to our previous episodes, they cover how to get started with generative AI, privacy and security strategy, and shaping the law firm of the future. This episode will briefly touch on these topics, but we'll quickly dive into killer use cases for gen AI.
(01:04):
Today, we welcome two visionaries in the field of generative AI, Dr. Lance Eliot, a regular Forbes contributor and AI fellow at Stanford University, plus Dazza Greenwood, a renowned Fortune 100 legal tech advisor and the executive director of law.mit.edu. Both gentlemen have been leading the industry with their research and publications. They routinely counsel attorneys, CEOs, educators, and business leaders. They're both on the cutting edge of gen AI trends, developing roadmaps for legal teams, engineers, and executives to transform the future. Dazza and Dr. Eliot, thank you so much for joining us.
Dazza Greenwood (01:57):
Oh, thank you. It is such a pleasure to be on the show. I'm a longtime listener, first time caller.
Natalie Pierce (02:04):
I love that.
Dr. Lance Eliot (02:05):
This is Lance Eliot, and I also would like to say thank you so much for having me on the show. I'm very excited about the coverage that you'll be doing.
Natalie Pierce (02:13):
Well, it is such a true privilege to have you both joining us, so looking forward to this discussion. Okay, just to kick us off, wow, what a year it's been. Most of us never imagined that we'd have such a powerful tool at our disposal. Would you each mind sharing with our listeners initial reactions when ChatGPT first hit the scene?
Dazza Greenwood (02:43):
This is Dazza Greenwood and my initial reaction when ChatGPT first hit the scene was one of astonishment and delight because I wasn't expecting it. I was familiar with GPT 2 and GPT 3 and comparable models and had been actually in the midst of some projects pushing them for all they were worth for legal contexts. ChatGPT just appeared one day. Somebody emailed me about it. I started using it.
(03:13):
Well, I'm sure you're familiar with it, but it really works very astoundingly well. It was just cutting through legal use cases and challenges one after the other when I started. I honestly still haven't gotten over it. It's almost a year now, and I am still just delighted and astonished by what it can do. I know we'll get into the limits and the risks, but my first reaction, which continues to be my prominent reaction, is wow, this stuff is really good. It's a serious advance over what was available, broadly available, just really for all time prior to this.
Natalie Pierce (03:51):
Yeah, a lot of us having those OMG moments. I think I recall you saying, Dazza, at one point you said, "It's a tool for us," like you said, to a room full of lawyers and law students. It's a tool for all of us. How about you, Dr. Eliot?
Dr. Lance Eliot (04:11):
Yeah, so echo the same comments that were just made and would add a little bit of an additional item to it, which is that for many of the AI insiders, we knew that what OpenAI, the maker of ChatGPT, was doing is they were trying to overcome a prior big problem for generative AI systems that were released, which was that oftentimes, right away, they would get into trouble because they would emit offensive outputs or people would try to spur them to do that. The question was, would ChatGPT get caught up in that same aspect that right away it might become something that people would complain about because it was saying things that we would not expect in polite society to be said?
(04:59):
It was known that they were using a technique called RLHF, reinforcement learning from human feedback, trying to try to hone ChatGPT, so that it wouldn't get quite as caught up in that. When it came out then, what did surprise me was how effective that had been to make it that this was something that the public at large could use and not get all sorts of just wild, zany, and insulting comments coming out of it. Then of course, it took off after that.
Natalie Pierce (05:29):
Yeah, it took off in a very, very big way. Dr. Eliot, you wrote, I believe it was last month in force, something that I very much agreed with you when you said it. The reality is that we're on a conveyor belt that is going to keep pushing us along on advancing AI and leveraging AI into all quarters of society. I think the reason why we are seeing that now, why we'll continue to see that is because yeah, it's surprising how good it is, how good these new iterations. We have new language models and iterations coming out all the time, but it is truly astounding.
Dr. Lance Eliot (06:12):
Absolutely. For any listeners that somehow haven't played with generative AI, which you'd almost have to, I think, live in a cave not to have tried it in some way, but the thing I think that catches most people by surprise is the fluency of it because we're used to natural language processing types of systems in the past where it was very stilted, awkward to use. Many, when they get their first taste of using a modern day generative AI, whether it's ChatGPT or any of the others, finds themselves raising their eyebrows a bit surprised and pleasantly finding that they can almost carry on a fluent like conversation.
Dazza Greenwood (06:50):
Yeah, that's so true. If I could pick up on that a little bit.
Natalie Pierce (06:53):
Yeah.
Dazza Greenwood (06:54):
So Lance, you had just said a moment ago that part of what caught your eye when ChatGPT came out was that they had more or less successfully grappled with the alignment and guardrails problems to make sure that the toxic outputs didn't come out just because they were maybe statistically more likely from a training set, but you could take the transcript to polite societies as you said it. I think that's big, but there's two other functional things that we'd be remiss if we didn't bring up that are different. One of them gets to why it feels more fluent. So, in addition to the reinforcement learning with human feedback sanitize the content, the format's different too.
(07:39):
So, before this ChatGPT modality, it was a text completion. So, you can start writing words and it would auto complete an entire sentence or a paragraph or more. That's okay, but it's limited usefulness. What they did with the packaging around a chat interface could be more turn-based, which is fundamentally how we communicate as humans. So, we do a prompt of some kind, a question or an instruction, and then it provides its output. Then that brings me to the other thing, which is in addition to reinforcement learning, there's also InstructGPT, which is one of the other many innovations.
(08:18):
So, this instruction set let it basically understand what kind of form and answer it should take. So, is the form of a poem or is it the form of an answer to a factual question? Is it an interrogatory? Is it a memo? Is it an email? You could instruct it or explicitly or it could infer what the form is and then it could provide it in that shape. These are among the profound advances that have all come together to create this almost human-like experience of communication.
Dr. Lance Eliot (08:50):
Yeah, I absolutely agree with that. To your comments, the idea of taking turns, what I've discovered is newbies who've never used generative AI before who are perhaps familiar with let's say an Alexa or a Siri or something like that, they're used to the idea that you give very curt commands or direct instructions and that's about as far as you go. I think most people would say, "Yeah, you don't really have a conversation with those kinds of systems."
(09:16):
With generative AI, as you say, an important part for anyone who's savvy using it is that you have to think of this not simply as, "Oh, I enter something as a prompt and I get a result and then I walk away." It's more of I enter something, it says something back, I enter something else in reaction to that. It reacts to the me and you effectively carry on a conversation. As you say, taking turns doing this, which is again, quite a surprise for many people that have that genuine interactivity with an AI system.
Natalie Pierce (09:48):
Both of you have been so generous with your knowledge and expertise in terms of putting content out there and prompt engineering and legal prompt engineering is a thing. This truly is a technology for us, lawyers. Dr. Eliot, I know you're not a lawyer, but you get it. You've written so extensively about the application in our legal industry. So, I want to just dive right into gen AI killer use cases. If we're going to be discussing gen AI killer use cases, and I think we do need to set the foundation by defining a framework, if you will. Dazza, let's start with you. Can you provide some brief steps on a framework that you should follow to be successful with gen AI? You also mentioned being responsible as well.
Dazza Greenwood (10:44):
Yeah, certainly, Natalie. I think one essential step towards harnessing the power of generative AI responsibly in a legal context like for law practice and for legal processes is exactly that. It's creating a robust framework. I think the key steps, the way I look at it at a very high level are number one, to be able to formulate well-constructed prompts that provide the necessary context and other parameters in order to get the results that you're looking for. That opens this topic of prompt engineering that you just foreshadowed a little bit. I think the second one, the second I guess leg of the framework would be to get really good at reviewing and critiquing the outputs, using your judgment, your expertise, your skill. That's essential.
(11:41):
Then finally, to be able to iterate on that, to learn and to adapt, because the prompts that you used you may discover are creating outputs that are not optimal in various ways. You can go back. In fact, you should go back according to this third leg of iteration and try changing the prompts in different ways and see how that changes the output. The output is very sensitive to every little word in the prompt, and you'll see that. Then finally, establishing human oversight and controls for the overall process, so that you have a person that's accountable, that can stand behind the outputs of these models when those outputs are an input to law practice. So, I'd say those are at a very high level, the basic elements of a framework to get started with this.
Natalie Pierce (12:36):
Excellent, excellent. Dr. Eliot, you wrote about 50 crucial golden nuggets you need to know about AI and the law. From that list, which I would encourage all of our listeners to read, if you haven't already, from that list, what has surfaced recently would you say is the most critical things? I know that's a tough question, because to get to the 50, you really had to do a lot of whittling down of what was a list of... I think you said well over 100 golden nuggets.
Dr. Lance Eliot (13:09):
Well, first, yes, thank you for pointing out that indeed, when people first hear that there were 50 items, they say, "Well, you must be stretching to get to 50," and then the answer is no. It's more of I had to choose judiciously to get it down to 50. But to your question, I'll cover two that I think would be very valuable for the listeners to be cognizant of. The first one is for law firms and lawyers, one of the things in working with a lot of law firms and with lawyers individually, that they should be aware of the synergy between applying AI to the law.
(13:44):
We're using AI to assist you in some legal-oriented task, plus what I call the application of a law to AI in the sense of there's a lot going on about AI and generative AI related to, for example, intellectual property rights is a big issue these days, but there are many other issues as well that for example, generative AI that a client of a law firm, a lawyer might be using, could embed various biases that affect how they're using that AI in the act of producing their products or their services. They may have legal exposures that they're not even aware of.
(14:25):
So, what's happening right now is we're having both law firms and lawyers looking at and adopting AI for the performance of legal tasks, and in addition, augmenting, enhancing, or maybe even going into the area of legal services associated with clients of theirs or prospective clients that are using AI. Those two have a synergy between them, because generally, what happens is a law firm that adopts AI learns more about the nature of what generative AI is like and how it works, which further informs them when they're trying to advise their clients or prospective clients about that. So, that's the first item. Then the second one is that AI is coming into law practices whether lawyers or law firms want it or not.
(15:17):
There's some that I work with that say, "Well, I'll wait. It's a fad. It's going to just fade away. It's one of those things come and gone and I'll just ignore it until it's disappeared." But my view is, and there are many others that hold the same view, that no, we're seeing emerging disruption in the legal field as a result of generative AI. This is not going away. It's simply going to get more and more significant over time. The AI is going to get more advanced. Also, there's a line that everyone knows who's going to the inside in AI and law, and that is those that are equipped with and know how to ably use AI are going to be outdoing those lawyers and law firms that do not. So, those are the two items that I thought might be especially worthy out of the 50 to bring attention to the listeners.
Natalie Pierce (16:05):
Thank you. Could not possibly agree more. I think law firm leaders, there's no question that lawyers and everyone else who can on the planet will be using this incredibly powerful tool. Leaders at the end of the day should be striving to foster a culture of responsible AI usage by establishing an environment that really does balance lawyers' exploration and use of the technology while still adhering to all the ethical, legal, and professional obligations. So, I think that's a really, really important point.
(16:46):
So, listen, with that foundation, I'd like to focus the rest of the time on what is the headline for this episode and that's your research, findings on killer use cases. Starting with prompts and prompt engineering, you break down how you focused on prompts, because you've both been doing this quite a bit in an extremely helpful way, why you focused on prompts and why they're important. Dr. Eliot, let's start with you this time.
Dr. Lance Eliot (17:22):
Where I usually begin that conversation about prompts and prompt engineering, there's an old line in the computer field, and I think everyone pretty much knows it that's ever used computers. That's the garbage in, garbage out. When that's applied to the generative AI, when you enter a prompt, some believe it's simply something you do ad hoc, off the top of your head. The problem with that is that typically, the kind of output that you're going to get or the kind of conversation you're going to have with generative AI is bound to not be as productive, as useful for you, otherwise if you thought beforehand and tried to be a bit more systematic. So, prompt engineering is all about the idea.
(18:03):
Maybe there are good ways to craft prompts and carry on conversations that will be beneficial for the person who's interacting with the generative AI in the sense of that they give the generative AI a better sense of where to head and what kind of responses to produce, whether it's essays or narratives or just answering questions or solving problems. Just a quick example of that is one thing that research has shown is that in a prompt, if you say something simple such as think about this step-by-step or respond in that way or act in that way, what tends to happen is the generative AI usually tends to produce a better result in a way. It's not because it's sentient by the way. It's because the computational pattern matching goes into a mode where basically it can be more thorough.
(18:59):
So, a simple couple of words of how to word something as a prompt, prompt engineering gives various guidance and techniques that one, for example, is known as the chain of thought technique. So, if you know these techniques and they don't take that long to learn, though maybe to master just take a little while longer, you can really do a lot more with generative AI than otherwise just poking around and hoping that what you happen to enter will get you some good results.
Natalie Pierce (19:29):
Well, I have to say another one of your pieces in Forbes last month where you had links to so many different lessons for boosting your prompting skills, 20 and all. I dug in and I started educating myself, but the skills like determining when best to use the show me versus tell me prompting strategy, vagueness as a useful prompt engineering tool, and then the flipped interaction I thought was really explaining how it's a crucial prompt engineering technique that everyone should know. That's just to name a few. So, thank you for putting that out there with all of those links. Dazza, you had a guest on your podcast, I love this, who called gen AI sent our lawyering better, faster, and stronger.
(20:29):
To the point that Dr. Eliot just made, it said, "It's really important when you are using prompt engineering as a lawyer to try and be as descriptive as possible, articulate the desired output, speak to the language model clearly as you would, for example, to a first year associate in that way, in a sense, trying to teach a language model to react." I'm very aware, Dr. Eliot, that you don't like to associate sentient beings to the model, but in any event, really training to have a reaction like a lawyer. So, Dazza, what's your reaction?
Dazza Greenwood (21:13):
When it comes to prompting large language models like ChatGPT or Anthropic's Claude 2, Google's Bing, lawyers are in a great starting position, because all of our training and our skillset is focused on words and articulating ideas, on being able to take apart and evaluate and assess and put back together thoughts for different purposes. This fluency is perfectly well-matched for the current generation of generative AI. It's word based. It's language based. So, we're in a great starting position. It's like a hand in glove fit. The main guidance, I'd say, especially when getting started, is to keep the prompts simple and clear and to remember that you don't have to ask a question.
(22:13):
You can give an instruction of different types like instruct it to review a memo or to create questions that you can ask a client at intake or to help you with an email. Instruction is critical and you can iterate. Not only can you go deeper, as Lance was starting to mention earlier in a conversational set of inputs and outputs, you can also just restart the session and try again and try again. So, those three guiding lights. I think at a high level, prompt engineering is just a vital aspect of harnessing the power of generative AI effectively. What I've been doing at MIT is we're calling legal prompt engineering. It's a subset of prompt engineering that's particularly focused on common legal tasks and legal concepts and legal processes and the legal context.
(23:11):
So, you can see more of that tutorials, including the one that you just referenced earlier at law.mi.edu/ai and all of that is free and open source. I guess I'd be remiss if I didn't hit on some of the practical things. Summarization I think is maybe one of the killer apps in the law for this technology. Part of this is because when you give these models the content you wanted to summarize as part of the prompt, it puts all of its attention on that text and it is much less likely, virtually never starts to stray or hallucinate or confabulate or get off point. It just puts its attention on the text and does a good job summarizing it. That's a very common thing that we need help with in the law.
(24:10):
You can have it critique things. I've been doing that a lot with scientific papers lately and with PhD students, some hypotheses. It does a pretty good job doing a critique or finding holes or problems or things like that. You can have an extract information like extract all of the named parties in this affidavit or what have you or in this set of email or Slack or whatever it's trying to do and see the relationships. You can do that reasoning. It's great at reformatting things, translating to different languages as a start, not perfect, but you can transform completely from a bunch of emails into a memo or that thing. You can make completely new things. So, as you get better at instructing and having an idea of goal, if you can articulate it, and as I said, we can articulate things in the law.
(25:02):
If you can envision and imagine and articulate what you want it to do and then write that in a simple clear way in the prompt, it's very good at generating a first draft of things that have never existed before. So, I guess I'll just end by saying before we got on this podcast today, I've been working on an automated prompt engineering little script that makes it easier to basically put the same prompt into a model a bunch of times and measure the differences, make sure it'll be reliable and look at a distribution of that. Then the other thing it does is it can change the prompt in subtle ways and then run it a bunch of times to see what the full impacts of that are. I'm going to make that available at dazzagreenwood.com on a blog probably by the time people hear this podcast.
Natalie Pierce (25:53):
Oh, that's fantastic.
Dazza Greenwood (25:54):
I encourage people to try it manually, but also think about trying it in a more programmatic automated way. So, you can get a good sense of what you can rely on it for and what you can't rely on it for.
Natalie Pierce (26:05):
Fantastic. At Gunderson, we've been brainstorming the best ways to leverage ChatGD, which is our homegrown generative AI chat app. We represent thousands of startups. So, as the head of Gunderson's employment law practice, we're really looking with our innovation team at creating our own employment collection within ChatGD. We've started collecting our greatest examples and samples with the ultimate goal really being to be able to ask questions back and forth, to quickly get tailored employment information and generate templates that we can use as a starting base for our clients. Just because we have retrieval augmented generation and because of that, we will do things like feed a document that is a leave policy.
(27:02):
Now, again, these are startups, so they're not going to have an employment lawyer on their team. They're not even necessarily going to have a head of human resources or people or talent. So, saying, "Okay, take this policy, it's a new policy, and come up with talking points for managers, come up with questions and answers that employees might have," yeah, it's incredibly effective for so many things that we lawyers do. You mentioned a lot of them, and even just converting legalese into plain language as for example. So, the applications are truly endless. I would like to stick with you, Dazza, if we could. Can you give any examples, the applications for prompt that you would describe as killer use cases?
Dazza Greenwood (27:55):
Absolutely. Yeah. The first one right out the box, I mentioned a moment ago, but it's summarization and there's more and more research coming out that shows just how powerful these modern large language models are at it. There's something I just posted about on LinkedIn a few days ago. It was a small scale experiment, so you can't put too much weight on it, but still it's relevant and it's some evidence. It showed that the summarizations were overall better than PhD students. Then there was some evidence that the summarization actually hallucinated by a technical definition of that word. The LL one hallucinated less than the students did as they started bringing in analogies and real world knowledge.
(28:38):
We're all fallible. We get things wrong a little bit, but when you're looking at the first draft, it compared very favorably. Of course, it does the draft in like three seconds. So, wow. Summarization is something I do probably 50, 60 times a day to get started on things. I now have a lot more understanding of a lot more things that people send me than I ever had the opportunity to even scan, much less read all the way through in the past. But some other high value applications include more structured document reviews and there's aspects of legal research and analysis that are good. I was just working with a team... I don't want to bring back traumatic memories, but remember in law school when we did IRAC, issue, rule, application, et cetera.
(29:28):
So, there's ways you can provide a template not just for to think into, but for it to write into by formatting into the template. So, it tells it the right things to look at, to extract, to reason about, and then to generate into the right template, which is I think also similar to what you were just describing for some of the workflows at your firm. It did a great job on cases when we made cases up and it did a perfectly good job, so it never would've seen it in the training set. I think also it's pretty good, surprisingly good in my view at risk detection. So, looking at something and trying to identify how could this go wrong.
(30:09):
Hallucinating is bad when you want a factual case to cite that may or may not know about, but when you're trying to imagine hypothetical futures, things could happen, it is a powerful augmentation to be able to imagine and suppose such things. Oh, God, I can just go on and off. I think the main thing is it's still relatively new and I think the task at hand now for people listening to this podcast is to try things and evaluate critically. So, I think we're still only scratching the surface on how this could work. I'm on the American Bar Association's task force for AI. We had a meeting earlier today.
Natalie Pierce (30:48):
Fantastic.
Dr. Lance Eliot (30:49):
One of the things that really was clear is the conversation generally up to now has been frameworks for ethics and for risk mitigation and for safety, all of which is appropriate. I'm sure we'll talk about that in due course, but the theme that really emerged and which I'm absolutely an advocate of myself is let's look in the law, what this is good at and how we embrace it and how we can apply it and adapt and evolve the practice of law to supercharge our role and our capabilities and the value that we provide to clients and to the economy and society at large. I think this is a huge unlock for lawyers and the law.
Natalie Pierce (31:34):
If you are a lawyer out there and you have not started and there are so many, again, excellent examples. Dazza, you said something that reminded me of how you can be an expert, how you can be the PhD candidate for example and want to find ways to, yes, summarize some works, but also communicate, because sometimes when you're an expert, you just get so lost in the weeds, because you know so much. That brings me back, Dr. Eliot, to what you had really recommended in one of your articles about that flipped interaction is a really critical prompt engineering technique, because you as the expert can ask the generative AI to ask you questions. You gave a very funny example of the expert in underwater basket weaving.
(32:35):
I thought that was priceless and really the way that it worked. I want you to ask me questions about underwater basket weaving, so that I can share with you the key principles involved. Then after asking these questions and you limit the number of questions, it gives you this recap at the end, but I thought that the questions were asked and finding a way to really communicate that expertise, I thought that was just a fantastic example. But Dr. Eliot, I'll let you add any other examples you want to give for applications for prompt that you would describe as killer use cases.
Dr. Lance Eliot (33:14):
Absolutely. Everything I've heard so far echoes what I would definitely say as well. I oftentimes try to shape the use of generative AI into the legal realm. I'm doing a workshop, in fact, yesterday, today and tomorrow, where I'm walking lawyers through different aspects of killer use cases for AI and the law. Also, I have an article in the California Lawyers Association periodical that covers this. Just real quick, what I typically do is I try to do five key killer use cases that will sound somewhat similar, I believe, to you and Dazza have just described. I'll just quickly rattle them off.
(33:57):
The first one is carry on a legal dialogue involving legal arguments to see what you might be up against in court or against a legal adversary by having that flipped interaction, where you have the generative AI try to challenge you and see how you respond to that. Another aspect is we use generative AI to examine voluminous materials. I know Dazza mentioned that.
(34:23):
For example, during e-discovery, when you're trying to ferret out what's significant to a case and what makes this different than what you would conventionally do, you conventionally might look for a particular keyword, but then the problem with keywords typically is that you can get back all sorts of junk that really doesn't contextually make sense for what you're looking with generative AI because of the computational pattern matching involved that it's able to invoke. It's able to have a better semblance of context, bring back hits or what you're trying to find in voluminous material in a way that keywords can't. Then the other three are similar to what you said.
(35:00):
So, using generative AI to compose a first pass of a legal document and then doing a human lawyer review to see how it comes out. That's very important because one aspect mentioned so far here is the idea of hallucinations and what that refers to. It's a term I don't like because it implies a semblance of sentience. We think of humans having hallucinations, but it's taken hold and so we have to just roll with it. But what it refers to is the generative AI effectively making stuff up. The problem with that is that it makes stuff up, but it often has an air of confidence to it in the sense that it might make up an entire legal case that doesn't really exist, but give so many details about it that you would swear when you look at it, well, it must be true, but often times, it isn't.
(35:52):
You have to be very careful of that. The other two, use generative AI to review a lawyer's composed legal document and see what generative AI says about the weaknesses, strengths, issues, ambiguities in something that a human lawyer composed. Then the last one, convert a legal document to plain English or possibly sharing with your clients, but once again, only after an additional human lawyer review to make sure that it's reflective of what the legal document is.
(36:21):
So, those are the five ways that in my workshops and that I've written about that I find are very handy for lawyers and law firms that are trying to get into this. That or anything in that realm is a good way to get started, to get a feel for what this is like, what prompting is like, what prompting engineering is about, the kinds of techniques that you can use, and how far you can potentially take this in a legal realm.
Natalie Pierce (36:46):
Perfect, perfect. Dazza, there's a Sam Altman quote that I just so love. I'm not going to be able to get my fingers on it, but what Dr. Eliot just said definitely reminded me when he was talking about the hallucinations. Just that it's really good, but it can again, with that confidence, give as semblance of correctness. I don't have the quote in front of me, maybe you know which one I'm talking about.
Dazza Greenwood (37:14):
Yeah, well, there's so many to choose from.
Natalie Pierce (37:16):
True, true.
Dazza Greenwood (37:18):
It's well-known behavior and it's generally considered a flaw if what you're trying to do is something that's fact based. I agree with Lance, anthropomorphizing these things and attributing human cognition and sanctions to them is not helpful. It means people will reason about them wrong. You'll come to the wrong conclusions about what they're doing and what they're likely to output. So, we shouldn't do that. For hallucination, at least in my clients and with my research team, I just use the term fabricate at this point. Hallucination actually, it has like a DSM pathological definition that we don't want to use and we don't want to impugn humans by diminishing those conditions either. But the main thing is we're going to reason incorrectly by misunderstanding fundamentally what the capability is.
(38:12):
You can edit this out if you want to, but I feel like we should mention what the technology is, because Lance just asserted quite rightly that this does not operate like Boolean search, what we do in Westlaw or Lexus or the type of pattern that we would even put in a search engine or even like a SQL query to a database. What's happening here is it's not based on find the word or the sequence of words, rather this is essential to what the technology is. It takes all of this knowledge, all of all these documents and other writings as part of the training set and part of the training method. It atomizes them down into these tiny little constituent elements and expresses them as math basically on something called vectors in vector space.
(39:06):
So, there are all these little parameters, these little points on these vectors and sets of vectors and matrices, billions of them in the larger models. So, you can end up seeing the deep relationship of the word if, to, and, and the verb to the noun in this sentence to that sentence. So, it's almost like seeing into the shape of the concepts that are happening. So, when you want to ask it about something or to do something, you don't need to use the words, do the exact words, rather it looks at the idea and the concept. It's great at analogies and metaphors or translating knowledge from one domain to another, or as you said, just even doing plain language, you have to get the concepts and restate them in a different way. This is completely different from how computation used to work, and it is very, very powerful.
(39:58):
It's really good for the law when you want to look at a cause of action or a situation and look across all the bodies of law to see where that comes up. You have to get lucky by putting the right words in. In the old days or hope that somebody caught a headnote correctly, not anymore. Now, we can finally pare deeply into the essence and the nature and the shape of the law. We can see the law for the first time, and that means we can use it. We can understand it. We can apply it. So, the high dimensional vector space is good for us, and that's fundamentally what's happening here.
Natalie Pierce (40:35):
Thank you for that, Dazza. We're definitely going to weave that into the episode. That gen AI workshop that you did and I don't remember the name of your guest, but he just worked through, showing exactly how this can work. In that case, I believe it was writing and helping write an opposition to a motion for summary judgment.
Dazza Greenwood (40:58):
Exactly. Yeah, Damien Real, who's another great fellow traveler, prompt engineer at the forefront. He was also a very experienced litigator, which is even more well testimony as it were to how important it's to bring our expertise and to bring our knowledge and our skill and our wisdom to the task of articulating what we're trying to do. The more articulate we are because of the way these systems work, their inference engines are really good and they do a pretty good job of getting what we're trying to say and then putting it in the shape that we want and hitting a lot of the right points conceptually.
(41:41):
Again, it's not like a fact. It's like you have to feed it facts that you want it to stick to. That's what we did in that show, which is it was an Idea Flow episode at law.mit.edu/media if you want to find it. Basically, he took motion to dismiss, I believe, or for summary judgment in one of the IP cases on I think involving OpenAI. He basically took the table of contents of the motion and then he saw what an answer would be, basically a rebuttal to each one. He did that and then he unfolded the rebuttal into what would be the elements that we'd have to prove for each thing. Then he went down and down and down. You saw it yourself.
Natalie Pierce (42:24):
Yeah, I was pretty blown away.
Dr. Lance Eliot (42:27):
It's basically like magic. I mean, I do basically understand what's going on, but every time I see this, I am just floored by how powerful and how on point this technology is for law.
Natalie Pierce (42:40):
Absolutely. All right. Well, Dr. Eliot, before we head into our quick fire challenge, which you've both agreed to, I'm going to leave our listeners with a final thought or tip.
Dr. Lance Eliot (42:56):
I guess one of the final tips I might provide is in the news these days, there's a lot of banner headlines of an AI doomsday heading towards us. The idea behind it is that as sophisticated as this computational pattern matching underlying the AI seems to be, are we on the cusp of what some would call artificial general intelligence, AGI? Meaning a type of AI that essentially would be indistinguishable from how humans think, which of course, we still to this day don't really know exactly how humans think, but in any case, the AI would be so good that you could not differentiate it from that, which you might do when you interact with a human. There's a famous test called the Turing Test named after Alan Turing. It relates to that.
(43:49):
What I believe, and there's a lot of controversy over this, is that we are not on the verge of that AGI doomsday. I find it overshadows unfortunately and distracts from the reality of what AI and generative AI can do today and will be doing in the next several years. So, I don't want to downplay how these advances are going to really impact society will be tremendously so, including in areas such as autonomous weapons systems, the idea of using AI that can control weapons and not have any human in the loop. That's a very serious aspect. What I find happens is people become, I think, a little too preoccupied with the existential risk idea that AGI is coming and it's going to wipe us all out.
(44:38):
I tell people, "Well, look, you use conventional AI of what we know of today or in the next several years, and you use that in an autonomous weapons system that can create mass destruction." To me, that's something I'd be more worried about than the idea of AGI that suddenly enslaves us or decides with a wave of a computer system to wipe us all out. I guess just the tip is there's a lot of banner headlines about it. It's a very controversial topic. There are many that agree or disagree about how close or how far we are away from this. My view is focus on the reality of what AI can do today and what I believe will be happening in the next 5 to 10 years. Let's certainly talk about the existential risks, but let's not let us be blinded by that and overlook or ignore the realities of what AI can and cannot do today.
Natalie Pierce (45:34):
Hear, hear. There's so many potential applications of AI for good, and we're already seeing countless such applications. So, I will remain an optimist and we are going to head right into our quickfire challenge as we are almost at the hour here. So, we've talked so much about technology in the future. Dazza, we'll start with you. What was your favorite book or movie growing up as a young adult or child and why?
Dazza Greenwood (46:10):
I really liked The Foundation series and I really liked Lord of the Rings so much. It just did world building. It was creative. It sparked curiosity. It showed what courage and fear and relationships and honor and dedication and getting over adversity looked like. It showed a bigger picture and how it played out with individual actions. I just love them. Now, of course, they're all major Hollywood productions, which is even better. All the stuff from my childhood is big screen now, so I'm loving that.
Natalie Pierce (46:47):
Oh, I love it. How about you, Dr. Eliot?
Dr. Lance Eliot (46:50):
So between books and movies, I'd like to do books and places or place if I could. The book would be when I was in college as a computer science major, I read the Handbook of Artificial Intelligence by Feigenbaum and Barr. This now is a dated book unfortunately, but at the time, it had what was considered the state-of-the-art comprehensively about the field of AI. It's what then launched me into basically the rest of my career doing consulting work, being a Chief Information Officer, Chief Technology Officer using AI, a scholar in AI, and so on was that book back in that day. I think it's still worth reading, but of course, it's dated now.
(47:28):
So, you'd have to certainly augment it with more contemporary aspects, but it still had the fundamentals in it. The other would be, if I could just quickly mention, is in terms of place, so I was born and raised in Southern California. I'm very lucky in the sense that I lived near Disneyland and I went there all the time. I went there on weekends, evenings, summers, just all the time. The reason I bring that up, my focus at Disneyland was Tomorrowland. At Tomorrowland, it was a showcase of the future-
Natalie Pierce (47:54):
My favorite.
Dr. Lance Eliot (47:55):
... especially technology that we might one day have. Yes. So, as a child going there all the time, I would just be amazed at the technology that I saw, which of course today, we'd laugh at or something like that to think how crude it was at the time. But it inspired me to go down the computer and technology route with the dream in my mind, perhaps coming from Disney and himself in a sense that maybe I could do something about the future in terms of being able to develop this technology. I'll just add one other thing, one point. I actually worked at the Walt Disney Imagineering, WDI, that conceives and designs the theme parks. So, it turned full circle around for me.
Natalie Pierce (48:39):
Wow. Wow. Okay. Then one more question before I let you both go. Dazza, we are talking about being far from AGI. Hopefully, that is true. So, let's talk about humans. What is your favorite or least favorite human trait?
Dazza Greenwood (48:58):
Yes, there may be a bit of a theme with my answer to the last question about why those books.
Natalie Pierce (49:02):
I was thinking there might be.
Dazza Greenwood (49:06):
There's a lot of human traits I really like. I guess I go with curiosity and playful creativity. That never gets old and applying that with friends and relationships, but even at work, there's a place for that. When you're especially doing innovative things, that attitude can be very helpful to look beyond the horizon and to have the courage to try things. I guess I'm thinking about not just an individual person like in a capsule, isolated, being curious and playful, but with other people. When people in groups have the group dynamics of curiosity and playfulness, I love that. That's partly why I'm associated with MIT, because the Media Lab has got a culture of the intrepid hacker spirit where we get together and prototype things and do stuff like that.
(50:01):
I live in Oakland now and there's a lot of interesting cultural communities here where we build things, do stuff. So, I'd say that would be my favorite characteristic. I know we're getting away from the theme of the show, but I have to mention it is very well tuned for how to look at and address and have the courage to look in the face of the big changes that are happening now and to face them and to adapt to them and to make the best of it, so we can have a better tomorrow.
Natalie Pierce (50:32):
Great answer. Great answer. Dr. Eliot, how about you, favorite or least favorite human trait?
Dr. Lance Eliot (50:40):
Well, I guess, Dazza and I, we know each other. We're great colleagues and I guess we think alike in many ways. His comment about the idea of curiosity and so on, for me, the human trait that I really favor is the quest to figure out the unknown. That's why I went into AI. I wanted to try to figure out how is it that humans think and how is it that we can make technology that appears to do what humans do in terms of thinking, whether it actually replicates that or in some way simulates that.
(51:11):
That's an open question, but just the ability to get technology to behave in that same fashion is really what has drove me into the area and continues to keep me in that area today. To a comment that you made just a few moments ago, I do that also because I also strongly believe that if we can attain that kind of AI, that it will help us to do things like cure cancer or possibly deal with climate change or aid us in resolving a lot of major issues facing humankind that I believe AI as a beneficial tool can really help us with.
Natalie Pierce (51:49):
Wonderful. Well, I say let's all stay curious and creative and lifelong learners. I think we all have an optimistic view of what's possible and what the future holds. Unfortunately, we're out of time. So, I just want to thank you both so much for joining us today for our season finale and what was a great conversation on killer use cases for gen AI. I truly thank you both so much.
Dr. Lance Eliot (52:24):
Thank you so much.
Dazza Greenwood (52:26):
Thank you so much. Great speaking with you. It's always great to be in conversation with Lance again, so thanks so much.
Natalie Pierce (52:32):
Wonderful, wonderful. Thank you listeners for joining us. The successful future of the legal industry is intertwined with the ability of lawyers to responsibly embrace new technologies like generative AI. Bye-bye for now.
Voiceover (52:50):
You've just listened to the Future Work Playbook. This podcast series is brought to you by Gunderson Dettmer, the world's number one law firm representing venture capital funds and high growth companies. Join Natalie Pierce on our next episode as she and her guests help prepare your organization for the future. Please subscribe to the Future Work Playbook.
