WEBVTT - This file was automatically generated by VIMEO
0
00:00:02.565 --> 00:00:05.905
The Institute of Internal Auditors presents all things
1
00:00:06.345 --> 00:00:08.345
internal audit AI podcast.
2
00:00:09.645 --> 00:00:13.025
In this episode, Sue King of KPMG
3
00:00:13.405 --> 00:00:16.985
and Grant Osler of Workiva discuss the potential
4
00:00:16.985 --> 00:00:21.265
of generative AI to speed up internal audit processes
5
00:00:21.565 --> 00:00:24.945
and support risk management in evolving risk areas.
6
00:00:25.765 --> 00:00:29.105
Hi, I'm here with Grant from Workiva and Sue from KPMG,
7
00:00:29.245 --> 00:00:31.105
and we're talking about generative ai.
8
00:00:31.735 --> 00:00:33.945
What are some best practices for, uh,
9
00:00:34.425 --> 00:00:36.385
entering prompts into the, uh, into the tool?
10
00:00:36.605 --> 00:00:39.265
That's a great question. So I think, you know, just being
11
00:00:39.405 --> 00:00:41.225
as specific as you can be, right?
12
00:00:41.305 --> 00:00:43.785
I mean, you can, you can write a very, very simple
13
00:00:43.885 --> 00:00:45.665
or open plain language prompt,
14
00:00:45.725 --> 00:00:48.105
but the more specific you can be of like, compare it
15
00:00:48.105 --> 00:00:51.345
to this framework or like specific to, like you were saying,
16
00:00:51.345 --> 00:00:53.865
like specific to audit or like, uh, yeah.
17
00:00:54.005 --> 00:00:56.265
The more specificity, the better an as you can get.
18
00:00:56.385 --> 00:00:57.905
Yeah. The more context you provide,
19
00:00:57.905 --> 00:00:59.345
the better you're gonna get results back.
20
00:00:59.525 --> 00:01:00.665
And one of the things we were talking about
21
00:01:00.665 --> 00:01:02.865
before was we things in preparing for a session,
22
00:01:02.865 --> 00:01:05.225
we talked this, I like to, like,
23
00:01:05.375 --> 00:01:07.825
I'll rewrite the prompt three or four different ways.
24
00:01:08.055 --> 00:01:10.105
I'll tweak it and I'll compare the results back.
25
00:01:10.105 --> 00:01:11.785
Say, oh, okay, these things are common.
26
00:01:11.815 --> 00:01:14.225
Okay, I know that's important. These things are outliers.
27
00:01:14.225 --> 00:01:16.545
Where am I going? And ask for references like,
28
00:01:16.545 --> 00:01:17.665
where did you get this from?
29
00:01:17.965 --> 00:01:19.785
So you can take some of the hallucinations,
30
00:01:20.095 --> 00:01:22.065
it's gonna do it, but I can identify
31
00:01:22.185 --> 00:01:23.225
'em, I can deal with them a lot easier.
32
00:01:23.375 --> 00:01:24.745
Will will it cite its references?
33
00:01:25.165 --> 00:01:26.585
It will if you ask it. Right. That's great.
34
00:01:26.885 --> 00:01:29.385
And it's, and you don't have to, you don't have to be that
35
00:01:29.955 --> 00:01:31.065
articulate in doing it.
36
00:01:31.135 --> 00:01:33.225
Yeah. It fills in the blanks pretty well,
37
00:01:34.005 --> 00:01:35.185
but the more you do it,
38
00:01:35.185 --> 00:01:36.705
just like anything else, the better you get.
39
00:01:37.045 --> 00:01:38.385
So practice a lot. Yeah.
40
00:01:38.405 --> 00:01:40.465
And the more you can be, you know, tell it
41
00:01:40.465 --> 00:01:41.865
to like, okay, that's great.
42
00:01:42.545 --> 00:01:44.825
Summarize it or what, what are the highlights out of this?
43
00:01:44.935 --> 00:01:48.225
Like, to, to really get it to fine tune, fine tune answers
44
00:01:48.225 --> 00:01:50.705
so you don't end up with pages and pages. Right.
45
00:01:51.245 --> 00:01:54.265
And what, what are some of the most common uses
46
00:01:54.265 --> 00:01:56.545
of artificial intelligence by internal auditors?
47
00:01:57.255 --> 00:01:58.545
What, what are the, what are you seeing
48
00:01:58.565 --> 00:01:59.785
out there? Uh, how are people
49
00:02:00.025 --> 00:02:01.025
Adopting? I think what we're seeing a lot in neutral audit,
50
00:02:01.025 --> 00:02:02.145
a lot of,
51
00:02:02.205 --> 00:02:03.625
lot of work on the front end saying, look,
52
00:02:03.735 --> 00:02:04.985
what are the risks in this area?
53
00:02:05.495 --> 00:02:06.785
What are the most important ones?
54
00:02:06.785 --> 00:02:08.305
How are they gonna be, how would,
55
00:02:08.305 --> 00:02:09.825
what controls would I look for in that?
56
00:02:10.015 --> 00:02:11.865
Like what kind of procedures would I do for that?
57
00:02:11.965 --> 00:02:13.505
So a lot in the planning is really big.
58
00:02:13.765 --> 00:02:15.305
We see a lot on the backend.
59
00:02:16.425 --> 00:02:18.785
I don't know about anybody else, but we used to spend a lot
60
00:02:18.785 --> 00:02:20.025
of time wordsmithing reports.
61
00:02:21.045 --> 00:02:23.145
Gen AI writes better than we do most of the time.
62
00:02:23.845 --> 00:02:25.625
Let it take the pass. Then you go into refine it.
63
00:02:25.625 --> 00:02:27.825
It's hard to start with a blank piece of paper, right?
64
00:02:27.825 --> 00:02:28.985
But if I can start with something
65
00:02:29.185 --> 00:02:32.345
that's pretty well crafted, I can then go in and adjust it
66
00:02:32.345 --> 00:02:33.985
and bring it to where I need to be really quickly.
67
00:02:34.485 --> 00:02:37.265
It frees your time up to do more value added audit instead
68
00:02:37.265 --> 00:02:38.785
of spending your time wordsmithing reports.
69
00:02:39.165 --> 00:02:41.985
And how much data entry does it take to, to get it
70
00:02:41.985 --> 00:02:45.625
or, you know, to get it to, uh, write that draft report.
71
00:02:45.695 --> 00:02:49.015
Like how much, uh, prompting do you have to give it?
72
00:02:49.615 --> 00:02:52.015
I, yeah, I don't feel like it really needs that much.
73
00:02:52.285 --> 00:02:53.855
It's like, you know, I've got this exception.
74
00:02:53.855 --> 00:02:56.375
Write me a recommendation, uh, you know,
75
00:02:56.375 --> 00:02:57.575
make it action oriented.
76
00:02:57.645 --> 00:02:58.855
Make it detailed, right.
77
00:02:58.855 --> 00:03:00.165
Write, you can put all of that in there.
78
00:03:00.465 --> 00:03:01.485
Do you want it in bullet point?
79
00:03:01.485 --> 00:03:02.765
You can tell it what you want
80
00:03:02.825 --> 00:03:05.405
and it will give you what you ask for. It's amazing.
81
00:03:05.905 --> 00:03:07.925
So I think that's, that's one of the great use cases,
82
00:03:07.975 --> 00:03:09.605
right, is like the, the planning
83
00:03:09.705 --> 00:03:12.005
and you know, making you smarter about you going to a,
84
00:03:12.085 --> 00:03:14.045
a new area that you don't really know as much about
85
00:03:14.045 --> 00:03:15.725
that's kind of more technical
86
00:03:15.745 --> 00:03:17.925
or, you know, specialized to an industry.
87
00:03:18.035 --> 00:03:20.645
Like make, helps you get smarter in that planning phase.
88
00:03:21.145 --> 00:03:22.445
Um, and then as Grant says,
89
00:03:22.445 --> 00:03:23.845
like gets you to that recommendation,
90
00:03:24.195 --> 00:03:25.325
Does it have the ability yet
91
00:03:25.325 --> 00:03:27.005
to generate the work programs?
92
00:03:27.925 --> 00:03:29.725
Absolutely. It does. It does. Yeah, absolutely it does.
93
00:03:29.985 --> 00:03:31.925
So, so I think, yeah, so doing the planning,
94
00:03:32.025 --> 00:03:33.645
the recommendation, but then I think, you know,
95
00:03:33.925 --> 00:03:36.485
actually doing the testing as well to you can,
96
00:03:36.505 --> 00:03:40.365
you can give it documents to ingest, um, you know, be they,
97
00:03:40.585 --> 00:03:43.045
you know, whatever format, Excel, PDF or whatever.
98
00:03:43.105 --> 00:03:45.725
And you can tell it like, you know, compare and contrast.
99
00:03:45.745 --> 00:03:48.365
So you can start doing some testing procedures to say,
100
00:03:48.705 --> 00:03:51.405
you know, like, okay, like the example that we went
101
00:03:51.405 --> 00:03:52.645
through in our session yesterday is
102
00:03:52.645 --> 00:03:54.285
around terminated user testing, right?
103
00:03:54.425 --> 00:03:57.645
Do I have people who are, who've got active access
104
00:03:57.645 --> 00:03:58.965
that are not employees?
105
00:03:59.545 --> 00:04:02.285
But then the beauty of AI is that then you can say like,
106
00:04:02.285 --> 00:04:03.525
okay, well once I found people
107
00:04:03.525 --> 00:04:05.965
that should have been terminated, then you know,
108
00:04:05.965 --> 00:04:08.005
you can give it another data source and say, well go
109
00:04:08.005 --> 00:04:10.165
and look and see did that person actually do anything
110
00:04:10.165 --> 00:04:11.325
after their termination date?
111
00:04:11.585 --> 00:04:13.085
Or like you can give it the tickets
112
00:04:13.225 --> 00:04:16.485
and say, was the ticket issued for that terminated user
113
00:04:16.865 --> 00:04:18.325
and it just didn't get executed on.
114
00:04:18.345 --> 00:04:21.645
So you can start getting into a lot more of those, um, kind
115
00:04:21.645 --> 00:04:24.325
of what if and look back type analysis very easily.
116
00:04:25.045 --> 00:04:27.245
'cause there's been been a focus for a long time of like,
117
00:04:27.245 --> 00:04:29.165
how do we get to the point where internal auditors
118
00:04:29.165 --> 00:04:30.645
or value added, right?
119
00:04:30.785 --> 00:04:34.165
And so I think really being able to go to a control owner
120
00:04:34.165 --> 00:04:36.245
and not only say like, oh, you've got an exception,
121
00:04:36.245 --> 00:04:37.685
which okay, they hate hearing that.
122
00:04:38.105 --> 00:04:40.405
But then if you've been able to get to the next level
123
00:04:40.505 --> 00:04:42.845
and say, well, I've done root cause analysis,
124
00:04:42.865 --> 00:04:45.445
the issue is like, the tickets are getting issued,
125
00:04:45.445 --> 00:04:46.965
but they're not getting closed out properly.
126
00:04:47.545 --> 00:04:51.565
Or, um, or even saying like, Hey, I did this analysis.
127
00:04:51.785 --> 00:04:54.685
If you first line were to, were to take that activity,
128
00:04:55.155 --> 00:04:56.605
this would be your control activity
129
00:04:56.605 --> 00:04:58.445
and here's how we would, how we would resolve it.
130
00:04:58.465 --> 00:05:01.205
So I think getting, again, by getting back to that,
131
00:05:01.265 --> 00:05:05.245
you know, action oriented value added to the the first line,
132
00:05:05.925 --> 00:05:07.125
I think that's a huge point.
133
00:05:07.255 --> 00:05:10.485
Being able to shift from assurance to advisory
134
00:05:11.025 --> 00:05:13.365
and help the first and second line do their job better.
135
00:05:14.215 --> 00:05:17.565
We're able to amplify the impact we have. It's huge.
136
00:05:17.945 --> 00:05:20.285
And so the more we can help this move upstream,
137
00:05:20.625 --> 00:05:22.125
the more impactful we are as auditors.
138
00:05:22.125 --> 00:05:23.885
What, what are some of the most creative uses
139
00:05:23.885 --> 00:05:26.525
that you've seen of, of AI by auditors?
140
00:05:26.545 --> 00:05:29.365
You know, how, uh, have you seen anything where, you know,
141
00:05:29.365 --> 00:05:31.605
you're surprised at, uh, at what it could do
142
00:05:31.605 --> 00:05:32.605
Or? Well, I think there's, I
143
00:05:32.605 --> 00:05:33.245
think there's a lot
144
00:05:33.245 --> 00:05:34.405
of different things, right?
145
00:05:34.505 --> 00:05:37.325
So we're, you know, we're experimenting with using,
146
00:05:37.975 --> 00:05:42.325
using copilot for example, to, uh, to document, uh,
147
00:05:42.435 --> 00:05:43.725
from a walkthrough that we're having.
148
00:05:44.345 --> 00:05:46.525
And then, you know, it can create your first draft
149
00:05:46.525 --> 00:05:48.965
of a walkthrough and then you can say like, okay,
150
00:05:49.065 --> 00:05:50.565
now compare the walkthrough of
151
00:05:50.565 --> 00:05:52.165
what the process owner just described,
152
00:05:52.265 --> 00:05:53.685
the, the process to me.
153
00:05:53.915 --> 00:05:56.925
Compare that to the process narrative or to the rackum
154
00:05:57.105 --> 00:05:59.685
or to the flowchart and tell me why there are differences.
155
00:06:00.265 --> 00:06:03.605
And, and you know, the, the beauty of it is the speed
156
00:06:03.835 --> 00:06:07.205
that then you can go back to the control owner, you know,
157
00:06:07.205 --> 00:06:09.285
the next day and say, Hey, when we talked about this,
158
00:06:09.465 --> 00:06:10.925
you never, you didn't touch on this.
159
00:06:11.355 --> 00:06:14.605
Like, was that we just omitted it or has it changed? Right?
160
00:06:14.945 --> 00:06:17.725
As opposed to, you know, historically it might have taken,
161
00:06:17.865 --> 00:06:19.965
you know, a week or to get back to it.
162
00:06:19.985 --> 00:06:21.605
And then the, the control owner's like, oh,
163
00:06:21.605 --> 00:06:22.645
I don't remember what we talked about.
164
00:06:22.785 --> 00:06:25.845
So, so, you know, some of it is, uh, as Grant said,
165
00:06:25.845 --> 00:06:29.525
you know, kind of takes, takes some of the, uh, kind of more
166
00:06:29.525 --> 00:06:31.205
of the grunt work kind of out of this, right?
167
00:06:31.305 --> 00:06:34.165
So that we are more focused on, on value add. Yeah.
168
00:06:34.265 --> 00:06:36.405
Um, but the speed thing is, is also terrific.
169
00:06:37.165 --> 00:06:38.805
I think that point about taking some of that,
170
00:06:38.805 --> 00:06:42.085
that busy monotonous workout, every auditor I know,
171
00:06:42.085 --> 00:06:43.685
every audit leader is trying
172
00:06:43.685 --> 00:06:45.085
to staff their team with the right people.
173
00:06:45.385 --> 00:06:46.525
And it's a struggle right now.
174
00:06:46.985 --> 00:06:48.965
If you can get your people working on the right things.
175
00:06:49.245 --> 00:06:51.885
A they like their job more. Nobody likes right?
176
00:06:51.885 --> 00:06:54.365
Narratives, I don't, maybe there is, but I don't know.
177
00:06:54.405 --> 00:06:55.885
'em, they never work for me, right?
178
00:06:56.305 --> 00:06:58.325
So take the stuff they don't like to do,
179
00:06:58.845 --> 00:06:59.925
automate as much of that as you can.
180
00:07:00.305 --> 00:07:02.005
Let them focus on things that adds more
181
00:07:02.005 --> 00:07:03.165
value, that is more fun for them.
182
00:07:03.505 --> 00:07:06.325
Our retention will get better, our ability to keep them to,
183
00:07:06.325 --> 00:07:07.725
and move those people into higher value
184
00:07:07.725 --> 00:07:08.925
roles in the organization gets better.
185
00:07:09.185 --> 00:07:11.925
So it's, it's kinda that virtuous cycle is my opinion.
186
00:07:12.945 --> 00:07:15.525
And what do you see as like the next frontier for, uh,
187
00:07:15.685 --> 00:07:16.925
internal audits use of ai?
188
00:07:16.925 --> 00:07:18.605
Right? Where, where are we going next?
189
00:07:20.165 --> 00:07:23.135
Well, I just, you know, I think going back to
190
00:07:23.135 --> 00:07:24.495
what I was saying earlier, right?
191
00:07:24.575 --> 00:07:27.215
I feel like we've all been stuck, you know, doing SOCs or,
192
00:07:27.315 --> 00:07:29.895
or doing some of the kind of like the standard audits.
193
00:07:30.315 --> 00:07:33.495
Um, you know, but I think this tool really enables us to,
194
00:07:34.075 --> 00:07:37.175
to speed up and again, like really get into that value added
195
00:07:37.355 --> 00:07:40.095
and, and to be able to use that knowledge to say like, okay,
196
00:07:40.095 --> 00:07:42.055
something that maybe I might've needed
197
00:07:42.175 --> 00:07:43.255
a specialist to help me.
198
00:07:43.255 --> 00:07:45.855
Like, you know, you've got much more of a, a of a jumpstart.
199
00:07:45.855 --> 00:07:49.895
But I do think that whole like value added piece, um, and,
200
00:07:49.915 --> 00:07:53.215
and more insight being able to, to really dig into that, uh,
201
00:07:53.215 --> 00:07:54.735
that root, root cause analysis
202
00:07:54.735 --> 00:07:56.655
that we've all been talking about for a long time.
203
00:07:57.815 --> 00:08:00.255
I think one of the challenges auditors have is writing
204
00:08:01.675 --> 00:08:03.955
a real business case in a recommendation, right?
205
00:08:03.955 --> 00:08:05.435
Really make the sale. And,
206
00:08:05.435 --> 00:08:06.635
and we don't always do that very well.
207
00:08:07.535 --> 00:08:09.915
You can ask Gen I to write it in a tone
208
00:08:10.005 --> 00:08:13.435
where it will do a much better job that'll drive adoption.
209
00:08:14.335 --> 00:08:15.515
If, if things don't change.
210
00:08:15.535 --> 00:08:16.675
We write up a finding we're
211
00:08:16.675 --> 00:08:17.955
of no value, let's just be honest.
212
00:08:18.605 --> 00:08:20.515
We're of value when the operation gets better
213
00:08:20.515 --> 00:08:22.235
and they change and they have less risk
214
00:08:22.395 --> 00:08:23.915
'cause they've got better controls or other things in place.
215
00:08:24.255 --> 00:08:26.075
And I think gen I can help in that a lot
216
00:08:26.135 --> 00:08:27.515
and it's gonna keep getting better.
217
00:08:28.975 --> 00:08:32.355
Can Gene AI make recommendations at this point
218
00:08:32.455 --> 00:08:34.875
and uh, you know, like beyond just a high level generic,
219
00:08:34.895 --> 00:08:36.115
can they, you know, can,
220
00:08:36.175 --> 00:08:37.475
can it really generate some
221
00:08:37.525 --> 00:08:39.195
insightful recommendations? Yeah.
222
00:08:39.195 --> 00:08:41.035
Yeah. We've seen it. We've seen it do that. Yeah.
223
00:08:41.455 --> 00:08:43.875
At, at a detailed control, a control level, right?
224
00:08:43.915 --> 00:08:45.155
Yeah. Yeah. For sure Isn't,
225
00:08:45.735 --> 00:08:47.195
But it's not on autopilot, right?
226
00:08:47.335 --> 00:08:50.195
You still want the professional who has experience to say,
227
00:08:50.585 --> 00:08:51.595
does this make sense?
228
00:08:52.095 --> 00:08:54.235
Is that right? Based on what we saw, right?
229
00:08:54.235 --> 00:08:57.125
Because it, it, Charles King who is mm-hmm.
230
00:08:57.205 --> 00:09:00.445
From KPMG who was on our group yesterday was ready, is like,
231
00:09:00.635 --> 00:09:01.965
it's a huge algorithm, right?
232
00:09:01.965 --> 00:09:03.045
It's, it's doing math.
233
00:09:03.825 --> 00:09:06.445
You need to make sure that, that it's still on point, right?
234
00:09:06.865 --> 00:09:09.925
So I think that, again, I don't see this replacing auditors.
235
00:09:10.045 --> 00:09:12.805
I see this in letting auditors be a whole lot more
236
00:09:12.805 --> 00:09:14.045
important, a lot more impactful,
237
00:09:14.335 --> 00:09:15.335
Right? Yeah. Yeah. I,
238
00:09:15.335 --> 00:09:16.645
I, I totally agree. Right?
239
00:09:16.675 --> 00:09:17.765
It's like that first version,
240
00:09:17.865 --> 00:09:20.805
but we all know it's way easier to edit something than it is
241
00:09:20.805 --> 00:09:22.405
to be staring at a blank sheet of paper. Yeah.
242
00:09:22.455 --> 00:09:24.765
White tape are scary, Right? Pretty
243
00:09:24.765 --> 00:09:25.765
Sure. Uh, are
244
00:09:25.765 --> 00:09:29.205
there ways to connect, uh, an AI platform
245
00:09:29.265 --> 00:09:33.125
to your enterprise systems so that the AI can be monitoring
246
00:09:33.225 --> 00:09:35.045
or analyzing the, the data?
247
00:09:35.045 --> 00:09:38.245
Yeah, so, and I can speak to Workiva solution,
248
00:09:38.245 --> 00:09:39.765
I can't speak to anybody else, but you know,
249
00:09:39.825 --> 00:09:42.165
our system is designed with APIs that connect other systems
250
00:09:42.185 --> 00:09:43.805
and bring the data in and, and
251
00:09:43.805 --> 00:09:46.485
because the gen AI is right native in our application,
252
00:09:46.525 --> 00:09:48.365
I can go right to that data and I think it's there.
253
00:09:48.745 --> 00:09:53.045
So yeah, it's really there, um, understanding
254
00:09:53.155 --> 00:09:55.525
what data you need and getting the right connections in.
255
00:09:55.525 --> 00:09:56.605
If I don't have a connection
256
00:09:56.605 --> 00:09:58.165
and maybe it's not something I'm gonna do repetitively,
257
00:09:58.565 --> 00:09:59.525
I can bring it in in the form
258
00:09:59.525 --> 00:10:00.725
of a spreadsheet or something like that.
259
00:10:00.725 --> 00:10:03.125
So it's amazingly simple actually
260
00:10:03.125 --> 00:10:04.485
to bring the data in and connect up
261
00:10:04.485 --> 00:10:05.845
to it. Yeah. Which is really fun.
262
00:10:06.555 --> 00:10:07.605
Yeah. So
263
00:10:08.045 --> 00:10:10.525
Ultimately, how do you see, uh, internal audit
264
00:10:11.075 --> 00:10:13.685
incorporating artificial intelligence into its work,
265
00:10:13.905 --> 00:10:15.085
you know, five years from now?
266
00:10:15.105 --> 00:10:17.525
How, how would, what would you envision, you know,
267
00:10:17.525 --> 00:10:19.445
in internal audit's use of AI to be?
268
00:10:19.445 --> 00:10:21.125
Well, it's interesting, we were talking about it, right?
269
00:10:21.145 --> 00:10:23.045
You know, like things are moving so quickly.
270
00:10:23.225 --> 00:10:25.885
So five years, by five years from now, like, gosh,
271
00:10:25.905 --> 00:10:28.405
we might be on the moon, but, um, you know,
272
00:10:28.405 --> 00:10:31.205
but I, I do think like this, this, you know,
273
00:10:31.205 --> 00:10:32.845
there's been this push, like how do we,
274
00:10:32.905 --> 00:10:34.845
how do we use more technology
275
00:10:34.845 --> 00:10:37.485
and more automation in the way that we're auditing,
276
00:10:37.485 --> 00:10:39.445
whether it's internal audit or whether it's sox.
277
00:10:39.865 --> 00:10:41.965
And I think this has really given us that, you know,
278
00:10:41.965 --> 00:10:44.205
that low code, no code solution Mm-hmm.
279
00:10:44.425 --> 00:10:46.045
To be able to start doing analysis.
280
00:10:46.305 --> 00:10:48.445
You know, I think, you know, like is it the,
281
00:10:48.825 --> 00:10:52.285
is it the perfect solution if you wanna do continuous
282
00:10:52.285 --> 00:10:54.005
monitoring or whatever, maybe not.
283
00:10:54.305 --> 00:10:57.525
But what it allows you to do is to do that, you know, like,
284
00:10:57.795 --> 00:10:59.765
okay, well I, I did it using gen AI
285
00:10:59.765 --> 00:11:01.645
and I proved that like, this is gonna be valuable
286
00:11:01.645 --> 00:11:02.885
and useful, so now let's go
287
00:11:02.885 --> 00:11:05.445
and figure out like, what is the right technology solution.
288
00:11:05.665 --> 00:11:08.365
So I, I just think, uh, um, yeah, it,
289
00:11:08.395 --> 00:11:09.965
it's got a myriad of possibilities.
290
00:11:10.545 --> 00:11:12.245
Um, it's not gonna, as Grant says,
291
00:11:12.265 --> 00:11:13.685
not gonna replace auditors,
292
00:11:13.745 --> 00:11:16.965
but really allows us a, again, to be much more
293
00:11:17.565 --> 00:11:18.725
valuable for the first line.
294
00:11:18.865 --> 00:11:20.245
Um, which I think is terrific. Yeah.
295
00:11:20.515 --> 00:11:23.845
Yeah. I mean it's, it's what, 15 or 16 months old now?
296
00:11:23.865 --> 00:11:25.445
And we're in a whole different world,
297
00:11:25.825 --> 00:11:27.565
so I don't, I'm not that good.
298
00:11:27.605 --> 00:11:28.965
A prognosticator say five years.
299
00:11:29.065 --> 00:11:30.685
I'm not sure I know what a year from now looks like.
300
00:11:31.265 --> 00:11:35.245
But, but what I see is that as people get in
301
00:11:35.245 --> 00:11:38.045
and use it, they're learning ai,
302
00:11:38.045 --> 00:11:39.325
like everything else comes with risks.
303
00:11:39.945 --> 00:11:42.325
And if you're not in it using it, your audit team,
304
00:11:42.465 --> 00:11:44.085
you're not understanding those risks well enough
305
00:11:44.085 --> 00:11:45.285
to really advise your company
306
00:11:45.705 --> 00:11:46.765
and you're not helping your company.
307
00:11:47.245 --> 00:11:48.805
I can guarantee you somebody in marketing
308
00:11:48.805 --> 00:11:49.805
or somewhere else is using it.
309
00:11:50.345 --> 00:11:52.685
And if you're not helping put the right governance in place
310
00:11:52.685 --> 00:11:54.380
around this, you're not doing your
311
00:11:54.380 --> 00:11:55.445
job as an auditor in my opinion.
312
00:11:55.695 --> 00:11:58.525
Right? So you've gotta be well enough versed to have
313
00:11:58.525 --> 00:12:00.005
that conversation with the other people,
314
00:12:00.105 --> 00:12:01.725
to put things in place, have those
315
00:12:01.725 --> 00:12:02.765
right guardrails in place.
316
00:12:03.065 --> 00:12:05.005
And I think that's maybe the next step for us.
317
00:12:05.005 --> 00:12:06.725
I don't know how many, three or four steps, I dunno what
318
00:12:06.725 --> 00:12:09.125
that looks like, but I think right now it's get in,
319
00:12:09.125 --> 00:12:10.885
get familiar, use it practice
320
00:12:10.885 --> 00:12:12.965
and use it so that you can really sit down
321
00:12:12.965 --> 00:12:15.605
and have an intelligent conversation with your business.
322
00:12:15.605 --> 00:12:16.925
Say, look, these are risks.
323
00:12:17.225 --> 00:12:19.965
Here's how we're addressing them. What are you doing?
324
00:12:20.465 --> 00:12:22.205
How do we do this? And really put things in place.
325
00:12:22.345 --> 00:12:23.485
By the way, I'll help you write the policy
326
00:12:24.225 --> 00:12:26.205
And then just, uh, one thing that, that sparks.
327
00:12:26.205 --> 00:12:27.845
So the other thing we haven't talked about, right?
328
00:12:27.845 --> 00:12:29.485
We're talking about using gen ai,
329
00:12:29.785 --> 00:12:32.565
but I think as auditors, exactly as Grant says, you have
330
00:12:32.565 --> 00:12:35.165
to be familiar with it because, you know, the rest
331
00:12:35.165 --> 00:12:37.325
of the business is going to be using it.
332
00:12:37.345 --> 00:12:38.845
And so we need to look at it, we need
333
00:12:38.845 --> 00:12:40.845
to come at it from a controls perspective as well, right?
334
00:12:40.845 --> 00:12:42.005
And say like, okay,
335
00:12:42.005 --> 00:12:43.405
what's the governance we've got over this?
336
00:12:43.415 --> 00:12:45.205
Let's make sure that we're managing it.
337
00:12:45.745 --> 00:12:48.645
Are there any other, uh, you know, things that you know,
338
00:12:48.645 --> 00:12:50.885
you're most excited about in, in AI
339
00:12:50.885 --> 00:12:53.165
and like, you know, bringing it to your clients
340
00:12:53.305 --> 00:12:54.925
or incorporating it into your platform.
341
00:12:54.925 --> 00:12:57.005
What, what, what, what's the, uh,
342
00:12:57.705 --> 00:12:58.765
you know, the, the next target.
343
00:12:58.945 --> 00:13:01.605
Nobody knows better about chasing down data
344
00:13:01.605 --> 00:13:03.085
to make sure it's complete and accurate than auditors.
345
00:13:03.085 --> 00:13:04.125
That's what we do every day.
346
00:13:04.705 --> 00:13:08.485
And so I think there's frontiers in partnering
347
00:13:08.485 --> 00:13:10.525
with the business and all these things in these new emerging
348
00:13:10.595 --> 00:13:11.845
risk areas, right?
349
00:13:12.135 --> 00:13:14.725
Where we don't know, but it can help us get speed.
350
00:13:14.725 --> 00:13:17.485
So we can again, have those really meaningful conversations
351
00:13:17.755 --> 00:13:19.925
with people who do know it and help them get there faster.
352
00:13:20.355 --> 00:13:21.805
This is about us helping the first
353
00:13:21.805 --> 00:13:24.245
and second line be more impactful. That's our job.
354
00:13:24.745 --> 00:13:26.525
You know, I think when you look at the speed
355
00:13:26.525 --> 00:13:28.365
that regulation is coming at us, right?
356
00:13:28.385 --> 00:13:30.045
So whether it's, you know, the cyber rule,
357
00:13:30.045 --> 00:13:31.685
which we've still gotta deal with, right?
358
00:13:31.985 --> 00:13:36.685
The new climate rule, we've got, uh, the European AI rules,
359
00:13:37.005 --> 00:13:38.525
I mean, it just like coming at us,
360
00:13:38.905 --> 00:13:41.805
but, um, you know, being able to use Gen AI to, to
361
00:13:42.325 --> 00:13:43.805
summarize it, get me to the high points.
362
00:13:44.435 --> 00:13:46.205
Some, somebody in our session, uh,
363
00:13:46.205 --> 00:13:48.165
the other day was talking about policies of like,
364
00:13:48.165 --> 00:13:49.405
you can tell it, read a policy
365
00:13:49.465 --> 00:13:52.765
and find all of the, all of the must, must dos, right?
366
00:13:52.765 --> 00:13:54.325
What, like, so then you can come up
367
00:13:54.325 --> 00:13:55.525
with a list so I can use a checklist.
368
00:13:55.745 --> 00:13:58.125
So like, again, just that ability of it
369
00:13:58.185 --> 00:13:59.525
to consume information
370
00:13:59.525 --> 00:14:02.005
and summarize it for you is just, is just awesome.
371
00:14:02.435 --> 00:14:04.405
It's life changing. I mean, it really is.
372
00:14:05.185 --> 00:14:06.565
If I'd known this, I might still be
373
00:14:06.565 --> 00:14:07.925
auditing instead of doing what I do now.
374
00:14:08.595 --> 00:14:09.765
It's, it's making it fun.
375
00:14:10.235 --> 00:14:11.605
Well, thank you Grant and Sue for talking
376
00:14:11.605 --> 00:14:13.245
to us about artificial intelligence
377
00:14:13.245 --> 00:14:14.725
and, uh, it's been very informative.
378
00:14:15.525 --> 00:14:16.725
Terrific. Thanks for having us. Appreciate fun
379
00:14:16.725 --> 00:14:17.725
To be here. Thanks.
380
00:14:17.725 --> 00:14:17.885
381
00:14:19.105 --> 00:14:22.765
If you like this podcast, please subscribe and rate us.
382
00:14:23.385 --> 00:14:26.165
You can subscribe wherever you get your podcasts.
383
00:14:27.065 --> 00:14:31.805
You can also catch each episode on YouTube or@theiia.org.
384
00:14:32.385 --> 00:14:35.685
That's THE iia.org.
0
00:00:02.565 --> 00:00:05.905
The Institute of Internal Auditors presents all things
1
00:00:06.345 --> 00:00:08.345
internal audit AI podcast.
2
00:00:09.645 --> 00:00:13.025
In this episode, Sue King of KPMG
3
00:00:13.405 --> 00:00:16.985
and Grant Osler of Workiva discuss the potential
4
00:00:16.985 --> 00:00:21.265
of generative AI to speed up internal audit processes
5
00:00:21.565 --> 00:00:24.945
and support risk management in evolving risk areas.
6
00:00:25.765 --> 00:00:29.105
Hi, I'm here with Grant from Workiva and Sue from KPMG,
7
00:00:29.245 --> 00:00:31.105
and we're talking about generative ai.
8
00:00:31.735 --> 00:00:33.945
What are some best practices for, uh,
9
00:00:34.425 --> 00:00:36.385
entering prompts into the, uh, into the tool?
10
00:00:36.605 --> 00:00:39.265
That's a great question. So I think, you know, just being
11
00:00:39.405 --> 00:00:41.225
as specific as you can be, right?
12
00:00:41.305 --> 00:00:43.785
I mean, you can, you can write a very, very simple
13
00:00:43.885 --> 00:00:45.665
or open plain language prompt,
14
00:00:45.725 --> 00:00:48.105
but the more specific you can be of like, compare it
15
00:00:48.105 --> 00:00:51.345
to this framework or like specific to, like you were saying,
16
00:00:51.345 --> 00:00:53.865
like specific to audit or like, uh, yeah.
17
00:00:54.005 --> 00:00:56.265
The more specificity, the better an as you can get.
18
00:00:56.385 --> 00:00:57.905
Yeah. The more context you provide,
19
00:00:57.905 --> 00:00:59.345
the better you're gonna get results back.
20
00:00:59.525 --> 00:01:00.665
And one of the things we were talking about
21
00:01:00.665 --> 00:01:02.865
before was we things in preparing for a session,
22
00:01:02.865 --> 00:01:05.225
we talked this, I like to, like,
23
00:01:05.375 --> 00:01:07.825
I'll rewrite the prompt three or four different ways.
24
00:01:08.055 --> 00:01:10.105
I'll tweak it and I'll compare the results back.
25
00:01:10.105 --> 00:01:11.785
Say, oh, okay, these things are common.
26
00:01:11.815 --> 00:01:14.225
Okay, I know that's important. These things are outliers.
27
00:01:14.225 --> 00:01:16.545
Where am I going? And ask for references like,
28
00:01:16.545 --> 00:01:17.665
where did you get this from?
29
00:01:17.965 --> 00:01:19.785
So you can take some of the hallucinations,
30
00:01:20.095 --> 00:01:22.065
it's gonna do it, but I can identify
31
00:01:22.185 --> 00:01:23.225
'em, I can deal with them a lot easier.
32
00:01:23.375 --> 00:01:24.745
Will will it cite its references?
33
00:01:25.165 --> 00:01:26.585
It will if you ask it. Right. That's great.
34
00:01:26.885 --> 00:01:29.385
And it's, and you don't have to, you don't have to be that
35
00:01:29.955 --> 00:01:31.065
articulate in doing it.
36
00:01:31.135 --> 00:01:33.225
Yeah. It fills in the blanks pretty well,
37
00:01:34.005 --> 00:01:35.185
but the more you do it,
38
00:01:35.185 --> 00:01:36.705
just like anything else, the better you get.
39
00:01:37.045 --> 00:01:38.385
So practice a lot. Yeah.
40
00:01:38.405 --> 00:01:40.465
And the more you can be, you know, tell it
41
00:01:40.465 --> 00:01:41.865
to like, okay, that's great.
42
00:01:42.545 --> 00:01:44.825
Summarize it or what, what are the highlights out of this?
43
00:01:44.935 --> 00:01:48.225
Like, to, to really get it to fine tune, fine tune answers
44
00:01:48.225 --> 00:01:50.705
so you don't end up with pages and pages. Right.
45
00:01:51.245 --> 00:01:54.265
And what, what are some of the most common uses
46
00:01:54.265 --> 00:01:56.545
of artificial intelligence by internal auditors?
47
00:01:57.255 --> 00:01:58.545
What, what are the, what are you seeing
48
00:01:58.565 --> 00:01:59.785
out there? Uh, how are people
49
00:02:00.025 --> 00:02:01.025
Adopting? I think what we're seeing a lot in neutral audit,
50
00:02:01.025 --> 00:02:02.145
a lot of,
51
00:02:02.205 --> 00:02:03.625
lot of work on the front end saying, look,
52
00:02:03.735 --> 00:02:04.985
what are the risks in this area?
53
00:02:05.495 --> 00:02:06.785
What are the most important ones?
54
00:02:06.785 --> 00:02:08.305
How are they gonna be, how would,
55
00:02:08.305 --> 00:02:09.825
what controls would I look for in that?
56
00:02:10.015 --> 00:02:11.865
Like what kind of procedures would I do for that?
57
00:02:11.965 --> 00:02:13.505
So a lot in the planning is really big.
58
00:02:13.765 --> 00:02:15.305
We see a lot on the backend.
59
00:02:16.425 --> 00:02:18.785
I don't know about anybody else, but we used to spend a lot
60
00:02:18.785 --> 00:02:20.025
of time wordsmithing reports.
61
00:02:21.045 --> 00:02:23.145
Gen AI writes better than we do most of the time.
62
00:02:23.845 --> 00:02:25.625
Let it take the pass. Then you go into refine it.
63
00:02:25.625 --> 00:02:27.825
It's hard to start with a blank piece of paper, right?
64
00:02:27.825 --> 00:02:28.985
But if I can start with something
65
00:02:29.185 --> 00:02:32.345
that's pretty well crafted, I can then go in and adjust it
66
00:02:32.345 --> 00:02:33.985
and bring it to where I need to be really quickly.
67
00:02:34.485 --> 00:02:37.265
It frees your time up to do more value added audit instead
68
00:02:37.265 --> 00:02:38.785
of spending your time wordsmithing reports.
69
00:02:39.165 --> 00:02:41.985
And how much data entry does it take to, to get it
70
00:02:41.985 --> 00:02:45.625
or, you know, to get it to, uh, write that draft report.
71
00:02:45.695 --> 00:02:49.015
Like how much, uh, prompting do you have to give it?
72
00:02:49.615 --> 00:02:52.015
I, yeah, I don't feel like it really needs that much.
73
00:02:52.285 --> 00:02:53.855
It's like, you know, I've got this exception.
74
00:02:53.855 --> 00:02:56.375
Write me a recommendation, uh, you know,
75
00:02:56.375 --> 00:02:57.575
make it action oriented.
76
00:02:57.645 --> 00:02:58.855
Make it detailed, right.
77
00:02:58.855 --> 00:03:00.165
Write, you can put all of that in there.
78
00:03:00.465 --> 00:03:01.485
Do you want it in bullet point?
79
00:03:01.485 --> 00:03:02.765
You can tell it what you want
80
00:03:02.825 --> 00:03:05.405
and it will give you what you ask for. It's amazing.
81
00:03:05.905 --> 00:03:07.925
So I think that's, that's one of the great use cases,
82
00:03:07.975 --> 00:03:09.605
right, is like the, the planning
83
00:03:09.705 --> 00:03:12.005
and you know, making you smarter about you going to a,
84
00:03:12.085 --> 00:03:14.045
a new area that you don't really know as much about
85
00:03:14.045 --> 00:03:15.725
that's kind of more technical
86
00:03:15.745 --> 00:03:17.925
or, you know, specialized to an industry.
87
00:03:18.035 --> 00:03:20.645
Like make, helps you get smarter in that planning phase.
88
00:03:21.145 --> 00:03:22.445
Um, and then as Grant says,
89
00:03:22.445 --> 00:03:23.845
like gets you to that recommendation,
90
00:03:24.195 --> 00:03:25.325
Does it have the ability yet
91
00:03:25.325 --> 00:03:27.005
to generate the work programs?
92
00:03:27.925 --> 00:03:29.725
Absolutely. It does. It does. Yeah, absolutely it does.
93
00:03:29.985 --> 00:03:31.925
So, so I think, yeah, so doing the planning,
94
00:03:32.025 --> 00:03:33.645
the recommendation, but then I think, you know,
95
00:03:33.925 --> 00:03:36.485
actually doing the testing as well to you can,
96
00:03:36.505 --> 00:03:40.365
you can give it documents to ingest, um, you know, be they,
97
00:03:40.585 --> 00:03:43.045
you know, whatever format, Excel, PDF or whatever.
98
00:03:43.105 --> 00:03:45.725
And you can tell it like, you know, compare and contrast.
99
00:03:45.745 --> 00:03:48.365
So you can start doing some testing procedures to say,
100
00:03:48.705 --> 00:03:51.405
you know, like, okay, like the example that we went
101
00:03:51.405 --> 00:03:52.645
through in our session yesterday is
102
00:03:52.645 --> 00:03:54.285
around terminated user testing, right?
103
00:03:54.425 --> 00:03:57.645
Do I have people who are, who've got active access
104
00:03:57.645 --> 00:03:58.965
that are not employees?
105
00:03:59.545 --> 00:04:02.285
But then the beauty of AI is that then you can say like,
106
00:04:02.285 --> 00:04:03.525
okay, well once I found people
107
00:04:03.525 --> 00:04:05.965
that should have been terminated, then you know,
108
00:04:05.965 --> 00:04:08.005
you can give it another data source and say, well go
109
00:04:08.005 --> 00:04:10.165
and look and see did that person actually do anything
110
00:04:10.165 --> 00:04:11.325
after their termination date?
111
00:04:11.585 --> 00:04:13.085
Or like you can give it the tickets
112
00:04:13.225 --> 00:04:16.485
and say, was the ticket issued for that terminated user
113
00:04:16.865 --> 00:04:18.325
and it just didn't get executed on.
114
00:04:18.345 --> 00:04:21.645
So you can start getting into a lot more of those, um, kind
115
00:04:21.645 --> 00:04:24.325
of what if and look back type analysis very easily.
116
00:04:25.045 --> 00:04:27.245
'cause there's been been a focus for a long time of like,
117
00:04:27.245 --> 00:04:29.165
how do we get to the point where internal auditors
118
00:04:29.165 --> 00:04:30.645
or value added, right?
119
00:04:30.785 --> 00:04:34.165
And so I think really being able to go to a control owner
120
00:04:34.165 --> 00:04:36.245
and not only say like, oh, you've got an exception,
121
00:04:36.245 --> 00:04:37.685
which okay, they hate hearing that.
122
00:04:38.105 --> 00:04:40.405
But then if you've been able to get to the next level
123
00:04:40.505 --> 00:04:42.845
and say, well, I've done root cause analysis,
124
00:04:42.865 --> 00:04:45.445
the issue is like, the tickets are getting issued,
125
00:04:45.445 --> 00:04:46.965
but they're not getting closed out properly.
126
00:04:47.545 --> 00:04:51.565
Or, um, or even saying like, Hey, I did this analysis.
127
00:04:51.785 --> 00:04:54.685
If you first line were to, were to take that activity,
128
00:04:55.155 --> 00:04:56.605
this would be your control activity
129
00:04:56.605 --> 00:04:58.445
and here's how we would, how we would resolve it.
130
00:04:58.465 --> 00:05:01.205
So I think getting, again, by getting back to that,
131
00:05:01.265 --> 00:05:05.245
you know, action oriented value added to the the first line,
132
00:05:05.925 --> 00:05:07.125
I think that's a huge point.
133
00:05:07.255 --> 00:05:10.485
Being able to shift from assurance to advisory
134
00:05:11.025 --> 00:05:13.365
and help the first and second line do their job better.
135
00:05:14.215 --> 00:05:17.565
We're able to amplify the impact we have. It's huge.
136
00:05:17.945 --> 00:05:20.285
And so the more we can help this move upstream,
137
00:05:20.625 --> 00:05:22.125
the more impactful we are as auditors.
138
00:05:22.125 --> 00:05:23.885
What, what are some of the most creative uses
139
00:05:23.885 --> 00:05:26.525
that you've seen of, of AI by auditors?
140
00:05:26.545 --> 00:05:29.365
You know, how, uh, have you seen anything where, you know,
141
00:05:29.365 --> 00:05:31.605
you're surprised at, uh, at what it could do
142
00:05:31.605 --> 00:05:32.605
Or? Well, I think there's, I
143
00:05:32.605 --> 00:05:33.245
think there's a lot
144
00:05:33.245 --> 00:05:34.405
of different things, right?
145
00:05:34.505 --> 00:05:37.325
So we're, you know, we're experimenting with using,
146
00:05:37.975 --> 00:05:42.325
using copilot for example, to, uh, to document, uh,
147
00:05:42.435 --> 00:05:43.725
from a walkthrough that we're having.
148
00:05:44.345 --> 00:05:46.525
And then, you know, it can create your first draft
149
00:05:46.525 --> 00:05:48.965
of a walkthrough and then you can say like, okay,
150
00:05:49.065 --> 00:05:50.565
now compare the walkthrough of
151
00:05:50.565 --> 00:05:52.165
what the process owner just described,
152
00:05:52.265 --> 00:05:53.685
the, the process to me.
153
00:05:53.915 --> 00:05:56.925
Compare that to the process narrative or to the rackum
154
00:05:57.105 --> 00:05:59.685
or to the flowchart and tell me why there are differences.
155
00:06:00.265 --> 00:06:03.605
And, and you know, the, the beauty of it is the speed
156
00:06:03.835 --> 00:06:07.205
that then you can go back to the control owner, you know,
157
00:06:07.205 --> 00:06:09.285
the next day and say, Hey, when we talked about this,
158
00:06:09.465 --> 00:06:10.925
you never, you didn't touch on this.
159
00:06:11.355 --> 00:06:14.605
Like, was that we just omitted it or has it changed? Right?
160
00:06:14.945 --> 00:06:17.725
As opposed to, you know, historically it might have taken,
161
00:06:17.865 --> 00:06:19.965
you know, a week or to get back to it.
162
00:06:19.985 --> 00:06:21.605
And then the, the control owner's like, oh,
163
00:06:21.605 --> 00:06:22.645
I don't remember what we talked about.
164
00:06:22.785 --> 00:06:25.845
So, so, you know, some of it is, uh, as Grant said,
165
00:06:25.845 --> 00:06:29.525
you know, kind of takes, takes some of the, uh, kind of more
166
00:06:29.525 --> 00:06:31.205
of the grunt work kind of out of this, right?
167
00:06:31.305 --> 00:06:34.165
So that we are more focused on, on value add. Yeah.
168
00:06:34.265 --> 00:06:36.405
Um, but the speed thing is, is also terrific.
169
00:06:37.165 --> 00:06:38.805
I think that point about taking some of that,
170
00:06:38.805 --> 00:06:42.085
that busy monotonous workout, every auditor I know,
171
00:06:42.085 --> 00:06:43.685
every audit leader is trying
172
00:06:43.685 --> 00:06:45.085
to staff their team with the right people.
173
00:06:45.385 --> 00:06:46.525
And it's a struggle right now.
174
00:06:46.985 --> 00:06:48.965
If you can get your people working on the right things.
175
00:06:49.245 --> 00:06:51.885
A they like their job more. Nobody likes right?
176
00:06:51.885 --> 00:06:54.365
Narratives, I don't, maybe there is, but I don't know.
177
00:06:54.405 --> 00:06:55.885
'em, they never work for me, right?
178
00:06:56.305 --> 00:06:58.325
So take the stuff they don't like to do,
179
00:06:58.845 --> 00:06:59.925
automate as much of that as you can.
180
00:07:00.305 --> 00:07:02.005
Let them focus on things that adds more
181
00:07:02.005 --> 00:07:03.165
value, that is more fun for them.
182
00:07:03.505 --> 00:07:06.325
Our retention will get better, our ability to keep them to,
183
00:07:06.325 --> 00:07:07.725
and move those people into higher value
184
00:07:07.725 --> 00:07:08.925
roles in the organization gets better.
185
00:07:09.185 --> 00:07:11.925
So it's, it's kinda that virtuous cycle is my opinion.
186
00:07:12.945 --> 00:07:15.525
And what do you see as like the next frontier for, uh,
187
00:07:15.685 --> 00:07:16.925
internal audits use of ai?
188
00:07:16.925 --> 00:07:18.605
Right? Where, where are we going next?
189
00:07:20.165 --> 00:07:23.135
Well, I just, you know, I think going back to
190
00:07:23.135 --> 00:07:24.495
what I was saying earlier, right?
191
00:07:24.575 --> 00:07:27.215
I feel like we've all been stuck, you know, doing SOCs or,
192
00:07:27.315 --> 00:07:29.895
or doing some of the kind of like the standard audits.
193
00:07:30.315 --> 00:07:33.495
Um, you know, but I think this tool really enables us to,
194
00:07:34.075 --> 00:07:37.175
to speed up and again, like really get into that value added
195
00:07:37.355 --> 00:07:40.095
and, and to be able to use that knowledge to say like, okay,
196
00:07:40.095 --> 00:07:42.055
something that maybe I might've needed
197
00:07:42.175 --> 00:07:43.255
a specialist to help me.
198
00:07:43.255 --> 00:07:45.855
Like, you know, you've got much more of a, a of a jumpstart.
199
00:07:45.855 --> 00:07:49.895
But I do think that whole like value added piece, um, and,
200
00:07:49.915 --> 00:07:53.215
and more insight being able to, to really dig into that, uh,
201
00:07:53.215 --> 00:07:54.735
that root, root cause analysis
202
00:07:54.735 --> 00:07:56.655
that we've all been talking about for a long time.
203
00:07:57.815 --> 00:08:00.255
I think one of the challenges auditors have is writing
204
00:08:01.675 --> 00:08:03.955
a real business case in a recommendation, right?
205
00:08:03.955 --> 00:08:05.435
Really make the sale. And,
206
00:08:05.435 --> 00:08:06.635
and we don't always do that very well.
207
00:08:07.535 --> 00:08:09.915
You can ask Gen I to write it in a tone
208
00:08:10.005 --> 00:08:13.435
where it will do a much better job that'll drive adoption.
209
00:08:14.335 --> 00:08:15.515
If, if things don't change.
210
00:08:15.535 --> 00:08:16.675
We write up a finding we're
211
00:08:16.675 --> 00:08:17.955
of no value, let's just be honest.
212
00:08:18.605 --> 00:08:20.515
We're of value when the operation gets better
213
00:08:20.515 --> 00:08:22.235
and they change and they have less risk
214
00:08:22.395 --> 00:08:23.915
'cause they've got better controls or other things in place.
215
00:08:24.255 --> 00:08:26.075
And I think gen I can help in that a lot
216
00:08:26.135 --> 00:08:27.515
and it's gonna keep getting better.
217
00:08:28.975 --> 00:08:32.355
Can Gene AI make recommendations at this point
218
00:08:32.455 --> 00:08:34.875
and uh, you know, like beyond just a high level generic,
219
00:08:34.895 --> 00:08:36.115
can they, you know, can,
220
00:08:36.175 --> 00:08:37.475
can it really generate some
221
00:08:37.525 --> 00:08:39.195
insightful recommendations? Yeah.
222
00:08:39.195 --> 00:08:41.035
Yeah. We've seen it. We've seen it do that. Yeah.
223
00:08:41.455 --> 00:08:43.875
At, at a detailed control, a control level, right?
224
00:08:43.915 --> 00:08:45.155
Yeah. Yeah. For sure Isn't,
225
00:08:45.735 --> 00:08:47.195
But it's not on autopilot, right?
226
00:08:47.335 --> 00:08:50.195
You still want the professional who has experience to say,
227
00:08:50.585 --> 00:08:51.595
does this make sense?
228
00:08:52.095 --> 00:08:54.235
Is that right? Based on what we saw, right?
229
00:08:54.235 --> 00:08:57.125
Because it, it, Charles King who is mm-hmm.
230
00:08:57.205 --> 00:09:00.445
From KPMG who was on our group yesterday was ready, is like,
231
00:09:00.635 --> 00:09:01.965
it's a huge algorithm, right?
232
00:09:01.965 --> 00:09:03.045
It's, it's doing math.
233
00:09:03.825 --> 00:09:06.445
You need to make sure that, that it's still on point, right?
234
00:09:06.865 --> 00:09:09.925
So I think that, again, I don't see this replacing auditors.
235
00:09:10.045 --> 00:09:12.805
I see this in letting auditors be a whole lot more
236
00:09:12.805 --> 00:09:14.045
important, a lot more impactful,
237
00:09:14.335 --> 00:09:15.335
Right? Yeah. Yeah. I,
238
00:09:15.335 --> 00:09:16.645
I, I totally agree. Right?
239
00:09:16.675 --> 00:09:17.765
It's like that first version,
240
00:09:17.865 --> 00:09:20.805
but we all know it's way easier to edit something than it is
241
00:09:20.805 --> 00:09:22.405
to be staring at a blank sheet of paper. Yeah.
242
00:09:22.455 --> 00:09:24.765
White tape are scary, Right? Pretty
243
00:09:24.765 --> 00:09:25.765
Sure. Uh, are
244
00:09:25.765 --> 00:09:29.205
there ways to connect, uh, an AI platform
245
00:09:29.265 --> 00:09:33.125
to your enterprise systems so that the AI can be monitoring
246
00:09:33.225 --> 00:09:35.045
or analyzing the, the data?
247
00:09:35.045 --> 00:09:38.245
Yeah, so, and I can speak to Workiva solution,
248
00:09:38.245 --> 00:09:39.765
I can't speak to anybody else, but you know,
249
00:09:39.825 --> 00:09:42.165
our system is designed with APIs that connect other systems
250
00:09:42.185 --> 00:09:43.805
and bring the data in and, and
251
00:09:43.805 --> 00:09:46.485
because the gen AI is right native in our application,
252
00:09:46.525 --> 00:09:48.365
I can go right to that data and I think it's there.
253
00:09:48.745 --> 00:09:53.045
So yeah, it's really there, um, understanding
254
00:09:53.155 --> 00:09:55.525
what data you need and getting the right connections in.
255
00:09:55.525 --> 00:09:56.605
If I don't have a connection
256
00:09:56.605 --> 00:09:58.165
and maybe it's not something I'm gonna do repetitively,
257
00:09:58.565 --> 00:09:59.525
I can bring it in in the form
258
00:09:59.525 --> 00:10:00.725
of a spreadsheet or something like that.
259
00:10:00.725 --> 00:10:03.125
So it's amazingly simple actually
260
00:10:03.125 --> 00:10:04.485
to bring the data in and connect up
261
00:10:04.485 --> 00:10:05.845
to it. Yeah. Which is really fun.
262
00:10:06.555 --> 00:10:07.605
Yeah. So
263
00:10:08.045 --> 00:10:10.525
Ultimately, how do you see, uh, internal audit
264
00:10:11.075 --> 00:10:13.685
incorporating artificial intelligence into its work,
265
00:10:13.905 --> 00:10:15.085
you know, five years from now?
266
00:10:15.105 --> 00:10:17.525
How, how would, what would you envision, you know,
267
00:10:17.525 --> 00:10:19.445
in internal audit's use of AI to be?
268
00:10:19.445 --> 00:10:21.125
Well, it's interesting, we were talking about it, right?
269
00:10:21.145 --> 00:10:23.045
You know, like things are moving so quickly.
270
00:10:23.225 --> 00:10:25.885
So five years, by five years from now, like, gosh,
271
00:10:25.905 --> 00:10:28.405
we might be on the moon, but, um, you know,
272
00:10:28.405 --> 00:10:31.205
but I, I do think like this, this, you know,
273
00:10:31.205 --> 00:10:32.845
there's been this push, like how do we,
274
00:10:32.905 --> 00:10:34.845
how do we use more technology
275
00:10:34.845 --> 00:10:37.485
and more automation in the way that we're auditing,
276
00:10:37.485 --> 00:10:39.445
whether it's internal audit or whether it's sox.
277
00:10:39.865 --> 00:10:41.965
And I think this has really given us that, you know,
278
00:10:41.965 --> 00:10:44.205
that low code, no code solution Mm-hmm.
279
00:10:44.425 --> 00:10:46.045
To be able to start doing analysis.
280
00:10:46.305 --> 00:10:48.445
You know, I think, you know, like is it the,
281
00:10:48.825 --> 00:10:52.285
is it the perfect solution if you wanna do continuous
282
00:10:52.285 --> 00:10:54.005
monitoring or whatever, maybe not.
283
00:10:54.305 --> 00:10:57.525
But what it allows you to do is to do that, you know, like,
284
00:10:57.795 --> 00:10:59.765
okay, well I, I did it using gen AI
285
00:10:59.765 --> 00:11:01.645
and I proved that like, this is gonna be valuable
286
00:11:01.645 --> 00:11:02.885
and useful, so now let's go
287
00:11:02.885 --> 00:11:05.445
and figure out like, what is the right technology solution.
288
00:11:05.665 --> 00:11:08.365
So I, I just think, uh, um, yeah, it,
289
00:11:08.395 --> 00:11:09.965
it's got a myriad of possibilities.
290
00:11:10.545 --> 00:11:12.245
Um, it's not gonna, as Grant says,
291
00:11:12.265 --> 00:11:13.685
not gonna replace auditors,
292
00:11:13.745 --> 00:11:16.965
but really allows us a, again, to be much more
293
00:11:17.565 --> 00:11:18.725
valuable for the first line.
294
00:11:18.865 --> 00:11:20.245
Um, which I think is terrific. Yeah.
295
00:11:20.515 --> 00:11:23.845
Yeah. I mean it's, it's what, 15 or 16 months old now?
296
00:11:23.865 --> 00:11:25.445
And we're in a whole different world,
297
00:11:25.825 --> 00:11:27.565
so I don't, I'm not that good.
298
00:11:27.605 --> 00:11:28.965
A prognosticator say five years.
299
00:11:29.065 --> 00:11:30.685
I'm not sure I know what a year from now looks like.
300
00:11:31.265 --> 00:11:35.245
But, but what I see is that as people get in
301
00:11:35.245 --> 00:11:38.045
and use it, they're learning ai,
302
00:11:38.045 --> 00:11:39.325
like everything else comes with risks.
303
00:11:39.945 --> 00:11:42.325
And if you're not in it using it, your audit team,
304
00:11:42.465 --> 00:11:44.085
you're not understanding those risks well enough
305
00:11:44.085 --> 00:11:45.285
to really advise your company
306
00:11:45.705 --> 00:11:46.765
and you're not helping your company.
307
00:11:47.245 --> 00:11:48.805
I can guarantee you somebody in marketing
308
00:11:48.805 --> 00:11:49.805
or somewhere else is using it.
309
00:11:50.345 --> 00:11:52.685
And if you're not helping put the right governance in place
310
00:11:52.685 --> 00:11:54.380
around this, you're not doing your
311
00:11:54.380 --> 00:11:55.445
job as an auditor in my opinion.
312
00:11:55.695 --> 00:11:58.525
Right? So you've gotta be well enough versed to have
313
00:11:58.525 --> 00:12:00.005
that conversation with the other people,
314
00:12:00.105 --> 00:12:01.725
to put things in place, have those
315
00:12:01.725 --> 00:12:02.765
right guardrails in place.
316
00:12:03.065 --> 00:12:05.005
And I think that's maybe the next step for us.
317
00:12:05.005 --> 00:12:06.725
I don't know how many, three or four steps, I dunno what
318
00:12:06.725 --> 00:12:09.125
that looks like, but I think right now it's get in,
319
00:12:09.125 --> 00:12:10.885
get familiar, use it practice
320
00:12:10.885 --> 00:12:12.965
and use it so that you can really sit down
321
00:12:12.965 --> 00:12:15.605
and have an intelligent conversation with your business.
322
00:12:15.605 --> 00:12:16.925
Say, look, these are risks.
323
00:12:17.225 --> 00:12:19.965
Here's how we're addressing them. What are you doing?
324
00:12:20.465 --> 00:12:22.205
How do we do this? And really put things in place.
325
00:12:22.345 --> 00:12:23.485
By the way, I'll help you write the policy
326
00:12:24.225 --> 00:12:26.205
And then just, uh, one thing that, that sparks.
327
00:12:26.205 --> 00:12:27.845
So the other thing we haven't talked about, right?
328
00:12:27.845 --> 00:12:29.485
We're talking about using gen ai,
329
00:12:29.785 --> 00:12:32.565
but I think as auditors, exactly as Grant says, you have
330
00:12:32.565 --> 00:12:35.165
to be familiar with it because, you know, the rest
331
00:12:35.165 --> 00:12:37.325
of the business is going to be using it.
332
00:12:37.345 --> 00:12:38.845
And so we need to look at it, we need
333
00:12:38.845 --> 00:12:40.845
to come at it from a controls perspective as well, right?
334
00:12:40.845 --> 00:12:42.005
And say like, okay,
335
00:12:42.005 --> 00:12:43.405
what's the governance we've got over this?
336
00:12:43.415 --> 00:12:45.205
Let's make sure that we're managing it.
337
00:12:45.745 --> 00:12:48.645
Are there any other, uh, you know, things that you know,
338
00:12:48.645 --> 00:12:50.885
you're most excited about in, in AI
339
00:12:50.885 --> 00:12:53.165
and like, you know, bringing it to your clients
340
00:12:53.305 --> 00:12:54.925
or incorporating it into your platform.
341
00:12:54.925 --> 00:12:57.005
What, what, what, what's the, uh,
342
00:12:57.705 --> 00:12:58.765
you know, the, the next target.
343
00:12:58.945 --> 00:13:01.605
Nobody knows better about chasing down data
344
00:13:01.605 --> 00:13:03.085
to make sure it's complete and accurate than auditors.
345
00:13:03.085 --> 00:13:04.125
That's what we do every day.
346
00:13:04.705 --> 00:13:08.485
And so I think there's frontiers in partnering
347
00:13:08.485 --> 00:13:10.525
with the business and all these things in these new emerging
348
00:13:10.595 --> 00:13:11.845
risk areas, right?
349
00:13:12.135 --> 00:13:14.725
Where we don't know, but it can help us get speed.
350
00:13:14.725 --> 00:13:17.485
So we can again, have those really meaningful conversations
351
00:13:17.755 --> 00:13:19.925
with people who do know it and help them get there faster.
352
00:13:20.355 --> 00:13:21.805
This is about us helping the first
353
00:13:21.805 --> 00:13:24.245
and second line be more impactful. That's our job.
354
00:13:24.745 --> 00:13:26.525
You know, I think when you look at the speed
355
00:13:26.525 --> 00:13:28.365
that regulation is coming at us, right?
356
00:13:28.385 --> 00:13:30.045
So whether it's, you know, the cyber rule,
357
00:13:30.045 --> 00:13:31.685
which we've still gotta deal with, right?
358
00:13:31.985 --> 00:13:36.685
The new climate rule, we've got, uh, the European AI rules,
359
00:13:37.005 --> 00:13:38.525
I mean, it just like coming at us,
360
00:13:38.905 --> 00:13:41.805
but, um, you know, being able to use Gen AI to, to
361
00:13:42.325 --> 00:13:43.805
summarize it, get me to the high points.
362
00:13:44.435 --> 00:13:46.205
Some, somebody in our session, uh,
363
00:13:46.205 --> 00:13:48.165
the other day was talking about policies of like,
364
00:13:48.165 --> 00:13:49.405
you can tell it, read a policy
365
00:13:49.465 --> 00:13:52.765
and find all of the, all of the must, must dos, right?
366
00:13:52.765 --> 00:13:54.325
What, like, so then you can come up
367
00:13:54.325 --> 00:13:55.525
with a list so I can use a checklist.
368
00:13:55.745 --> 00:13:58.125
So like, again, just that ability of it
369
00:13:58.185 --> 00:13:59.525
to consume information
370
00:13:59.525 --> 00:14:02.005
and summarize it for you is just, is just awesome.
371
00:14:02.435 --> 00:14:04.405
It's life changing. I mean, it really is.
372
00:14:05.185 --> 00:14:06.565
If I'd known this, I might still be
373
00:14:06.565 --> 00:14:07.925
auditing instead of doing what I do now.
374
00:14:08.595 --> 00:14:09.765
It's, it's making it fun.
375
00:14:10.235 --> 00:14:11.605
Well, thank you Grant and Sue for talking
376
00:14:11.605 --> 00:14:13.245
to us about artificial intelligence
377
00:14:13.245 --> 00:14:14.725
and, uh, it's been very informative.
378
00:14:15.525 --> 00:14:16.725
Terrific. Thanks for having us. Appreciate fun
379
00:14:16.725 --> 00:14:17.725
To be here. Thanks.
380
00:14:17.725 --> 00:14:17.885
381
00:14:19.105 --> 00:14:22.765
If you like this podcast, please subscribe and rate us.
382
00:14:23.385 --> 00:14:26.165
You can subscribe wherever you get your podcasts.
383
00:14:27.065 --> 00:14:31.805
You can also catch each episode on YouTube or@theiia.org.
384
00:14:32.385 --> 00:14:35.685
That's THE iia.org.
