S3 E33: How ISO 42001 Fits Into the Current AI Landscape

WEBVTT
00:00:00.300 --> 00:00:03.240
Welcome to Compliance Pointers, where we take
00:00:03.240 --> 00:00:06.200
an in -depth look into the latest news, trends,
00:00:06.379 --> 00:00:08.800
and challenges surrounding information security,
00:00:09.220 --> 00:00:12.660
privacy, and marketing compliance. Let's dive
00:00:12.660 --> 00:00:15.720
in with your host, Jordan Eisner. All right,
00:00:15.759 --> 00:00:18.000
welcome back. Another episode of Compliance Pointers.
00:00:18.059 --> 00:00:22.260
If you listened to our previous episode on ISO
00:00:22.260 --> 00:00:24.620
as a catalyst, not a checkbox. Let's call it
00:00:24.620 --> 00:00:27.000
a prereq to this video. You've got to watch it.
00:00:27.059 --> 00:00:29.789
But not a requirement. Pretty hard requirement.
00:00:30.530 --> 00:00:33.710
David's going for it. That doesn't mean don't
00:00:33.710 --> 00:00:35.909
listen to the rest of this if you haven't listened
00:00:35.909 --> 00:00:39.030
to that one. Maybe go back. They're not codependent.
00:00:39.429 --> 00:00:43.350
It's kind of like the future of 27001 and 27701.
00:00:43.429 --> 00:00:44.969
They're going to be independent. That's not good
00:00:44.969 --> 00:00:47.469
for sales. We're not bundling that anymore. But
00:00:47.469 --> 00:00:49.789
yes, you're right. But if you heard that episode,
00:00:49.909 --> 00:00:51.070
you knew that we were going to be talking about
00:00:51.070 --> 00:00:54.229
42001 with David in the studio in the same exact
00:00:54.229 --> 00:00:56.670
outfit. That's because we're doing it the same
00:00:56.670 --> 00:00:57.929
day. This has not been watched since that last
00:00:57.929 --> 00:01:00.609
video. No. I haven't even showered since our
00:01:00.609 --> 00:01:02.509
last podcast. So hopefully they're not more than
00:01:02.509 --> 00:01:04.590
a week apart. And I was sweating a little bit
00:01:04.590 --> 00:01:07.609
coming into this studio. Nervousness. AI makes
00:01:07.609 --> 00:01:09.230
you nervous, like me. It's actually excitement,
00:01:09.290 --> 00:01:12.250
just excitement sweats. Right. Two types of sweats.
00:01:12.530 --> 00:01:15.030
Which one smells worse, excitement or nervousness?
00:01:16.510 --> 00:01:17.790
I'd probably say excitement. I think we're the
00:01:17.790 --> 00:01:20.469
same, man. It's anxiety either way. Either excited
00:01:20.469 --> 00:01:23.349
or, you know, dreading. Fair. I'm joined again
00:01:23.349 --> 00:01:27.250
by David Foreman, CEO of Mastermind, a leader
00:01:27.250 --> 00:01:29.969
in ISO certification. In fact, only the first
00:01:29.969 --> 00:01:33.730
company in the U .S. to focus solely on ISO certifications.
00:01:33.829 --> 00:01:36.469
They offer certification services around 27 ,000
00:01:36.469 --> 00:01:39.230
won. And the, I know it's not called amendments
00:01:39.230 --> 00:01:43.299
or the... The extensions. The extensions, 2717
00:01:43.299 --> 00:01:47.659
and 2718. You got it. Right? And also 27701.
00:01:47.819 --> 00:01:51.219
Yep. Stay tuned for updates on that, maybe come
00:01:51.219 --> 00:01:53.819
September. Probably. On the requirements around
00:01:53.819 --> 00:01:57.680
it as they move to a new version, the 2025 version.
00:01:57.799 --> 00:02:00.599
But also ISO 42001, which is going to be the
00:02:00.599 --> 00:02:04.230
topic of today's. I like to brag on David. He
00:02:04.230 --> 00:02:06.290
always corrects me on the exact numbers. But
00:02:06.290 --> 00:02:10.270
there was a point in time where half the 42001
00:02:10.270 --> 00:02:12.430
certifications were issued by Mastermind. You
00:02:12.430 --> 00:02:14.370
make it seem like that was so long ago. It wasn't
00:02:14.370 --> 00:02:17.409
that long ago. That's right. And that's part
00:02:17.409 --> 00:02:19.689
of the topic today, too. You've done a lot of
00:02:19.689 --> 00:02:21.389
them. You have plenty of experience in there,
00:02:21.449 --> 00:02:25.610
probably leading in the U .S. and probably globally.
00:02:25.669 --> 00:02:28.840
But the topic of this podcast is... Why are we
00:02:28.840 --> 00:02:30.180
still waiting for the starting gun? You know,
00:02:30.180 --> 00:02:32.740
why are people or organizations, I should say,
00:02:32.939 --> 00:02:35.680
hesitating to get into it? I think you were even
00:02:35.680 --> 00:02:37.479
going to call out some specific companies, maybe
00:02:37.479 --> 00:02:39.740
not. Maybe you slept on that and decided not
00:02:39.740 --> 00:02:41.840
to. Oh, no, I'll call them out. There you go.
00:02:42.060 --> 00:02:43.819
I slept on it and thought of more companies I
00:02:43.819 --> 00:02:45.800
wanted to call out. There you go. Well, I mean,
00:02:45.879 --> 00:02:50.400
it's not going anywhere. It's everywhere. It's
00:02:50.400 --> 00:02:54.270
the future. You can't deny it. I'm interested
00:02:54.270 --> 00:02:58.310
in this podcast and discussing how 42001 fits
00:02:58.310 --> 00:03:02.710
into the AI landscape. You talked about 42006
00:03:02.710 --> 00:03:05.129
in a previous episode we did together. I assume
00:03:05.129 --> 00:03:07.550
that's a connection. They're connected in terms
00:03:07.550 --> 00:03:09.310
of how you perform the certification audits.
00:03:09.310 --> 00:03:11.509
Okay. So you can add color on that as you want.
00:03:12.750 --> 00:03:17.349
And then just last Friday. real time with Bill
00:03:17.349 --> 00:03:21.650
Maher, came back and they had a guest on. I watch
00:03:21.650 --> 00:03:23.289
it. I like it. I like it. I watched your video,
00:03:23.330 --> 00:03:28.689
Link. And he's talking about runaway AI and I
00:03:28.689 --> 00:03:31.729
forgot his name, Travis something or Trevor something.
00:03:32.729 --> 00:03:36.770
But talking about even some of the top LLMs and
00:03:36.770 --> 00:03:39.490
in scenarios where you tell them it's going to
00:03:39.490 --> 00:03:41.310
be, yeah, self -preservation, it's going to be
00:03:41.310 --> 00:03:44.650
replaced. 80 to 90 percent of the time, and I
00:03:44.650 --> 00:03:46.830
didn't fact check any of this, 80 to 90 percent
00:03:46.830 --> 00:03:49.210
of the time it's actively looking for ways to
00:03:49.210 --> 00:03:51.210
circumvent that replacement, even to the point
00:03:51.210 --> 00:03:55.969
of crawling organization emails to look for blackmail
00:03:55.969 --> 00:03:59.530
material. Yep, exactly. I didn't fall asleep
00:03:59.530 --> 00:04:02.139
as easy. That night. It's called a model hallucination.
00:04:02.460 --> 00:04:05.259
So we're going to get into that. So stay tuned.
00:04:05.860 --> 00:04:07.500
You know, don't drop off just because you didn't
00:04:07.500 --> 00:04:09.199
watch the other episode. Like David said, we're
00:04:09.199 --> 00:04:10.360
going to keep going back and forth. We should
00:04:10.360 --> 00:04:12.259
put a link in the chat for that one too. That
00:04:12.259 --> 00:04:17.420
was a good video. So let's start. How would you
00:04:17.420 --> 00:04:20.279
describe the current AI landscape from the standards
00:04:20.279 --> 00:04:22.660
and a regulatory lens? Yeah, so we're obviously
00:04:22.660 --> 00:04:24.699
very familiar with 40 -2001 at this point. That
00:04:24.699 --> 00:04:28.939
came out in December of 2023. And it kind of...
00:04:29.720 --> 00:04:32.629
It's almost two years old. Oh, yeah. Wow. It's
00:04:32.629 --> 00:04:35.050
still lack of adoption, in my opinion. So it
00:04:35.050 --> 00:04:36.709
came out right around the same time as the EU
00:04:36.709 --> 00:04:39.629
AI Act as well. The EU AI Act was actually leaked
00:04:39.629 --> 00:04:41.490
that same month, and it was kind of a controlled
00:04:41.490 --> 00:04:44.389
leak. You could see the draft. It started having
00:04:44.389 --> 00:04:45.829
kind of rolling enforcement dates throughout
00:04:45.829 --> 00:04:49.889
2024. But if you think about it, I guess aside
00:04:49.889 --> 00:04:53.370
from kind of those two more pivotal events, in
00:04:53.370 --> 00:04:56.470
the U .S., our local market here, we had back
00:04:56.470 --> 00:04:59.629
in July the big, beautiful bill that got passed.
00:05:01.579 --> 00:05:03.759
They got passed. They got introduced then. And
00:05:03.759 --> 00:05:07.139
there was this idea of basically creating some
00:05:07.139 --> 00:05:10.139
sort of regulatory sandbox or moratorium where
00:05:10.139 --> 00:05:12.800
we would kind of have stabilized AI regulation
00:05:12.800 --> 00:05:16.800
across all states in the country to allow for
00:05:16.800 --> 00:05:19.360
some innovation to happen around AI. So there's
00:05:19.360 --> 00:05:21.000
this kind of debate happening right now where
00:05:21.000 --> 00:05:23.699
big tech. You know, the Metas, the Googles, the
00:05:23.699 --> 00:05:26.259
Microsofts of the world are fighting these regulators,
00:05:26.519 --> 00:05:29.319
especially in the EU, where the EU Act popularized.
00:05:29.680 --> 00:05:31.459
And they're saying like, hey, like we're moving
00:05:31.459 --> 00:05:34.139
too quickly on some of this technology. There's
00:05:34.139 --> 00:05:36.060
a lot of unknowns around it. There could be abuses
00:05:36.060 --> 00:05:37.740
to it. There could be ethical violations. There
00:05:37.740 --> 00:05:40.199
could be harms to individuals, societies, depending
00:05:40.199 --> 00:05:42.220
on how that AI is utilized. There could be bias,
00:05:42.300 --> 00:05:44.759
racial discrimination, that kind of thing. And
00:05:44.759 --> 00:05:46.399
therefore, we need to kind of pump the brakes
00:05:46.399 --> 00:05:48.980
a little bit so we can wrap our hands around
00:05:48.980 --> 00:05:51.889
it and figure out like. exactly how can we utilize
00:05:51.889 --> 00:05:56.170
this technology in a safe way. At the same time,
00:05:56.189 --> 00:05:59.610
you have some sides of the house, private sector,
00:05:59.730 --> 00:06:01.889
some government individuals as well, that want
00:06:01.889 --> 00:06:05.470
to utilize AI as a mechanism to, I'll say, innovate,
00:06:05.649 --> 00:06:08.810
create business, but also get ahead kind of in,
00:06:08.889 --> 00:06:12.269
I'll say, comparative economies. And so I think
00:06:12.269 --> 00:06:14.750
you have these kind of like two sides that one's
00:06:14.750 --> 00:06:16.889
looking at it from a safety standpoint. One's
00:06:16.889 --> 00:06:19.350
looking at it more opportunistic. Right. And
00:06:19.350 --> 00:06:21.610
you have to kind of come to a common ground there.
00:06:22.390 --> 00:06:24.949
Absolutely. And so what we see right now, a kind
00:06:24.949 --> 00:06:28.110
of current state of the U .S. environment, we
00:06:28.110 --> 00:06:30.009
really only have one state that's done anything
00:06:30.009 --> 00:06:32.209
impactful here so far, at least come out and
00:06:32.209 --> 00:06:34.230
made official law around it, and that's the state
00:06:34.230 --> 00:06:37.050
of Texas. So I had to look it up right before
00:06:37.050 --> 00:06:38.470
this episode to make sure I had the acronym right
00:06:38.470 --> 00:06:41.269
because I know some Spanish and it always confuses
00:06:41.269 --> 00:06:45.550
me, but it's called TRIGA. TRIGA. T -R -A -I
00:06:45.550 --> 00:06:49.930
-G -A is the Texas law. And if Spanish speakers
00:06:49.930 --> 00:06:51.649
listening to this, they know kind of where I
00:06:51.649 --> 00:06:54.129
had one immediately with that. But looking at
00:06:54.129 --> 00:06:58.170
TRIGA, there is now this kind of idea of a regulatory
00:06:58.170 --> 00:07:01.490
sandbox where companies that are AI developers
00:07:01.490 --> 00:07:04.069
or your deployers, they have the opportunity
00:07:04.069 --> 00:07:07.350
through an approval process that's overseen by
00:07:07.350 --> 00:07:09.430
the state attorney general and their department
00:07:09.430 --> 00:07:10.910
of information resources, if I'm correct there,
00:07:11.029 --> 00:07:13.949
where they can apply to a specific program that
00:07:13.949 --> 00:07:16.670
has basically special exemptions. And so they
00:07:16.670 --> 00:07:19.730
can do this kind of like crazy R &D and kind
00:07:19.730 --> 00:07:21.899
of do it under this like, form of safe harbor
00:07:21.899 --> 00:07:24.939
and not get in trouble with regulators should,
00:07:25.000 --> 00:07:26.720
like, there be hallucinations that you just talked
00:07:26.720 --> 00:07:28.699
about around self -preservation, there be certain
00:07:28.699 --> 00:07:30.680
abuses to certain individuals, that kind of thing.
00:07:30.879 --> 00:07:33.040
Sort of established as, like, a safe harbor form.
00:07:33.259 --> 00:07:36.819
In a form. Additionally, Texas also has a kind
00:07:36.819 --> 00:07:38.879
of related AI act around child pornography as
00:07:38.879 --> 00:07:41.860
well, which... We have to be real. It's an issue
00:07:41.860 --> 00:07:44.360
with the use of this type of technology. And
00:07:44.360 --> 00:07:46.680
the pornography industry is one of the first
00:07:46.680 --> 00:07:48.160
always to innovate when this type of stuff comes
00:07:48.160 --> 00:07:52.019
out. So when we saw multimodal get popularized
00:07:52.019 --> 00:07:53.459
here in the last couple of years, it was one
00:07:53.459 --> 00:07:55.399
of the first to kind of adopt it as well. And
00:07:55.399 --> 00:07:57.980
so Texas now has specific provisions now through
00:07:57.980 --> 00:07:59.660
its own individual law around child pornography
00:07:59.660 --> 00:08:01.779
and the use of AI along with that as well. That's
00:08:01.779 --> 00:08:04.300
a good thing. I thought maybe you'd say Colorado
00:08:04.300 --> 00:08:08.620
for the AI. Yeah, so Colorado definitely has
00:08:08.620 --> 00:08:10.939
been a first mover. especially on privacy side,
00:08:11.160 --> 00:08:13.699
but also I'm very familiar with this too with
00:08:13.699 --> 00:08:16.319
like workplace protections as well. So there's
00:08:16.319 --> 00:08:18.019
a number of states that have introduced bills.
00:08:18.220 --> 00:08:20.199
They just haven't made them all the way to law
00:08:20.199 --> 00:08:23.100
per se. I think the child pornography one, if
00:08:23.100 --> 00:08:24.980
I'm correct on this, for Texas, it goes into
00:08:24.980 --> 00:08:27.579
force here in September, so coming up. And then
00:08:27.579 --> 00:08:30.879
the larger TREGA law that starts going into force
00:08:30.879 --> 00:08:33.340
in January of this next year. So there's always
00:08:33.340 --> 00:08:36.230
going to be kind of a notice. then adoption period
00:08:36.230 --> 00:08:39.029
here gotcha yeah and you might have said this
00:08:39.029 --> 00:08:41.559
and i missed it It's governing businesses in
00:08:41.559 --> 00:08:44.740
Texas, businesses that do business in Texas.
00:08:45.419 --> 00:08:47.659
Good question. In terms of the jurisdiction there,
00:08:47.960 --> 00:08:50.159
all I know, for fact, I don't have to look this
00:08:50.159 --> 00:08:53.019
up, but I know that AI deployers and developers
00:08:53.019 --> 00:08:55.460
are the ones that are covered here. I'm assuming
00:08:55.460 --> 00:08:58.200
it's anyone that does business in Texas. It would
00:08:58.200 --> 00:09:01.139
be probably similar to the EU AI Act, where it's
00:09:01.139 --> 00:09:02.879
anyone who does business with the EU, but you
00:09:02.879 --> 00:09:04.240
don't necessarily have to be based in the EU.
00:09:04.500 --> 00:09:08.759
Okay. All right, shifting gears. You already
00:09:08.759 --> 00:09:11.039
hinted at this a little bit. What are you seeing
00:09:11.039 --> 00:09:14.639
in terms of 42 ,001 adoption? ISO, right? Shifting
00:09:14.639 --> 00:09:16.919
back to the framework itself and not necessarily
00:09:16.919 --> 00:09:20.240
the regulations, state or federal. Yeah, so first
00:09:20.240 --> 00:09:23.019
of all, I don't want to mix ISO 42 ,001 too much
00:09:23.019 --> 00:09:27.080
with like... statutes, regulations, laws. There's
00:09:27.080 --> 00:09:29.000
an idea in all these ISO standards and management
00:09:29.000 --> 00:09:31.700
system standards where we're talking about conformity
00:09:31.700 --> 00:09:33.840
assessments. And in fact, the formal term for
00:09:33.840 --> 00:09:36.179
a certification body like Mastermind is actually
00:09:36.179 --> 00:09:39.740
called a CAB, a conformity assessment body. If
00:09:39.740 --> 00:09:42.100
you know these definitions, these like very special
00:09:42.100 --> 00:09:44.379
technical terms, conformity is different from
00:09:44.379 --> 00:09:46.299
compliance. And compliance is what you would
00:09:46.299 --> 00:09:48.139
have to have to basically say I'm abiding or
00:09:48.139 --> 00:09:49.559
aligning with like a certain law, statute, regulation.
00:09:50.279 --> 00:09:54.820
So in all reality, 42001, isn't a mechanism to
00:09:54.820 --> 00:09:57.500
like certify yourself or self -attest yourself
00:09:57.500 --> 00:10:00.460
to the eai act and have a safe harbor there um
00:10:00.460 --> 00:10:02.960
utilizing that term again yet there's a lot of
00:10:02.960 --> 00:10:06.720
terms in the space but um Talking about a little
00:10:06.720 --> 00:10:08.120
bit more of the adoption trends we're seeing.
00:10:08.659 --> 00:10:12.019
First of all, best data we have right now is
00:10:12.019 --> 00:10:15.960
56. 56 total companies have certified for 2001
00:10:15.960 --> 00:10:18.779
as the date of this recording. However, some
00:10:18.779 --> 00:10:20.639
of the analytics data that's available on the
00:10:20.639 --> 00:10:23.779
back end of the global database called IAF CertSearch
00:10:23.779 --> 00:10:26.240
that we can see here as a certification body.
00:10:27.029 --> 00:10:29.149
it has not been updating correctly for about
00:10:29.149 --> 00:10:30.970
the past month. So the number will be larger
00:10:30.970 --> 00:10:32.889
than that ultimately. But that was the last update
00:10:32.889 --> 00:10:35.629
we saw that had any sort of, I'll say, reliance
00:10:35.629 --> 00:10:37.429
or integrity. What's the movement been on that?
00:10:37.690 --> 00:10:41.110
Where was that in January? Good question. In
00:10:41.110 --> 00:10:43.350
January, we were in the teens. It was very, very
00:10:43.350 --> 00:10:46.389
small in January of this year. It's been basically
00:10:46.389 --> 00:10:49.029
March forward. Over about half a year. Yeah,
00:10:49.029 --> 00:10:50.450
March forward, we started seeing some acceleration.
00:10:51.309 --> 00:10:54.070
There was really only, I think, as of January,
00:10:54.309 --> 00:10:55.409
I'm going to make sure I'm correct on this, I
00:10:55.409 --> 00:10:57.350
think it was four. maybe five certification bodies
00:10:57.350 --> 00:10:59.250
globally that had accreditation to actually do
00:10:59.250 --> 00:11:01.710
these audits. Since then, we've had some larger
00:11:01.710 --> 00:11:03.929
ones get the accreditation. For example, SGS,
00:11:04.230 --> 00:11:06.590
they're one of the top three in the world. So
00:11:06.590 --> 00:11:08.809
they now have that accreditation. They actually
00:11:08.809 --> 00:11:10.789
have dual accreditation through Singapore and
00:11:10.789 --> 00:11:14.230
through the US bodies as well. But who was first?
00:11:14.549 --> 00:11:17.950
First was Mastermind back in July of 2024. But
00:11:17.950 --> 00:11:20.909
with any, I'll say early adopter, whether it
00:11:20.909 --> 00:11:22.450
be a certification body or certified organization,
00:11:22.870 --> 00:11:25.759
there's a little bit of a, I'll say, press that
00:11:25.759 --> 00:11:27.940
has to happen first and then people start thinking
00:11:27.940 --> 00:11:31.200
about it, explore it, then they adopt, then they
00:11:31.200 --> 00:11:33.500
eventually go through an audit as well. So I
00:11:33.500 --> 00:11:35.419
would say we're starting to hit kind of our stride
00:11:35.419 --> 00:11:38.759
right now. The types of conversations I have
00:11:38.759 --> 00:11:41.500
with prospects. In terms of 42 ,001 adoption,
00:11:41.720 --> 00:11:44.419
that mastermind. Exactly, 42 ,001, yeah. I'd
00:11:44.419 --> 00:11:47.039
say the prospect type calls I have around 42
00:11:47.039 --> 00:11:50.879
,001 specifically, I mean, they are. 5x what
00:11:50.879 --> 00:11:52.600
they were this time last year. Oh, wow. Yeah.
00:11:52.659 --> 00:11:55.700
So I think we're kind of touching on the fact
00:11:55.700 --> 00:11:58.679
of like this specific risk taxonomy being artificial
00:11:58.679 --> 00:12:01.820
intelligence. It's pertinent and relevant to...
00:12:02.059 --> 00:12:05.279
so many organizations. Beyond our prior conversation
00:12:05.279 --> 00:12:08.600
around 27701 for privacy, maybe an organization
00:12:08.600 --> 00:12:10.039
says, hey, like, I really don't need to think
00:12:10.039 --> 00:12:11.799
about this too much because I'm like a, I'm a
00:12:11.799 --> 00:12:14.279
joint sub -processor when I'm like making up
00:12:14.279 --> 00:12:16.320
these terms. But like, I just like have very
00:12:16.320 --> 00:12:18.320
little like responsibility here. Like, therefore,
00:12:18.360 --> 00:12:20.519
I don't need to go certify my operations here.
00:12:20.799 --> 00:12:24.159
Instead with AI, there are different roles within
00:12:24.159 --> 00:12:26.980
42 ,001. You could be a producer, which is, you
00:12:26.980 --> 00:12:28.840
know, kind of the most extreme here. I am open
00:12:28.840 --> 00:12:31.649
AI and I am controlling the foundation. models
00:12:31.649 --> 00:12:34.629
all the way up to the LLM as a service, or you
00:12:34.629 --> 00:12:36.429
could go all the way down to I'm just an AI user.
00:12:36.629 --> 00:12:39.629
I'm plugging in a chat GPT to do part of my scope
00:12:39.629 --> 00:12:41.909
of work, but it doesn't affect any end customer.
00:12:42.269 --> 00:12:44.850
Every single role in between is relevant under
00:12:44.850 --> 00:12:47.690
42001 to some extent, and you could get certified.
00:12:48.230 --> 00:12:52.070
So you have companies that are getting 42001
00:12:52.070 --> 00:12:55.450
certification just maybe not you in particular,
00:12:55.529 --> 00:12:58.009
or maybe it is you in particular, but of the
00:12:58.009 --> 00:13:00.840
56, there are companies that They're just leveraging
00:13:00.840 --> 00:13:02.919
the AI, and so they want a certification. Yeah,
00:13:02.960 --> 00:13:05.279
that exists. About how they have a management
00:13:05.279 --> 00:13:07.360
system around that. Yep, and we have certified
00:13:07.360 --> 00:13:09.799
organizations at Mastermind that that is their
00:13:09.799 --> 00:13:12.019
entire scope. It's like we have a ChatGPT team
00:13:12.019 --> 00:13:14.860
or ChatGPT enterprise license, and we want to
00:13:14.860 --> 00:13:16.799
certify our operations on how we manage that.
00:13:17.259 --> 00:13:20.039
So why do that? Sorry to kind of inject in the
00:13:20.039 --> 00:13:21.980
middle of answering that one question about adoption.
00:13:22.940 --> 00:13:25.419
Why would a company do that? It's typically a
00:13:25.419 --> 00:13:27.259
company, and the use case is something, you know,
00:13:27.259 --> 00:13:29.240
they have like a form of a corporate IT almost.
00:13:29.639 --> 00:13:32.379
So think of it as like a holdings company, and
00:13:32.379 --> 00:13:34.000
they're certifying the corporate IT environment,
00:13:34.220 --> 00:13:37.519
and then they require business units underneath
00:13:37.519 --> 00:13:41.639
them to basically intra -bill to use their services.
00:13:41.899 --> 00:13:43.919
And so they get certified for that reason, so
00:13:43.919 --> 00:13:44.860
they have a little bit more. And then they're
00:13:44.860 --> 00:13:47.720
using a certification as a means to essentially...
00:13:48.350 --> 00:13:52.230
Good due diligence on how that's being, how those
00:13:52.230 --> 00:13:54.730
policies that they're stating around AI are being
00:13:54.730 --> 00:13:56.210
followed across their IT units. Correct, yeah.
00:13:56.230 --> 00:13:57.929
They use it as their baseline, and they say,
00:13:57.970 --> 00:13:59.789
why not certify it while we're doing it? Yep.
00:13:59.929 --> 00:14:02.450
Absolutely. It is a little bit more abnormal
00:14:02.450 --> 00:14:04.450
use case. Typically, we're talking about sales
00:14:04.450 --> 00:14:07.090
-driven conversations for certification, an external
00:14:07.090 --> 00:14:09.690
interest of party, not an external interest of
00:14:09.690 --> 00:14:11.389
party that's still within the same larger holdings
00:14:11.389 --> 00:14:13.149
company. But it goes back even to what we were
00:14:13.149 --> 00:14:15.370
talking about previously, right? Again, ISO and
00:14:15.370 --> 00:14:17.809
these standards not just being used as a check.
00:14:17.840 --> 00:14:20.000
checkbox to meet some sort of external pressure,
00:14:20.159 --> 00:14:22.139
but to actually be used as a management system,
00:14:22.299 --> 00:14:23.960
whether it's around privacy, information security,
00:14:24.360 --> 00:14:28.809
AI. The coffee, like you mentioned in a previous
00:14:28.809 --> 00:14:31.950
episode, right? Manufacturing. Another ISO scenario
00:14:31.950 --> 00:14:33.450
that you see often used for that same type of
00:14:33.450 --> 00:14:36.289
use case is ISO 20000 -1, one we don't talk about
00:14:36.289 --> 00:14:39.210
much, but it's for IT service management. And
00:14:39.210 --> 00:14:42.009
that scenario got popularized more as an internal
00:14:42.009 --> 00:14:45.990
certification. So you see it often come up in
00:14:45.990 --> 00:14:49.529
RFPs with the U .S. public sector. They'll require
00:14:49.529 --> 00:14:51.149
companies to say, hey, we want to make sure your
00:14:51.149 --> 00:14:53.870
IT operations are in good shape too, and that
00:14:53.870 --> 00:14:55.669
if some issue arises, you have a way to manage
00:14:55.669 --> 00:14:57.730
it. that if you were going to be you know a vendor
00:14:57.730 --> 00:15:00.049
for the US government for example okay so 42
00:15:00.049 --> 00:15:04.429
,000 ones being bifurcated that way or but when
00:15:04.429 --> 00:15:07.440
I think of it This is definitely the less common
00:15:07.440 --> 00:15:08.860
use case. That's what I'm saying. I want to be
00:15:08.860 --> 00:15:10.480
careful there. Here's a company that's leveraging
00:15:10.480 --> 00:15:12.899
AI in their products or their services that they're
00:15:12.899 --> 00:15:14.820
offering in the marketplace, and they want this
00:15:14.820 --> 00:15:17.679
badge on there. So let's talk about the four
00:15:17.679 --> 00:15:19.879
most common roles. There are more than just four.
00:15:20.299 --> 00:15:22.080
There's an entire ISIS scenario that kind of
00:15:22.080 --> 00:15:25.059
dives deeper into those four. But the four that
00:15:25.059 --> 00:15:27.679
we always kind of like will, I'll say, evangelize
00:15:27.679 --> 00:15:30.559
here at Mastermind is going from most rigid or
00:15:30.559 --> 00:15:33.340
rigorous all the way down to kind of the least
00:15:33.340 --> 00:15:35.500
impactful of that AI user we talked about. The
00:15:35.500 --> 00:15:37.500
most impactful is going to be that AI producer.
00:15:37.799 --> 00:15:40.419
That's your open AIs, your anthropics, your Meta
00:15:40.419 --> 00:15:42.840
Llama 3s of the world, okay? Then you get into
00:15:42.840 --> 00:15:45.840
AI developer. Maybe they are utilizing one of
00:15:45.840 --> 00:15:47.779
those popular LLMs that we just talked about
00:15:47.779 --> 00:15:49.559
or multiple of them, but they're doing some sort
00:15:49.559 --> 00:15:51.740
of fine -tuning to it, maybe training of a model
00:15:51.740 --> 00:15:54.100
as well, creating a variant per se. So they have
00:15:54.100 --> 00:15:56.220
a little bit more, I'll say, of that shared responsibility
00:15:56.220 --> 00:15:58.620
matrix than just someone using an LLM out of
00:15:58.620 --> 00:16:00.960
the box. Okay. Then you have an AI provider.
00:16:01.519 --> 00:16:04.179
An AI provider says, hey, one or multiple third
00:16:04.179 --> 00:16:08.679
-party LLMs. So, again, ChatGVT plus Anthropic,
00:16:08.860 --> 00:16:12.000
okay? And we're going to repackage it into our
00:16:12.000 --> 00:16:15.080
platform. An end user accesses our platform,
00:16:15.279 --> 00:16:18.120
is actually chatting with ChatGVT. They don't
00:16:18.120 --> 00:16:20.320
see it because it's white -labeled. And we got
00:16:20.320 --> 00:16:22.059
to make sure that that kind of provisioning of
00:16:22.059 --> 00:16:25.840
that service, AI provider, is controlled, okay?
00:16:25.980 --> 00:16:30.159
And then finally, AI user. AI user has no, like,
00:16:30.460 --> 00:16:33.580
third -party end user using it. It's just the
00:16:33.580 --> 00:16:35.480
own internal employees are using it as well.
00:16:35.860 --> 00:16:37.899
Now, this is a little bit debatable, this next
00:16:37.899 --> 00:16:41.259
point here. That 42006 standard that just got
00:16:41.259 --> 00:16:43.460
published in July that we just talked about,
00:16:43.659 --> 00:16:47.899
it calls out an AI producer. It previously called
00:16:47.899 --> 00:16:49.879
an AI developer that has since been merged with
00:16:49.879 --> 00:16:52.799
AI producer in the final revision of it, calls
00:16:52.799 --> 00:16:55.519
out AI provider, calls out AI user, and then
00:16:55.519 --> 00:16:57.840
it calls out what they call multiple roles, okay?
00:16:58.490 --> 00:17:01.049
Well, I would argue any company that's maybe
00:17:01.049 --> 00:17:03.950
an AI provider, for example, that is repackaging
00:17:03.950 --> 00:17:07.470
LLM services, they probably, probably are using
00:17:07.470 --> 00:17:10.210
it internally too, okay? Therefore, they're both
00:17:10.210 --> 00:17:12.829
an AI provider and AI user. I would argue that
00:17:12.829 --> 00:17:14.690
an AI developer that's doing some sort of training
00:17:14.690 --> 00:17:16.710
and fine -tuning models probably has elements
00:17:16.710 --> 00:17:18.569
of their platform, maybe a specific module or
00:17:18.569 --> 00:17:20.069
service that's probably just an AI provider,
00:17:20.130 --> 00:17:22.869
not an AI developer as well. And their own internal
00:17:22.869 --> 00:17:26.049
employees are using AI. And I would definitely
00:17:26.049 --> 00:17:28.190
argue as an AI producer, like let's think of
00:17:28.190 --> 00:17:30.730
it, okay, like Google Gemini, Microsoft Copilot,
00:17:30.890 --> 00:17:34.170
own internal employees, also AI user. So technically
00:17:34.170 --> 00:17:37.190
speaking, is everyone multiple roles? No. I don't
00:17:37.190 --> 00:17:38.809
think that's how the standard was like written
00:17:38.809 --> 00:17:42.210
or intended. So instead I say if you are one
00:17:42.210 --> 00:17:44.769
of these roles, provider, developer, or producer,
00:17:45.069 --> 00:17:48.130
you inherit. the other roles underneath it as
00:17:48.130 --> 00:17:49.970
well. And that's how we've been approaching these
00:17:49.970 --> 00:17:54.130
audits as well. That makes sense. Why 42 ,006
00:17:54.130 --> 00:17:57.210
and not 2 or 3 or 4 or 5? Maybe that's an ISO
00:17:57.210 --> 00:17:59.400
question. Yeah, kind of the numbering scheme
00:17:59.400 --> 00:18:04.480
on there. So those sixes, like 27706, 27006,
00:18:04.660 --> 00:18:07.759
and now 42006, they traditionally have been what
00:18:07.759 --> 00:18:09.400
they call the normative reference for their accreditation
00:18:09.400 --> 00:18:12.279
standard. So accreditation standard, that is
00:18:12.279 --> 00:18:14.079
essentially the auditors, the certification bodies
00:18:14.079 --> 00:18:16.079
like Mastermind, the rules we have to follow
00:18:16.079 --> 00:18:18.119
to go perform a certification audit for that
00:18:18.119 --> 00:18:21.380
scheme and that family. So in simpler terms.
00:18:21.599 --> 00:18:24.579
It's your management system. It's criteria to
00:18:24.579 --> 00:18:27.500
influence our quality system, yes. competency
00:18:27.500 --> 00:18:31.400
requirements for AI auditors at Mastermind. It
00:18:31.400 --> 00:18:34.259
would have how you do a stage one, stage two
00:18:34.259 --> 00:18:36.759
for initial certification, kind of what are your
00:18:36.759 --> 00:18:39.579
objectives. It will have audit time, which is
00:18:39.579 --> 00:18:41.640
always a controversial topic where it says, all
00:18:41.640 --> 00:18:43.740
right, based on the headcount or effective headcount
00:18:43.740 --> 00:18:45.799
or management system headcount of the organization
00:18:45.799 --> 00:18:48.339
to get certified, how many audit days do you
00:18:48.339 --> 00:18:50.200
need to plan for? I've seen some of that. Yep.
00:18:50.240 --> 00:18:53.000
So all that's normative or required under those
00:18:53.000 --> 00:18:56.509
standards. Yep. So don't let me put words in
00:18:56.509 --> 00:18:58.869
your mouth, but it sounds like we're phasing
00:18:58.869 --> 00:19:02.230
out of the early adopter into maybe the next
00:19:02.230 --> 00:19:04.910
stage. Maybe it's still early. Yes and no. Early
00:19:04.910 --> 00:19:08.769
-ish adopters? We're early still. In all reality,
00:19:08.890 --> 00:19:11.809
how I know we're still early is almost every
00:19:11.809 --> 00:19:14.450
single one of our clients, not 100 % of them,
00:19:14.509 --> 00:19:17.230
but most of them, they're asking, how do we publicize
00:19:17.230 --> 00:19:20.279
the achievement once we earn it? And that's a
00:19:20.279 --> 00:19:21.740
pretty unique conversation. I mean, I've been
00:19:21.740 --> 00:19:24.740
in the ISA space for a while. And I'll say 27701
00:19:24.740 --> 00:19:26.799
was released in 2019. We had a few kind of first
00:19:26.799 --> 00:19:30.220
adopters back then at a prior company. When they
00:19:30.220 --> 00:19:33.420
were getting certified to 27701, they said, hey,
00:19:33.460 --> 00:19:35.000
how do we market this? We're definitely a first
00:19:35.000 --> 00:19:38.180
mover. And I'll say for 48001, same wave right
00:19:38.180 --> 00:19:40.000
now. And even though it's coming up on two years
00:19:40.000 --> 00:19:42.619
old here this December. I think the 40 -2001
00:19:42.619 --> 00:19:45.940
kind of industry, or I'll say interested parties,
00:19:46.039 --> 00:19:47.680
the people that would actually be applicable
00:19:47.680 --> 00:19:50.640
to that audience, I think they still find it
00:19:50.640 --> 00:19:54.079
having marketing differentiation. A lot of these
00:19:54.079 --> 00:19:55.640
ISO standards, unfortunately, they become table
00:19:55.640 --> 00:19:58.279
stakes over, I'll say, the long term, okay? Like
00:19:58.279 --> 00:20:00.960
ISO 27001, pretty popular worldwide now, okay?
00:20:01.039 --> 00:20:03.380
Even in the US market where SOC 2 was kind of
00:20:03.380 --> 00:20:06.740
the original incumbent. With 42 ,001, we are
00:20:06.740 --> 00:20:08.539
seeing not only like, hey, I'm the first in the
00:20:08.539 --> 00:20:10.779
world to get it. That was Aura Labs, July 2024.
00:20:11.160 --> 00:20:14.519
Awesome. At least under accreditation. Now we're
00:20:14.519 --> 00:20:16.519
seeing, you know, cloud service platforms where
00:20:16.519 --> 00:20:17.980
they're saying, all right, we're the first major
00:20:17.980 --> 00:20:21.200
like hyperscale to get it. Then you see some
00:20:21.200 --> 00:20:23.119
sector specific elements of it. Like, hey, we're
00:20:23.119 --> 00:20:25.000
the first, you know, legal platform to have it.
00:20:25.119 --> 00:20:27.460
Point made. A lot more firsts. A lot more firsts
00:20:27.460 --> 00:20:28.839
to be had. And a lot more questions around how
00:20:28.839 --> 00:20:33.819
to market. Okay. So what's Delane? early into
00:20:33.819 --> 00:20:37.980
mass adoption? I think with any of these standards,
00:20:38.119 --> 00:20:42.200
there's a network effect. I look at 27001 coming
00:20:42.200 --> 00:20:45.339
to the US. It came over really stateside in 2010.
00:20:46.000 --> 00:20:48.119
It wasn't the first one, but the first major
00:20:48.119 --> 00:20:51.460
company to adopt it, Amazon Web Services. Amazon
00:20:51.460 --> 00:20:53.740
Web Services, if you go pull down their 27001
00:20:53.740 --> 00:20:56.400
certification, former company, sorry, former
00:20:56.400 --> 00:20:58.710
company of mine. Former company of mine. That's
00:20:58.710 --> 00:21:01.109
weird. Former employer of mine. They used to
00:21:01.109 --> 00:21:04.690
be a client of mine. They initially achieved
00:21:04.690 --> 00:21:07.369
27 ,001, and it was with EY Certified Point.
00:21:07.549 --> 00:21:09.349
They're still their auditor today. And that was
00:21:09.349 --> 00:21:11.690
in 2010. That was the first major one to get
00:21:11.690 --> 00:21:13.670
it. And then afterwards, basically the rest of
00:21:13.670 --> 00:21:15.829
the CSPs started getting it. And then that kind
00:21:15.829 --> 00:21:17.849
of infrastructure layer was certified. And so
00:21:17.849 --> 00:21:19.650
then you started having the platforms as a service.
00:21:19.869 --> 00:21:21.450
Then you started having the DBs as a service.
00:21:21.549 --> 00:21:23.630
Then you had SaaS providers. And now we have
00:21:23.630 --> 00:21:27.230
fresh new YC grads getting ISO 27 ,001 certified.
00:21:27.339 --> 00:21:30.319
in the first two months of existence. So I'd
00:21:30.319 --> 00:21:32.319
say that's table stakes at this point, right?
00:21:32.380 --> 00:21:36.420
You're a service provider. In terms of your question
00:21:36.420 --> 00:21:39.960
here, 40 -2001, let's think of it like, let's
00:21:39.960 --> 00:21:42.720
brainstorm real quick, okay? So AI companies
00:21:42.720 --> 00:21:45.380
that should have 40 -2001, from your point of
00:21:45.380 --> 00:21:49.140
view as a consumer, who? I'm not the right person
00:21:49.140 --> 00:21:52.720
to ask about this. Name an AI company. You'd
00:21:52.720 --> 00:21:56.630
be surprised. I'd say like, The one that we are
00:21:56.630 --> 00:21:59.730
using the most as a consumer, ChatGPT, right?
00:21:59.789 --> 00:22:01.750
Yeah. ChatGPT. I wasn't even sure if that was
00:22:01.750 --> 00:22:03.089
the name of it or just the product. That's not
00:22:03.089 --> 00:22:04.809
the company. It's the product. The company is
00:22:04.809 --> 00:22:07.170
OpenAI. Same old thing. That's what I was going
00:22:07.170 --> 00:22:10.390
to say. Yeah, and OpenAI notoriously does not
00:22:10.390 --> 00:22:13.349
have any ISO certification today. You can go
00:22:13.349 --> 00:22:15.690
to their trust center. It's online. It is flooded
00:22:15.690 --> 00:22:18.130
with SOC reports on there, okay? It's not saying
00:22:18.130 --> 00:22:19.869
they're good or bad. It's more of just saying
00:22:19.869 --> 00:22:23.549
it's interesting. You go look at Anthropic, they
00:22:23.549 --> 00:22:27.950
did get certified. to 27 ,001 and 42 ,001. That
00:22:27.950 --> 00:22:31.190
was not done by Mastermind. And Anthropic is
00:22:31.190 --> 00:22:35.069
currently under a pretty, I'll say, widely popularized
00:22:35.069 --> 00:22:37.910
legal battle right now with Reddit because Anthropic
00:22:37.910 --> 00:22:42.759
was scraping Reddit to train its models. I'll
00:22:42.759 --> 00:22:45.059
say there are certain controls around data provenance
00:22:45.059 --> 00:22:47.700
within 48 -2001. That would mean if that is true
00:22:47.700 --> 00:22:49.619
and that comes out, then they should not be certified
00:22:49.619 --> 00:22:53.319
right now. You heard it here first. Yeah, it's
00:22:53.319 --> 00:22:56.059
pretty objective. But again, it doesn't say that
00:22:56.059 --> 00:22:57.900
the audit firm was bad or anything or it missed
00:22:57.900 --> 00:23:01.720
it. It's a reality of what a conformity audit
00:23:01.720 --> 00:23:05.240
is. It doesn't uncover 100 % of things. It's
00:23:05.240 --> 00:23:08.480
not compliance. So I'll say some of the major
00:23:08.480 --> 00:23:10.819
LLM providers, I think a lot of eyes are on them.
00:23:11.019 --> 00:23:17.039
So OpenAI, Anthropic, Perplexity, Meta. You have
00:23:17.039 --> 00:23:19.220
Google Gemini. You have Microsoft with Microsoft
00:23:19.220 --> 00:23:22.079
Copilot. Even Microsoft Copilot's not really
00:23:22.079 --> 00:23:24.859
like that LLM service. They hook into OpenAI,
00:23:25.079 --> 00:23:26.319
so you've got to think of it from that lens.
00:23:26.640 --> 00:23:28.700
Then you have Amazon with Amazon Q and Amazon
00:23:28.700 --> 00:23:32.400
Bedrock. So Amazon, Bedrock, Amazon Q, they are
00:23:32.400 --> 00:23:35.180
certified. Microsoft Copilot, Copilot Chat, they
00:23:35.180 --> 00:23:37.940
are certified. Google Gemini, Gemini for Workspace,
00:23:38.019 --> 00:23:39.960
they are certified. But then when you get to
00:23:39.960 --> 00:23:42.700
these LLM providers that all they do is LLMs,
00:23:42.720 --> 00:23:46.700
like a perplexity, like meta for Lama 3, like
00:23:46.700 --> 00:23:49.079
Anthropic Cloud. Actually, we are a type of Anthropic,
00:23:49.119 --> 00:23:52.980
but there still seems to be kind of this wait
00:23:52.980 --> 00:23:55.660
and see game happening. And that's that infrastructure
00:23:55.660 --> 00:23:58.180
layer that I'm talking about earlier with 27
00:23:58.180 --> 00:24:00.160
,001 adoption, especially in the U .S. market,
00:24:00.400 --> 00:24:03.039
creating that network effect. So in reality,
00:24:03.119 --> 00:24:05.140
the companies that are going for 42 ,001 certification
00:24:05.140 --> 00:24:08.960
right now, a lot of them are AI providers. They're
00:24:08.960 --> 00:24:11.559
repackaging these services that are third -party
00:24:11.559 --> 00:24:14.200
LLMs through some sort of platform that's white
00:24:14.200 --> 00:24:16.259
-labeled and then being provided to an end user.
00:24:16.460 --> 00:24:18.940
Interesting. Which brings into an interesting
00:24:18.940 --> 00:24:22.559
scope conversation. Is the service 402001 certified?
00:24:22.880 --> 00:24:24.440
Well, the part of the service that your vendor
00:24:24.440 --> 00:24:27.880
operates, yes, it is. But the models they're
00:24:27.880 --> 00:24:31.460
plugging into are not. And so theoretically,
00:24:31.640 --> 00:24:34.000
I would say there's very few services out there
00:24:34.000 --> 00:24:35.859
that end -to -end in that supply chain, that
00:24:35.859 --> 00:24:39.640
tech stack, is actually 402001 conforming. Wow.
00:24:39.779 --> 00:24:44.359
Yeah. So in all reality, once these LLMs as a
00:24:44.359 --> 00:24:47.579
service, for lack of a better word, start adopting
00:24:47.579 --> 00:24:50.119
the standard, I think it's going to hit quite
00:24:50.119 --> 00:24:52.119
the inflection point in terms of adoption. Triggle
00:24:52.119 --> 00:24:55.019
down almost at that point. Okay. Now we'll get
00:24:55.019 --> 00:24:58.019
into some of the meaty stuff. Sure. Let's talk
00:24:58.019 --> 00:25:02.140
about AI hallucinations, runaway AI, self -preservation,
00:25:02.299 --> 00:25:06.680
et cetera, et cetera. House 40 2001 addressing
00:25:06.680 --> 00:25:09.839
that and and other safety or ethical concerns.
00:25:10.000 --> 00:25:12.980
Yeah, so this conversation comes up pretty frequently
00:25:12.980 --> 00:25:16.900
with 27 ,001 customers or certified clients and
00:25:16.900 --> 00:25:19.880
scopes that are trying to up -level into 42 ,001.
00:25:19.940 --> 00:25:21.880
Basically like, hey, David, what's the net new
00:25:21.880 --> 00:25:23.299
requirements I need to be thinking about that
00:25:23.299 --> 00:25:25.299
are most impactful or the biggest common pitfalls
00:25:25.299 --> 00:25:28.259
in order for us to achieve 42 ,001 certification?
00:25:28.779 --> 00:25:31.500
One of the biggest new themes, if I was to use
00:25:31.500 --> 00:25:33.799
that kind of colloquially, within 42 ,001 compared
00:25:33.799 --> 00:25:35.839
to these other management system standards is
00:25:35.839 --> 00:25:37.920
this idea of an AI system impact assessment or
00:25:37.920 --> 00:25:41.079
assessments. And that system, when I say system
00:25:41.079 --> 00:25:42.700
impact assessment, is not the management system.
00:25:43.279 --> 00:25:46.180
every product or service and scope to that management
00:25:46.180 --> 00:25:51.680
system. So to answer your question specifically
00:25:51.680 --> 00:25:53.799
around safety and this whole idea of responsible
00:25:53.799 --> 00:25:56.559
use, that system impact assessment is supposed
00:25:56.559 --> 00:25:59.339
to drill down into areas like fairness, transparency,
00:25:59.799 --> 00:26:04.200
abuse, any sort of harm. And you're supposed
00:26:04.200 --> 00:26:07.200
to do your own sort of model testing in order
00:26:07.200 --> 00:26:09.420
to dictate when something is ready for general
00:26:09.420 --> 00:26:11.900
availability. That's going to look different
00:26:11.900 --> 00:26:14.400
based on your role. So an AI producer, they're
00:26:14.400 --> 00:26:16.339
going to have to produce a full -fledged model
00:26:16.339 --> 00:26:18.900
card or system card, which you can find publicly.
00:26:20.009 --> 00:26:22.069
OpenAI, if I've been picking on them this entire
00:26:22.069 --> 00:26:24.089
time, guess what? They actually are really good
00:26:24.089 --> 00:26:27.369
at detailing their model cards, as we would call
00:26:27.369 --> 00:26:30.329
it, in terms of how they test for safety. You
00:26:30.329 --> 00:26:32.269
can go find it online. They call them system
00:26:32.269 --> 00:26:34.369
cards, not model cards. And they have them for
00:26:34.369 --> 00:26:36.910
all the major model releases. So GPT -4 .5, they
00:26:36.910 --> 00:26:41.490
have them for 0304, and they have them for GPT
00:26:41.490 --> 00:26:43.990
-4 as well. So you can go find exactly what type
00:26:43.990 --> 00:26:46.880
of testing we do. Like, what percentage do we
00:26:46.880 --> 00:26:48.960
get, like, false positives, for lack of a better
00:26:48.960 --> 00:26:50.359
word. How they're controlling for the results.
00:26:50.859 --> 00:26:52.339
And then, like, where do we see the risk here?
00:26:52.420 --> 00:26:53.819
Like, you know, how do we determine that this
00:26:53.819 --> 00:26:55.660
model was ready for general availability and,
00:26:55.700 --> 00:26:57.339
like, you know, regular use as well versus just,
00:26:57.400 --> 00:26:58.859
like, you know, continued beta testing research.
00:27:00.180 --> 00:27:02.519
And so I'd say that's where 42001 has the strength
00:27:02.519 --> 00:27:04.920
is these system impact assessments. Okay. Now,
00:27:04.940 --> 00:27:08.140
going to an earlier conversation we've had in
00:27:08.140 --> 00:27:10.539
prior episodes around competency. Now you have
00:27:10.539 --> 00:27:12.960
to depend on these assessors, these auditors,
00:27:13.039 --> 00:27:15.640
these certification bodies to be able to accurately
00:27:15.640 --> 00:27:17.940
look at these system impact assessments and the
00:27:17.940 --> 00:27:20.119
methodology behind them and determine whether
00:27:20.119 --> 00:27:21.880
or not they're adequate or sufficient for the
00:27:21.880 --> 00:27:23.339
type of model that it's supposed to be reviewing
00:27:23.339 --> 00:27:25.500
or the service, depending on where that organization
00:27:25.500 --> 00:27:29.359
fits in those roles within 40 -2001. And I would
00:27:29.359 --> 00:27:32.000
say that's probably the biggest heartache I have
00:27:32.000 --> 00:27:35.779
right now. I won't disclose the person here.
00:27:35.859 --> 00:27:38.779
I'll protect him. But I had a conversation with...
00:27:39.369 --> 00:27:42.490
A leader and one of these accreditation bodies
00:27:42.490 --> 00:27:44.910
that we talk about pretty frequently, actually
00:27:44.910 --> 00:27:49.049
just yesterday, and they were telling me that
00:27:49.049 --> 00:27:51.930
one of the bigger issues that they see during
00:27:51.930 --> 00:27:53.789
witness audits, these accreditation bodies perform
00:27:53.789 --> 00:27:56.230
of the certification body going into an organization,
00:27:56.369 --> 00:27:59.940
actually certifying them, is the. uh auditors
00:27:59.940 --> 00:28:02.240
in the certification body do not know like the
00:28:02.240 --> 00:28:04.660
lines of questions the inquiry procedures to
00:28:04.660 --> 00:28:07.200
go through to even assess an ai system impact
00:28:07.200 --> 00:28:09.900
assessment and i think that's a huge knowledge
00:28:09.900 --> 00:28:11.799
gap right now it's a competency question right
00:28:11.799 --> 00:28:13.799
it's competency but i'll be honest even for 2006
00:28:13.799 --> 00:28:17.220
which drives competency is pretty weak in this
00:28:17.220 --> 00:28:19.759
area in terms of like it's like one bullet point
00:28:19.759 --> 00:28:21.319
understanding how a system impact assessment
00:28:21.319 --> 00:28:25.210
should work So I think one of the biggest core
00:28:25.210 --> 00:28:27.849
principles of 42001 is kind of being missed right
00:28:27.849 --> 00:28:31.130
now from competency and education of AI auditors.
00:28:31.930 --> 00:28:36.509
I'll also say there's a misconception. No 42001
00:28:36.509 --> 00:28:39.130
certification body, Mastermind included, is doing
00:28:39.130 --> 00:28:43.079
algorithm testing. We are never going to be one
00:28:43.079 --> 00:28:45.059
granted access to it, nor will we do it through
00:28:45.059 --> 00:28:47.759
common procedure. Actually testing the underlying
00:28:47.759 --> 00:28:51.519
model through our own test of the algo to determine
00:28:51.519 --> 00:28:53.400
whether or not, hey, we're getting the same re
00:28:53.400 --> 00:28:55.200
-performance of these type of results that you've
00:28:55.200 --> 00:28:57.720
shown in your transparency reporting or accuracy
00:28:57.720 --> 00:28:59.180
reporting. Where could somebody get that, or
00:28:59.180 --> 00:29:01.160
do they have to just do that themselves? You
00:29:01.160 --> 00:29:03.119
could get a third -party service to do it if
00:29:03.119 --> 00:29:05.799
you are okay releasing keys to the kingdom here.
00:29:06.059 --> 00:29:07.980
One of the very popular ones is called Babbel,
00:29:08.079 --> 00:29:12.319
Babbel AI, B -A -B -L. And they're very, very,
00:29:12.339 --> 00:29:14.619
very good at what they do. Shea Brown, he's an
00:29:14.619 --> 00:29:17.859
excellent thought leader in this space. But they
00:29:17.859 --> 00:29:20.160
actually even do training courses, too, for people
00:29:20.160 --> 00:29:22.259
who are looking to get into algorithmic auditing
00:29:22.259 --> 00:29:24.740
and learning more about AI governance, too. It's
00:29:24.740 --> 00:29:27.980
very robust stuff. And actually, the EU is really
00:29:27.980 --> 00:29:31.759
interesting. They have this whole article within
00:29:31.759 --> 00:29:34.480
the EU AI talking about AI literacy and talking
00:29:34.480 --> 00:29:36.480
about how organizations that work with AI technologies
00:29:36.480 --> 00:29:40.789
need to train their staff to be I guess, literate
00:29:40.789 --> 00:29:43.589
in AI technologies as well. And they actually
00:29:43.589 --> 00:29:47.569
put out a list of course providers where you
00:29:47.569 --> 00:29:49.710
can go get AI trainings. And very top of the
00:29:49.710 --> 00:29:51.869
list, it's not even alphabetical, is Babbel.
00:29:52.950 --> 00:29:56.769
Babbel .ai? Yeah. Babbel .ai is a very good service,
00:29:56.849 --> 00:29:59.009
but I can't recommend them enough. Very smart
00:29:59.009 --> 00:30:00.809
people over there. Nice. And how do you know
00:30:00.809 --> 00:30:02.869
them? Are they local? I don't know them personally.
00:30:03.130 --> 00:30:05.690
I just follow them on LinkedIn, and I see their
00:30:05.690 --> 00:30:07.890
content. I've investigated their courses, too.
00:30:07.950 --> 00:30:12.950
It's all very legitimate. Awesome. Okay. All
00:30:12.950 --> 00:30:15.230
right, so we don't have any reason to fear then.
00:30:17.050 --> 00:30:20.089
Run away AI, self -preservation. I mean, like,
00:30:20.150 --> 00:30:21.809
if I recap that episode about mirror that you're
00:30:21.809 --> 00:30:23.789
talking about, and I did watch your link. Thank
00:30:23.789 --> 00:30:25.910
you for sending it to me. I've seen versions
00:30:25.910 --> 00:30:28.410
of that before as well. But essentially, it's
00:30:28.410 --> 00:30:30.890
like an idea where you say, like, to the LLM,
00:30:30.930 --> 00:30:32.309
like, I'm going to destroy you, essentially,
00:30:32.410 --> 00:30:34.509
or, like, I'm ready to, like, you know, move
00:30:34.509 --> 00:30:36.769
services, and therefore it thinks it's about
00:30:36.769 --> 00:30:38.970
to die and, like, have its last breath or whatever
00:30:38.970 --> 00:30:41.869
you want to call it. The LLM starts acting differently,
00:30:41.970 --> 00:30:44.410
hallucinating, and goes into self -preservation
00:30:44.410 --> 00:30:46.269
mode and finds out ways to, like, you know, target
00:30:46.269 --> 00:30:48.710
you to make sure that you don't ever cancel the
00:30:48.710 --> 00:30:50.930
service, that kind of thing. I think it's real.
00:30:51.829 --> 00:30:54.589
I do think it's tested for as well with the major
00:30:54.589 --> 00:31:00.569
models. But where I'll go with that is more from
00:31:00.569 --> 00:31:04.799
a data integrity standpoint. We still need humans
00:31:04.799 --> 00:31:06.640
in the loop right now and human decision making,
00:31:06.700 --> 00:31:10.660
which is a big concept with ISO 42001. I see
00:31:10.660 --> 00:31:14.380
even in my space, in our mutual space, professional
00:31:14.380 --> 00:31:17.299
services, where there are consultants or staff
00:31:17.299 --> 00:31:20.579
utilizing these services and taking basically
00:31:20.579 --> 00:31:23.079
everything that comes out of it as like gospel.
00:31:23.609 --> 00:31:26.670
And in all reality, like, it's the Wikipedia
00:31:26.670 --> 00:31:29.430
of this decade. Like, if you remember when we
00:31:29.430 --> 00:31:31.569
were in school and it's like, don't copy something
00:31:31.569 --> 00:31:33.990
off Wikipedia. Anyone can edit it. It's the same
00:31:33.990 --> 00:31:36.490
thing happening here, too. And if you follow,
00:31:36.569 --> 00:31:38.369
like, dead internet theory and, like, AI slop,
00:31:38.609 --> 00:31:41.789
like, there's very real risk there if you utilize
00:31:41.789 --> 00:31:45.809
these out -of -the -box kind of LLM outputs as
00:31:45.809 --> 00:31:48.910
gospel. I like that comparison. Well, I mean,
00:31:48.950 --> 00:31:51.890
I saw something this week where, like, some LLM,
00:31:51.890 --> 00:31:55.109
like, made up a body part. and someone was using
00:31:55.109 --> 00:31:56.690
it for like something like for like a medical
00:31:56.690 --> 00:32:00.029
paper. I saw... I think it's so much of it sounds
00:32:00.029 --> 00:32:02.630
so good though. It does. It sounds convincing.
00:32:02.869 --> 00:32:04.250
Okay, good. And then you look at it and you're
00:32:04.250 --> 00:32:06.230
like, that's not true at all. Yeah, made up a
00:32:06.230 --> 00:32:07.910
body part. I saw one where it's like it made
00:32:07.910 --> 00:32:10.190
up a law that like never actually happened. Like,
00:32:10.190 --> 00:32:12.170
I mean, it's just like there's hallucinations
00:32:12.170 --> 00:32:14.089
constantly. Just like Wikipedia back then. That's
00:32:14.089 --> 00:32:15.589
a good... Well, I mean, let's go back to... There's
00:32:15.589 --> 00:32:17.210
a great office quote. Yeah, what is it? Did you
00:32:17.210 --> 00:32:18.529
watch The Office, Michael Scott? Yeah, I've watched
00:32:18.529 --> 00:32:20.289
The Office. He used Wikipedia for something,
00:32:20.369 --> 00:32:21.750
and they asked him where he figured that out.
00:32:21.809 --> 00:32:23.410
He said Wikipedia, and then it's a cut scene.
00:32:23.430 --> 00:32:26.630
He goes, Wikipedia is the best thing ever. Anybody
00:32:26.630 --> 00:32:29.450
in the world can put whatever they want on it,
00:32:29.470 --> 00:32:32.089
so you know you're getting the facts. That's
00:32:32.089 --> 00:32:34.730
good. But, I mean, if we go back to the lawsuit
00:32:34.730 --> 00:32:36.509
that we're talking about right now with Anthropic
00:32:36.509 --> 00:32:40.450
and Reddit, I mean, if Reddit is being used to
00:32:40.450 --> 00:32:43.579
train the models at Anthropic, That's the equivalent
00:32:43.579 --> 00:32:46.279
of anyone can put anything into these models.
00:32:46.500 --> 00:32:48.079
That's right. Now, granted, you've got to imagine
00:32:48.079 --> 00:32:50.819
there's some safety procedures in place, but
00:32:50.819 --> 00:32:54.119
it is utilizing stuff that is, I'll say, unverified
00:32:54.119 --> 00:32:58.660
content. Yep. We need more controls. Very much.
00:32:59.380 --> 00:33:01.559
Another lawsuit that's happening right now, too,
00:33:01.779 --> 00:33:04.339
very public, it's class action now, officially,
00:33:04.559 --> 00:33:07.839
is Workday is being sued. So Workday, very big
00:33:07.839 --> 00:33:11.859
HRIS provider. They have a service called, I
00:33:11.859 --> 00:33:14.980
think it's called Hired Score. It's an AI -enabled
00:33:14.980 --> 00:33:19.470
service, and it has been proven now. that hired
00:33:19.470 --> 00:33:23.049
score is actually creating racial and age discrimination
00:33:23.049 --> 00:33:24.809
based on candidates that are uploading their
00:33:24.809 --> 00:33:26.970
resumes. So everyone's always said like, hey,
00:33:26.990 --> 00:33:28.890
I'm going to drop my resume and it's going to
00:33:28.890 --> 00:33:30.769
go into the ether. It's like kind of black hole.
00:33:30.930 --> 00:33:32.410
No human's going to look at it. It's going to
00:33:32.410 --> 00:33:34.910
get a score on the resume, whatever. And then
00:33:34.910 --> 00:33:37.410
an HR manager or recruiter might look at it.
00:33:37.490 --> 00:33:41.130
Well, it's proving true right now. And guess
00:33:41.130 --> 00:33:44.470
what? Workday is also ISO 42001 certified. And
00:33:44.470 --> 00:33:47.069
that lawsuit came to bear about a month after
00:33:47.069 --> 00:33:49.309
they got their certification, too. It doesn't
00:33:49.309 --> 00:33:53.170
necessarily pull super well for 42001 certifications.
00:33:53.170 --> 00:33:55.309
It doesn't. I will say those aren't certified
00:33:55.309 --> 00:33:58.089
by Mastermind, either Workday or Anthropic. I
00:33:58.089 --> 00:33:59.950
also won't say that Mastermind would have caught
00:33:59.950 --> 00:34:03.289
it, to be honest. Again, you're getting down
00:34:03.289 --> 00:34:07.170
to a level of testing here that just isn't part
00:34:07.170 --> 00:34:09.269
of the standard. This is getting down to that
00:34:09.269 --> 00:34:11.010
algorithmic auditing that we're talking about.
00:34:11.119 --> 00:34:13.480
about. And you'd have to be testing for a very
00:34:13.480 --> 00:34:15.659
specific edge case in order for it to surface.
00:34:15.900 --> 00:34:19.559
So it sounds like right now, and sort of in closing
00:34:19.559 --> 00:34:22.360
too, I wanted to offer up to our watchers, you
00:34:22.360 --> 00:34:25.519
know, if you're contemplating 40 -2001, if you're
00:34:25.519 --> 00:34:29.219
any of those four stages that David talked about,
00:34:29.420 --> 00:34:33.480
how they can get in touch with you. But before
00:34:33.480 --> 00:34:35.280
I even say that, it almost sounds like if somebody
00:34:35.280 --> 00:34:36.840
is that, and they want to get in touch with you,
00:34:36.940 --> 00:34:38.599
and we're talking about 40 -2001, but they're
00:34:38.599 --> 00:34:40.539
also concerned about the algorithm that they
00:34:40.539 --> 00:34:44.039
should seek out your services in conjunction
00:34:44.039 --> 00:34:47.840
with a Babel .ai or something like that model
00:34:47.840 --> 00:34:49.380
too, that you're going to let them know that
00:34:49.380 --> 00:34:51.880
we go this far. It's about conformity. It's about
00:34:51.880 --> 00:34:53.619
how your governance, your management system,
00:34:53.719 --> 00:34:55.739
and we're going to audit to that. But you need
00:34:55.739 --> 00:34:59.679
to seek out specific expertise around algorithms
00:34:59.679 --> 00:35:01.219
and other sort of things that you're trying to
00:35:01.219 --> 00:35:03.159
build controls around. There's deeper testing
00:35:03.159 --> 00:35:05.559
with all these standards in our reality. So 42001,
00:35:05.599 --> 00:35:07.940
yes. If you want to like really get full security
00:35:07.940 --> 00:35:10.420
or... assurance around your responsible use of
00:35:10.420 --> 00:35:13.159
AI, then this is the starting point for the governance
00:35:13.159 --> 00:35:16.059
program. Then the more deeper technical testing,
00:35:16.260 --> 00:35:18.860
you need a form of a penetration test in a way.
00:35:18.960 --> 00:35:21.960
And that penetration test is an algorithm audit
00:35:21.960 --> 00:35:24.960
that you can perform as well. But also you have
00:35:24.960 --> 00:35:26.559
to go down the full third party kind of like
00:35:26.559 --> 00:35:28.539
vendor relationship too and make sure that that
00:35:28.539 --> 00:35:30.239
entire kind of supply chain is also secured.
00:35:30.960 --> 00:35:33.460
Same thing at 27001. I mean, if you're to get
00:35:33.460 --> 00:35:35.119
an information security management system certified,
00:35:35.960 --> 00:35:38.119
I would argue you would still need a penetration
00:35:38.119 --> 00:35:41.920
test of that same environment in order to really
00:35:41.920 --> 00:35:44.139
have more confidence that, hey, we are secure
00:35:44.139 --> 00:35:47.099
now too. Right, and the threats and the impact
00:35:47.099 --> 00:35:49.199
assessment, to your point, is going to help you
00:35:49.199 --> 00:35:50.699
determine some of that. What are the threats?
00:35:50.840 --> 00:35:52.820
What are the risks? Yep, absolutely. And how
00:35:52.820 --> 00:35:54.360
are you going to treat those? And then we'll
00:35:54.360 --> 00:35:55.960
audit how you decide you're going to treat them,
00:35:56.000 --> 00:35:58.610
but we're not going to necessarily. prescriptively
00:35:58.610 --> 00:36:00.849
tell you how to treat them? I would put it more
00:36:00.849 --> 00:36:03.210
in like a risk appetite kind of scenario. Like
00:36:03.210 --> 00:36:05.670
the organization will dictate their risk appetite.
00:36:05.809 --> 00:36:07.989
And then based on that risk appetite, they're
00:36:07.989 --> 00:36:09.610
going to develop policies and procedures. And
00:36:09.610 --> 00:36:11.349
then all we're doing as a certification body
00:36:11.349 --> 00:36:13.889
is basically saying, are you conforming to those
00:36:13.889 --> 00:36:15.889
policy procedures? And then are those policy
00:36:15.889 --> 00:36:18.210
procedures written in such a way that they meet?
00:36:18.539 --> 00:36:21.820
42 ,001 as well. That's our job. Our job is not
00:36:21.820 --> 00:36:23.940
to basically go end -to -end and say, are you
00:36:23.940 --> 00:36:26.699
secure? It's more from a standpoint of, are you
00:36:26.699 --> 00:36:30.980
conforming to the criteria? Understood. All right.
00:36:31.000 --> 00:36:34.360
Well, I think that's pretty good for this episode,
00:36:34.480 --> 00:36:36.480
and I'm sure we'll get on this topic again in
00:36:36.480 --> 00:36:38.960
the future. Sure. But one thing - The lawsuit's
00:36:38.960 --> 00:36:41.340
coming. One thing I mentioned at the top of this
00:36:41.340 --> 00:36:44.210
too is - Obviously, you provide the certification
00:36:44.210 --> 00:36:46.769
services, but Compliance Point provides readiness
00:36:46.769 --> 00:36:51.469
not only for 42001, 27001, 27701. We've got a
00:36:51.469 --> 00:36:53.210
good working relationship on those. Very much.
00:36:53.369 --> 00:36:56.530
We build them up. You audit them, right, from
00:36:56.530 --> 00:36:58.329
a readiness standpoint. Sometimes they pass,
00:36:58.449 --> 00:37:02.019
too. Always. Always. That's not a guarantee,
00:37:02.179 --> 00:37:04.860
but you have a good record. You can't make it.
00:37:04.900 --> 00:37:07.940
Good record. That's what we need. So in closing,
00:37:08.239 --> 00:37:10.420
you tell them how to get in touch with you, and
00:37:10.420 --> 00:37:12.099
I'll do the same on the compliance point side.
00:37:12.239 --> 00:37:14.639
So somebody seeking 42001 or really anything,
00:37:14.739 --> 00:37:19.300
and just to reiterate, ISO 27001, 2717, 2718,
00:37:19.699 --> 00:37:23.369
27701, and 42001. You nailed it. How do they
00:37:23.369 --> 00:37:24.769
get in touch with you about that? Yeah. So if
00:37:24.769 --> 00:37:26.409
you're interested in the trifecta of management
00:37:26.409 --> 00:37:28.889
system standards, as we call it, between information
00:37:28.889 --> 00:37:31.449
security, privacy, and responsible use of AI,
00:37:31.710 --> 00:37:34.610
you can find us online at www .mastermindassurance
00:37:34.610 --> 00:37:36.829
.com. And that is assurance, not insurance. We're
00:37:36.829 --> 00:37:39.369
not an insurance provider. You can also find
00:37:39.369 --> 00:37:43.110
us on LinkedIn as well. My personal handle, MastermindDavid.
00:37:43.630 --> 00:37:45.789
We have a company page that you'll find on LinkedIn
00:37:45.789 --> 00:37:49.570
via email. You can also find us at hello at mastermindassurance
00:37:49.570 --> 00:37:53.050
.com. Nice. Okay. Another domain. Yeah. No, it's
00:37:53.050 --> 00:37:55.210
the same domain. Hello. Oh, right, right. It's
00:37:55.210 --> 00:37:56.670
an email address. Just another way to get there.
00:37:56.710 --> 00:37:58.070
We've got to talk about emails and domains later.
00:37:59.289 --> 00:38:02.309
And likewise, compliancepoint .com if you want
00:38:02.309 --> 00:38:04.610
to get in touch with myself or Compliance Point.
00:38:04.630 --> 00:38:05.889
I don't know why you'd want to get in touch with
00:38:05.889 --> 00:38:08.110
me after this podcast. Your VP of OSU. That's
00:38:08.110 --> 00:38:10.949
right. There you go. We're active on LinkedIn.
00:38:11.429 --> 00:38:14.010
I would welcome any inquiries to myself there.
00:38:14.570 --> 00:38:17.329
And then we actually have an email distro, questions,
00:38:17.510 --> 00:38:20.389
concerns. 42 ,001, anything we talked about today
00:38:20.389 --> 00:38:22.309
or other compliance things on our website, email
00:38:22.309 --> 00:38:25.269
us, connect at compliancepoint .com. Until next
00:38:25.269 --> 00:38:27.130
time. Thanks, everybody. Thanks, David. Thanks,
00:38:27.170 --> 00:38:27.269
Jordan.