Generative AI Meets Accessibility: Benchmarks, Breakthroughs, and Blind Spots with Joe Devon

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 01/05/2026 00:51:23
Duration: 3371.857
Channels: 1
1
00:00:05.120 --> 00:00:08.000
Hello, and welcome to the AI Engineering Podcast,
2
00:00:08.000 --> 00:00:12.160
your guide to the fast moving world of building scalable and maintainable AI systems.
3
00:00:17.285 --> 00:00:22.404
Unlock the full potential of your AI workloads with a seamless and composable data infrastructure.
4
00:00:22.725 --> 00:00:30.370
Bruin is an open source framework that streamlines integration from the command line, allowing you to focus on what matters most, building intelligent systems.
5
00:00:30.850 --> 00:00:38.850
Write Python code for your business logic and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement.
6
00:00:39.410 --> 00:00:46.535
With native support for ML and AI workloads, Bruin empowers data teams to deliver faster, more reliable, and scalable AI solutions.
7
00:00:47.094 --> 00:00:53.175
Harness Bruin's connectors for hundreds of platforms, including popular machine learning frameworks like TensorFlow and PyTorch.
8
00:00:53.574 --> 00:01:03.090
Build end to end AI workflows that integrate seamlessly with your existing tech stack. Join the ranks of forward thinking organizations that are revolutionizing their data engineering with Bruin.
9
00:01:03.329 --> 00:01:07.170
Get started today at aiengineeringpodcast.com/bruin.
10
00:01:07.170 --> 00:01:11.490
And for dbt cloud customers, enjoy a $1,000 credit to migrate to Bruin Cloud.
11
00:01:12.565 --> 00:01:20.805
Your host is Tobias Maci, and today I'm interviewing Joe Devon about opportunities for using generative AI to improve the accessibility of digital technologies.
12
00:01:20.805 --> 00:01:22.885
So, Joe, can you start by introducing yourself?
13
00:01:23.445 --> 00:01:27.940
Yes. My name is Joe Devon. I am the co founder of Global Accessibility
14
00:01:27.940 --> 00:01:34.420
Awareness Day, and this is a day that goes viral every year to the tune of at least 200,000,000
15
00:01:34.420 --> 00:01:38.100
people. We stopped counting once we hit that number on social media.
16
00:01:38.500 --> 00:01:41.515
And as a result of this going viral,
17
00:01:41.674 --> 00:01:45.755
I found myself square in the accessibility movement, but I started
18
00:01:46.235 --> 00:01:52.395
out as a developer. I started coding at 13 years old. Right now, I'm chair of the GAD Foundation,
19
00:01:52.475 --> 00:01:59.340
but I did have an interesting history with a variety of different jobs. I worked for a search engine company called Compassware
20
00:01:59.500 --> 00:02:01.979
that predated AltaVista, let alone Google,
21
00:02:02.140 --> 00:02:05.020
was a backend programmer for americanidol.com,
22
00:02:05.180 --> 00:02:12.425
and started my own dev shop and built it to about 100 employees before COVID had some other ideas
23
00:02:12.745 --> 00:02:19.305
for how to exit that company. And I guess last but not least, I'm the chief VIBE coding officer for my family.
24
00:02:19.385 --> 00:02:21.305
So that's where we end up in
25
00:02:23.305 --> 00:02:24.185
2025,
26
00:02:24.185 --> 00:02:24.745
2026.
27
00:02:25.690 --> 00:02:29.290
And do you remember how you first got started working in the AI space?
28
00:02:30.170 --> 00:02:38.090
It depends on what you call AI. I was asked to start a group called the Semweb Group, Semantic Web Group in
29
00:02:38.090 --> 00:02:38.490
LA.
30
00:02:38.965 --> 00:02:41.125
And a lot of the AI researchers
31
00:02:41.525 --> 00:02:42.965
were very disappointed
32
00:02:43.125 --> 00:02:49.605
that artificial intelligence didn't take off all those decades ago. So they kind of moved a lot of them to the Semantic Web space,
33
00:02:49.924 --> 00:02:53.045
which was just as successful, at least in terms of the marketing.
34
00:02:53.710 --> 00:02:54.430
And
35
00:02:54.990 --> 00:03:00.990
so I got to know lots of people in that space. So a lot of people dealing with graph stores and such,
36
00:03:01.310 --> 00:03:02.830
and then really didn't
37
00:03:03.070 --> 00:03:04.750
touch it as I
38
00:03:04.750 --> 00:03:15.905
was friends with the people that ran the Semtech biz group. I was on the board there. And then as that became a little bit less active, kind of forgot about it for a while. And then all of a sudden ChatGPT
39
00:03:15.905 --> 00:03:17.185
comes out with
40
00:03:17.505 --> 00:03:20.065
that version that just goes totally wild.
41
00:03:20.710 --> 00:03:22.550
I found Cursor and
42
00:03:22.710 --> 00:03:31.270
the rest was history. You see, I hadn't touched code for about fifteen years. I'd become one of those people running a company, and I'm not good at the
43
00:03:31.830 --> 00:03:34.710
context switching from dealing with human beings
44
00:03:35.215 --> 00:03:56.220
and being the head janitor, head salesperson, head everything of the company and dealing with people to what's much nicer dealing with code. But Cursor and AI really helped me get back into the code. So now I'm coding morning to night, but it's not really coding. It's just telling the AI what to do and trying to keep keep up basically.
45
00:03:57.900 --> 00:04:04.145
And so you have found yourself very much enmeshed in this space of accessibility.
46
00:04:04.145 --> 00:04:10.465
And for purposes of this conversation, I'm wondering if you can give your definition of what that term encompasses.
47
00:04:11.665 --> 00:04:15.665
Yeah. It's it's just about making technology work for people with disabilities.
48
00:04:16.009 --> 00:04:20.410
And what prompted me to write the blog post that became
49
00:04:20.570 --> 00:04:23.370
Global Accessibility Awareness Day was personal,
50
00:04:23.370 --> 00:04:26.330
which is how most people get into it. My dad was
51
00:04:26.650 --> 00:04:28.009
in his mid-80s
52
00:04:28.265 --> 00:04:31.305
and he couldn't bank anymore. He'd gotten fished.
53
00:04:31.465 --> 00:04:35.865
To get to the bank took all day, and the bank's website was inaccessible.
54
00:04:35.865 --> 00:04:54.645
And it sort of felt like a human rights thing to me because even though banking is a business, can't really survive in today's society without banking. And there was no reason that my dad, who spoke 11 languages and was brilliant, should be locked out of it. And so it's really about making technology work for people like my dad,
55
00:04:54.805 --> 00:04:55.605
for people
56
00:04:55.765 --> 00:05:00.405
that might be blind and they can't see the mouse on the screen, so they can't use a mouse.
57
00:05:00.725 --> 00:05:11.570
They can't see the screen, so they need a screen reader to speak it out to them. If you're deaf, you need captions. If you're deaf blind, you use a braille interpreter, and and so you need a transcript
58
00:05:11.650 --> 00:05:12.930
for the braille display.
59
00:05:13.170 --> 00:05:15.650
And then you have color contrast and
60
00:05:15.970 --> 00:05:26.395
visual clarity for people that are getting older or that just have low vision. So it runs the gamut, but it's really about making sure that technology is usable. It's usability, but for more people.
61
00:05:27.435 --> 00:05:32.875
Yeah. Absolutely. I mean, it's definitely one of those terms that becomes all encompassing,
62
00:05:33.169 --> 00:05:48.050
but it has a lot of breadth and it's obviously a very important aspect, actually, for my day job work in higher education. So every piece of technology that we produce is federally mandated to have a high degree of accessibility, so it's something that I deal with quite frequently.
63
00:05:48.565 --> 00:05:55.125
And I know that you mentioned some of the different types of disabilities that will be a hindrance to
64
00:05:55.285 --> 00:05:57.285
interacting with digital technologies,
65
00:05:57.285 --> 00:06:00.245
and also there is a lot of affordances
66
00:06:00.245 --> 00:06:04.770
for people with varying levels of capability in the physical world as well,
67
00:06:05.169 --> 00:06:07.970
notably curb cuts when you're going through a crosswalk.
68
00:06:07.970 --> 00:06:13.825
But I'm wondering if you can talk to some of the ways that having some of those
69
00:06:13.985 --> 00:06:16.385
handicaps or hindrances
70
00:06:16.465 --> 00:06:28.305
to experience interacting with digital technology, how that can act as a substantial hindrance to day to day activity for people who especially people as the world continues to be digitized.
71
00:06:29.000 --> 00:06:38.040
Well, let's take a simple example of if you're deaf, right? And think about the time of COVID where everybody moves to doing things
72
00:06:38.280 --> 00:06:39.960
on Zoom or one of these
73
00:06:41.405 --> 00:06:45.565
screen sharing. I can't remember the word, but you know what I'm talking about, the
74
00:06:46.685 --> 00:06:48.365
virtual meetings.
75
00:06:48.525 --> 00:06:53.005
If you are deaf, you would probably prefer email,
76
00:06:53.005 --> 00:06:53.725
writing,
77
00:06:54.205 --> 00:07:04.420
sending things that you can read as opposed to people speaking on a screen. So even if you can read lips well, the best you'll get is maybe 70%,
78
00:07:04.420 --> 00:07:06.100
and that it'll depend on the person.
79
00:07:06.500 --> 00:07:08.420
So you need to have these captions.
80
00:07:08.420 --> 00:07:20.665
And if the captions aren't good, then you're really out of luck because all of a sudden, you went from your way of communicating to everybody being on a screen. And now imagine that you're you're reading the captions,
81
00:07:20.745 --> 00:07:46.355
right, and you've got five, six, seven people on the screen. Who is speaking? Did the captions come out right, especially live captions, or were they more like craptions, as they're affectionately called in the business? And imagine you've just had a one hour meeting and you had to read a book, like literally read the text of a book because people are speaking so quickly. So the quality of those captions is really vital,
82
00:07:46.675 --> 00:08:06.840
and it really can impact your day to day activity. And I specifically brought up somebody who's blind because, not only because I have friends who have told me about this experience, I didn't make up that word reading a book. That was something that a friend of mine and former employee of mine said. But most people tend to think of screen readers, and they don't even realize that something like Zoom
83
00:08:07.294 --> 00:08:20.895
can be really difficult for for somebody who's deaf, let alone if you're deaf blind and you need to read a transcript because you have a refreshable braille display. So I I'd say that's just an example, but you can take that to any any individual disability.
84
00:08:21.650 --> 00:08:25.490
Your point of some of these video conferencing utilities
85
00:08:25.490 --> 00:08:29.889
as one of the major touch points for the need for accessibility
86
00:08:29.889 --> 00:08:30.530
is
87
00:08:30.690 --> 00:08:46.825
very enlightening because most of the time when people think accessibility beyond the physical world, they're probably thinking about a website. And I'm wondering if you can just talk to some of the various types of interfaces and experiences that are digitally mediated
88
00:08:46.825 --> 00:08:52.150
that do need this level of investment in accessibility improvements
89
00:08:52.150 --> 00:08:57.590
to allow people to actually take part in these various experiences that are so broadly shared?
90
00:08:58.070 --> 00:09:07.075
Yeah. So let's let's think about mobile apps. Right? With the mobile app, it's empowered so many activities that were previously impossible.
91
00:09:07.555 --> 00:09:25.329
And take for an example, and I use this one because I was at a restaurant, we had a whole big party for GAP, Global Accessibility Awareness Day, which from now on, I'll say GATT for short. And there were a whole bunch of people that were blind there, and they were all taking Ubers at the end. And we were out on a street that was very dodgy,
92
00:09:25.665 --> 00:09:30.545
and I wanted to make sure everybody got in their Uber. So imagine that you're at a restaurant,
93
00:09:30.865 --> 00:09:52.280
and before you have Uber or Lyft or one of these apps, you had to trust somebody to call up the cab company. You had to know when the cab was there, so you had to trust a stranger to tell you that they were there. You had to trust the cab driver. You had to trust them with the money that you were paying them. And there's so many stories of people that were abused in various ways as a result of this situation,
94
00:09:52.520 --> 00:10:01.595
and the mobile app changed all of that. And now imagine that you're in a restaurant and there was an update to the software that made it inaccessible.
95
00:10:01.595 --> 00:10:05.995
Five minutes ago, it was accessible and you got to the restaurant, but now you're sitting outside,
96
00:10:06.154 --> 00:10:08.875
the restaurant's closing, you're in a bad part of town,
97
00:10:09.459 --> 00:10:14.580
and your mobile app doesn't work anymore. That is a life threatening situation,
98
00:10:15.060 --> 00:10:27.385
and that's why this is so important, and it might not be obvious if you've never sort of been in that situation. But let's take another example just to give you Oh, and let me just add one more thing there. My mentee, Ben Ogilvie,
99
00:10:27.385 --> 00:10:30.345
who heads up accessibility for ArchTouch,
100
00:10:30.345 --> 00:10:33.225
created the state of mobile app accessibility report,
101
00:10:33.385 --> 00:10:34.985
where they took from
102
00:10:35.145 --> 00:10:45.480
about five different industries, they took the top mobile apps and they graded it for the most common user journeys. We're not talking about edge cases here. And 72%
103
00:10:45.480 --> 00:10:47.320
of the common user journeys
104
00:10:47.480 --> 00:11:05.390
had a poor or fail rate in their report. So those numbers are are pretty bad. And for the web, it's even worse. So this is not theoretical. These these are real problems. And now let's take another example since I know you've got a more technical audience. I created this benchmark
105
00:11:05.390 --> 00:11:17.790
for accessibility called AMAC, and I built an entire CLI for it because I was sure on the command line it would be accessible. Why wouldn't it be accessible? You just have words on a screen. There's so little
106
00:11:18.415 --> 00:11:19.295
visual
107
00:11:19.295 --> 00:11:23.775
stuff that should trip you up. So after building this entire CLI,
108
00:11:23.775 --> 00:11:26.335
I stupidly waited till the end to
109
00:11:26.895 --> 00:11:36.770
turn on a voiceover or a screen reader to test it, and I was shocked that almost nothing came out of it. I used all the common tools people used to build those CLIs,
110
00:11:37.010 --> 00:11:41.250
and literally, you could hear nothing, and I had to throw out the entire CLI
111
00:11:41.330 --> 00:11:43.410
and all of that work because
112
00:11:43.570 --> 00:12:16.575
there's almost no CLI that's accessible. There's like one app in Go that will help you sort of build something somewhat accessible. And Cloud Code, while I'm doing testing and I'm on Cloud Code so much, I have to turn it off because it's just all so much noise. And I have a friend who's blind who does code with Cloud Code, but he says it's very inaccessible and hard, but he's able to do it. I don't know how, but he's obviously better at screen reader. So those are just a couple of different examples of the variety of ways that that the different technologies
113
00:12:16.815 --> 00:12:18.735
interface with people with disabilities.
114
00:12:19.135 --> 00:12:42.965
On the command line utility, I also wonder if it's a factor of which terminal you're using as well because so many different terminals have varying levels of support and features even for some of the various control codes. And then you get into some of the fancy terminals such as warp that even changes a lot of the paradigms that we're used to from the v t 100 terms that we've been using for the past, I don't know, four or five decades.
115
00:12:43.365 --> 00:12:51.690
Yeah. No doubt that that's a big part of it. But in I've used Ghosty, the new one, and that's just a passion project from Mitchell Hashimoto,
116
00:12:51.690 --> 00:12:52.090
who
117
00:12:52.650 --> 00:13:04.250
came out with HashiCorp. And I kind of touched base with him about seeing if it could become a little more accessible, but it's very hard to ask somebody who's just building a passion project in their spare time to,
118
00:13:04.685 --> 00:13:05.644
there's
119
00:13:05.644 --> 00:13:15.565
no monetary thing. So like, what lever do you really have there to push it too hard? But he said he'd like to see it get there. But honestly, like, they're all a mess.
120
00:13:16.045 --> 00:13:28.230
So now digging into some of the role of AI in this overall accessibility space, you touched briefly on the the idea of captions or transcripts for virtual meetings.
121
00:13:28.470 --> 00:13:32.070
Obviously, there are closed captioning for commercially produced
122
00:13:32.545 --> 00:13:33.345
media,
123
00:13:33.425 --> 00:13:44.065
but not necessarily for some of the more mass media. So thinking in terms of YouTube, TikTok, all these various video platforms. I know that YouTube will auto generate transcripts,
124
00:13:44.065 --> 00:13:59.720
but they're obviously going to be of highly variable quality, especially depending on the level of sound quality and the recording, etcetera. Curious if you can just talk to some of the ways that you see AI playing a role in helping to improve the accessibility
125
00:13:59.720 --> 00:14:00.120
of
126
00:14:00.545 --> 00:14:11.985
digital properties as well as helping to circumvent some of the shortcomings of accessibility for some of those same media, whether that's websites, mobile apps, etcetera?
127
00:14:12.465 --> 00:14:15.090
Yeah. So I call this crossing
128
00:14:15.090 --> 00:14:16.210
the Rubicon
129
00:14:16.290 --> 00:14:17.090
AI.
130
00:14:17.090 --> 00:14:21.250
And essentially in any field, when AI gets better at something than a human,
131
00:14:21.490 --> 00:14:23.010
it crossed that Rubicon.
132
00:14:23.010 --> 00:14:25.090
And when it comes to captions,
133
00:14:25.170 --> 00:14:26.690
it's about when
134
00:14:26.930 --> 00:14:39.654
is AI going to do it better than humans? And I don't know if we're there or we're close to there, but I I think I could build within a day or so a system that'll probably be better than more most humans because what I would do very simply
135
00:14:39.894 --> 00:14:40.535
is
136
00:14:40.935 --> 00:14:42.375
try five different,
137
00:14:42.535 --> 00:14:43.495
you know, ASR
138
00:14:44.200 --> 00:14:46.120
tools to create captions,
139
00:14:46.279 --> 00:14:47.720
have them time coded,
140
00:14:48.120 --> 00:14:54.120
and then compare the captions. And then if you compare the captions across three or five different systems
141
00:14:54.200 --> 00:14:55.959
and you see where they agree or disagree,
142
00:14:56.334 --> 00:15:24.010
and then you pull out the diffs where they disagree and maybe make it semi automated, you'll probably get almost perfect captions. But even if you automate it and use an LLM as judge at the end to decide between the differences, you can probably get really far in terms of automated captions, but that's just you hacking it up yourself. Right? But it can go so much deeper than that. So I I have a story where I had a a group chat with two friends who
143
00:15:24.495 --> 00:15:29.855
are both blind, and I had three videos I wanted to show them that were in a series, and they literally
144
00:15:30.095 --> 00:15:37.855
only had visuals. There was no sound in it at all, so it would have been completely useless to send to them. So I'm like, Oh, why don't I try to Vibe Code a solution?
145
00:15:38.420 --> 00:15:45.780
I I Vibe code So because, again, I hadn't touched code in fifteen years. So I just used think it was Cloud Code to have FFmpeg
146
00:15:45.780 --> 00:15:55.395
download the video and then sent it to another model to analyze every frame and describe it, and then had another model summarize
147
00:15:55.715 --> 00:16:00.515
all of those frames and then turned it into an audio description track.
148
00:16:00.755 --> 00:16:07.740
Then I used AI to another model for speech to text that turned that into an audio file.
149
00:16:07.899 --> 00:16:10.620
And then again, used AI with FFmpeg,
150
00:16:10.620 --> 00:16:31.825
which in the past would have taken forever to learn exactly the right commands and all that, to stitch the audio together with the video. And an hour later, I had three videos with a perfect audio description. It was just sublime. And now it'd take me about five minutes to get that script to take a new video. And this was, like, six months ago. Now the models could have probably done it in ten minutes.
151
00:16:32.740 --> 00:16:38.100
And one of the, I think, main driving factors of why accessibility
152
00:16:38.259 --> 00:16:43.220
on the web, on mobile devices, etcetera, is generally so poor is because
153
00:16:43.300 --> 00:16:53.455
it is a non zero amount of effort and investment to add and support those various accessibility features. I know on the web, there are things like ARIA tags to
154
00:16:53.775 --> 00:17:01.230
assist the screen reader and having appropriate descriptions for some of the different content, whether that's for images for people who are visually impaired.
155
00:17:01.310 --> 00:17:05.150
And there are also issues as far as keyboard navigability.
156
00:17:05.150 --> 00:17:22.955
So making sure that the tab ordering on the various elements is correct so that it maps to the user experience. And because of the fact that a lot of these visual mediums are going to be changing so rapidly as designers come up with new layouts, as you add new pages, etcetera, there's a decent amount of effort involved.
157
00:17:23.115 --> 00:17:28.440
But with the fact that these LLMs have gotten so good at code generation,
158
00:17:28.840 --> 00:17:32.519
I'm wondering how you're seeing that change some of that calculus for
159
00:17:32.520 --> 00:17:33.960
developers and
160
00:17:34.120 --> 00:17:38.040
engineers and some of the people who own these businesses,
161
00:17:38.794 --> 00:17:47.355
how that changes the ways that they think about that investment and whether you're seeing any overall improvement in the level of accessibility across the board.
162
00:17:48.155 --> 00:18:17.284
Definitely not across the board. I would say that it's generally pretty bad if you just ask it without any kind of steering to just write some code for you. It's pretty good at alt text. It actually gets that right pretty often, most of the time, but it'll make all kinds of mistakes just like humans do and just like most of the code that it's trained on. And as a result of that, as I had been doing all of this Vibe coding, I really realized that this was going to be a make or break moment for accessibility.
163
00:18:17.365 --> 00:18:21.365
If we got the AI model companies to pay attention to accessibility,
164
00:18:22.159 --> 00:18:28.320
then we would have a much more accessible web and world. And if we don't, then it's just gonna
165
00:18:28.720 --> 00:18:30.559
be an order of magnitude worse.
166
00:18:30.880 --> 00:18:31.759
And so
167
00:18:32.000 --> 00:18:59.630
that's why I created AMAC, which is the AI model accessibility checker. I vibe coded the first version in three days, which I'm sure was very inaccessible, but it did get it did get the concept across, which was ask the top AI models to generate a web page and then use axe core, which is an automated testing tool, to grade all of them and then see how well all of the AI models did. So that first version was easy to write, but very
168
00:19:00.030 --> 00:19:06.864
hard to make accessible, and there were lots of bugs in it. And as anybody that's been playing with these AI coding tools knows,
169
00:19:06.945 --> 00:19:09.744
the models keep getting better, but those first versions,
170
00:19:10.065 --> 00:19:13.104
every time I fixed the bug, it introduced two more bugs.
171
00:19:13.184 --> 00:19:17.585
And before releasing it publicly, I found so many fundamental
172
00:19:17.585 --> 00:19:18.625
errors
173
00:19:18.430 --> 00:19:24.590
that it really required becoming a coder again and really looking at the data model especially. Once
174
00:19:24.750 --> 00:19:25.309
I
175
00:19:25.710 --> 00:19:55.710
sort of defined the data model myself and did not allow the AI to decide it, and I also sort of pushed everything to the database because I was more of a database guy back in the day, then I was able to sort of steer it. But as time goes on, if you don't learn how to steer the model, you're not gonna get good results. So now it's great to see who does well and who does who does poorly, and I was I was really surprised with the results. And now you mentioned this AMAC or AI model accessibility
176
00:19:55.710 --> 00:19:56.509
checker
177
00:19:56.590 --> 00:19:59.229
utility that you've built and wondering
178
00:19:59.230 --> 00:20:07.675
if you can give an overview about what your goals are with that utility and some of the ways that you're to see it used to improve
179
00:20:07.675 --> 00:20:09.755
the distribution of accessibility
180
00:20:09.755 --> 00:20:12.794
across the web and various other digital properties?
181
00:20:13.755 --> 00:20:20.875
Yeah. Well, literally just came out with the new version which come which compared all of the top models. And
182
00:20:21.409 --> 00:20:26.849
my goal is that the model companies take a look at the results, and since they compete on benchmarks,
183
00:20:27.010 --> 00:20:31.729
to have a benchmark that's noticed out there that gets picked up by them.
184
00:20:32.770 --> 00:20:47.144
And then they're going to compete on it, and they're going to do a great job because it's not that hard if you do the training for it. The data is known, the mistakes are known. This is not something that should be all that difficult for them to achieve compared to everything else they've done.
185
00:20:47.544 --> 00:20:55.910
And I bring up the results only because it will help illustrate why this is gonna make a difference. So OpenAI killed the competition.
186
00:20:56.070 --> 00:20:59.270
The top four models are all GPTs,
187
00:20:59.270 --> 00:21:04.310
and they've got five models in the top 10, and their open source model came in 11.
188
00:21:04.870 --> 00:21:10.215
So they have a lot to be proud of. And then you've got MiniMax M1
189
00:21:10.215 --> 00:21:16.054
is tied for fourth place with Codex Mini. So here you have a I think it's a Chinese model
190
00:21:16.215 --> 00:21:30.240
that is a small model that is doing extremely well and way better than the other models that cost a lot more and that are really well known. Even Quen got two models in the top 10, and Kimi came in ninth, and Anthropic
191
00:21:30.240 --> 00:21:35.760
only got to the tenth spot. So here you have an expensive model that everybody's talking about for accessibility,
192
00:21:36.175 --> 00:21:38.255
and I think it was Haiku that got
193
00:21:38.575 --> 00:21:54.340
into, just squeaked into the top 10, and Opus 4.5 is something like 19 on the benchmark, and it's very expensive. But the big surprise is Google came in with three point zero Pro. They came in dead last out of 36 models. And I think that when something like that happens,
194
00:21:54.820 --> 00:22:06.260
all of a sudden, it takes the shine off of an amazing model. And I don't think that they're gonna come in last again. So the real goal here is that AI model companies pay attention to accessibility.
195
00:22:06.715 --> 00:22:09.274
Honestly, as much work as I put into
196
00:22:09.595 --> 00:22:11.034
this benchmark,
197
00:22:11.115 --> 00:22:13.594
which I should mention is a GAD Foundation
198
00:22:13.914 --> 00:22:14.634
activity
199
00:22:14.794 --> 00:22:16.234
and that we partnered
200
00:22:17.115 --> 00:22:25.590
with ServiceNow to come out with it. I should at least give them props. They're not gonna make this mistake again. And our mission is to make sure that digital technology is accessible.
201
00:22:25.830 --> 00:22:27.830
So what better way to do it than this?
202
00:22:28.470 --> 00:22:36.885
I think it's interesting too that all of these models, especially the big frontier models have been trained on effectively the entire Internet.
203
00:22:36.965 --> 00:22:52.839
And the fact that a large portion of that Internet is not accessible is likely one of the contributing factors to why a lot of these frontier models do so poorly in generating accessible code because they don't have enough in their training corpus or at least a large enough portion for that to be the
204
00:22:53.000 --> 00:22:55.879
de facto means of code generation.
205
00:22:55.880 --> 00:23:04.284
And I'm curious, as you have gone through this work of benchmarking and published the results, what are some of the ways that you can imagine,
206
00:23:04.765 --> 00:23:17.340
whether it's the large model providers or somebody fine tuning an open weights model, building up that training corpora to provide the good examples of how best to achieve accessible interfaces
207
00:23:17.340 --> 00:23:25.019
and just some of the outcomes that you're hoping to see from the result of this benchmark and some of the ways that you're hoping to maintain
208
00:23:25.179 --> 00:23:29.740
that set of benchmarks and results as these models do continue to evolve and improve?
209
00:23:30.475 --> 00:23:37.994
Well, one thing that's interesting is WebAIM does a yearly survey of the top million websites on the Internet, just their homepages.
210
00:23:38.154 --> 00:23:48.370
And the data there, at least in terms of the the worst errors that come out, come from the color contrast. Something like 80% of WebAIM's
211
00:23:48.370 --> 00:23:49.330
top issues
212
00:23:49.570 --> 00:23:52.530
websites' top issues was color contrast,
213
00:23:52.770 --> 00:24:03.584
and we're finding the same thing. Maybe even a higher percentage of the of the issues are color contrast. If all of them got color contrast right, they would be well on their way to way more accessible code.
214
00:24:03.825 --> 00:24:10.464
So that's super interesting. But what I forgot to add earlier was this is just the first benchmark,
215
00:24:10.465 --> 00:24:15.200
which is very easy to do. If they go in in their training and they put in linters,
216
00:24:15.280 --> 00:24:19.279
axe core at the base, and they have a linter for accessibility,
217
00:24:19.520 --> 00:24:49.929
then all the automated stuff, they should all get a perfect score. This is not difficult for them to set up. They just have to actually try it. Right? And Google has Lighthouse. Right? They literally in their developer tools, they use the Lighthouse is based on X score. They literally have the tools to do all of this. They just need to put the attention on. Right? But in the future versions, we're going to do some manual testing and we're going to make it a little bit harder. But in terms of them having enough data, they have the data. It's all in the latent space. They have every piece of data they need.
218
00:24:50.170 --> 00:24:51.130
They just need to
219
00:24:51.715 --> 00:25:00.275
steer the model toward the right corners of the code the same way that they're gonna steer the models to the right corners of the web that they've all
220
00:25:00.435 --> 00:25:06.515
slurped up. They clean it up, and you've just gotta clean up that data. And I did hear the other day that
221
00:25:07.370 --> 00:25:08.730
there's a corpus of data,
222
00:25:08.970 --> 00:25:12.890
which I think what they've done is they took the accessibility
223
00:25:12.890 --> 00:25:24.034
bugs for all the open source projects on GitHub, and they took the before and after. So you have a ton of data that shows you what the fixes look like. So this is not a very challenging
224
00:25:24.274 --> 00:25:29.794
problem to solve. You just need some attention to it, and then we're gonna be in a way better place.
225
00:25:30.355 --> 00:25:35.794
One of the other interesting aspects of thinking about AI in the context of accessibility
226
00:25:35.875 --> 00:25:36.595
is
227
00:25:36.870 --> 00:25:44.309
the ways that some of the tools that are powered by AI and how accessible they are. So for some of these various chatbots,
228
00:25:44.550 --> 00:25:47.350
are they accessible? Do they work well with screen readers
229
00:25:47.590 --> 00:25:51.975
for some of the AI coding tools? How do they rank in terms of accessibility?
230
00:25:51.975 --> 00:25:53.335
You mentioned ClaudeCode.
231
00:25:53.575 --> 00:25:57.574
One of your friends was having a poor time with that. And I'm curious how
232
00:25:57.894 --> 00:26:03.654
you're seeing some of these model providers and and various models and coding tools
233
00:26:03.820 --> 00:26:13.260
either lean into or fail to lean into some of those questions of accessibility so that they can be used by as broad of an audience as possible?
234
00:26:13.660 --> 00:26:36.640
Yeah. I haven't personally tested it, so it's hard for me to to to give a robust answer, but just generally from my friends in the space. And I've co presented at GitHub Universe with some folks at GitHub. I know them pretty well. So I know that there are accessibility people on the teams that are building some of their AI products. And from what I understand, if you use Copilot on GitHub,
235
00:26:36.640 --> 00:26:42.640
it is accessible. I just know the people that have built that, so they assure me it is. So I don't think they would lie about that.
236
00:26:43.335 --> 00:26:55.335
And from what I've heard from some others, most of the others aren't, or they are to various degrees. But I assume that people have figured out how to get around it because most of my friends that are blind or have different disabilities
237
00:26:55.810 --> 00:26:59.169
are using ChatGPT and some of these other consumer
238
00:26:59.650 --> 00:27:04.049
products, so they found ways to do it. And then you also have to think about
239
00:27:04.530 --> 00:27:05.490
specialized
240
00:27:05.490 --> 00:27:07.330
services like Aira and
241
00:27:07.890 --> 00:27:10.775
Be My Eyes, which came out with a Be My AI
242
00:27:11.015 --> 00:27:14.135
product. And they were a launch partner with
243
00:27:14.535 --> 00:27:15.495
OpenAI,
244
00:27:15.495 --> 00:27:21.015
which is probably why OpenAI did so well in the benchmark, to tell you the truth, because
245
00:27:21.289 --> 00:27:26.169
I know that they've got a relationship there and that they're paying attention to accessibility,
246
00:27:26.409 --> 00:27:34.169
at least to some degree. And it's also been my experience using their models that it's been a lot easier to steer them. But with BeMyAI,
247
00:27:34.330 --> 00:27:36.649
what it is, is it started out as
248
00:27:36.895 --> 00:27:43.534
a volunteer group where people who can see would connect up with people who are blind, who say, okay,
249
00:27:43.775 --> 00:27:52.830
here's my camera. They can see the screen and then say, is what I'm wearing, does it match? Or, Can you find my keys on the floor? Or whatever they wanted.
250
00:27:53.150 --> 00:27:55.630
And the selling point was the anonymity.
251
00:27:55.630 --> 00:28:00.429
Whereas you have, Aira was another company and they went the other way where they trained people
252
00:28:00.750 --> 00:28:04.225
to be professional about how they are being helpful,
253
00:28:04.385 --> 00:28:14.145
which is just another way to go. So some people don't like the fact that they might know the person, sort of know the person on the other end that acts as their eyes, whereas in the other group, they want that professionality.
254
00:28:14.145 --> 00:28:21.399
So it all depends. But I think, don't I know about Eyre, but I think they've also started to build an AI product. But Be My AI,
255
00:28:21.559 --> 00:28:29.799
now you don't even need a human for a lot of the tasks. You have an AI that's looking at it. And I got a demo where, when it first came out,
256
00:28:30.715 --> 00:28:56.370
where a friend of mine who's blind, I said, tell me what's on this menu. Like, let's say I wanna get a lemon meringue pie. Can I check if that's on the menu? And took a picture of it and showed it right away. Today, I think we're already far enough along where that's not that impressive, but at the time it was like, oh my. And so if you're blind and you're using these tools, it's really big. And the Meta Ray Ban glasses, I know a lot of people that use it especially for airports,
257
00:28:56.765 --> 00:28:57.565
because
258
00:28:57.725 --> 00:29:08.525
at the airport, you can have an AI tell you where to go, where the gate is, you know, where to sit, like everything. It really can kind of direct you pretty well. So it's it's it's been a bit of a game changer for some.
259
00:29:09.419 --> 00:29:11.979
And so digging now into
260
00:29:12.220 --> 00:29:13.820
some of the perception
261
00:29:13.820 --> 00:29:15.580
of accessibility,
262
00:29:15.659 --> 00:29:19.179
the role of AI there, but also just generally the
263
00:29:19.340 --> 00:29:21.260
importance of accessibility.
264
00:29:21.260 --> 00:29:24.234
I know that you also work in the podcast space.
265
00:29:24.794 --> 00:29:31.994
And so I'm wondering, as you've been doing your own podcast and discussing some of these aspects of accessibility,
266
00:29:32.075 --> 00:29:35.674
the role of AI, what are some of the misconceptions
267
00:29:35.674 --> 00:29:36.154
or
268
00:29:36.600 --> 00:29:37.400
falsehoods
269
00:29:37.400 --> 00:29:38.600
that you've encountered
270
00:29:38.600 --> 00:29:42.039
as you've discussed some of these topic areas with folks?
271
00:29:42.679 --> 00:30:04.830
Well, I mean, nobody that comes on the podcast is really a doomsayer when it comes to AI, at least not yet. But, in the industry, there's definitely been plenty of doomsayers from the very beginning, and I still see a lot of it around. And I'm told sometimes not to say what I believe is is gonna come because it's not very popular in the accessibility space. I think part of it is just fear,
272
00:30:05.150 --> 00:30:10.669
and there's definitely legitimate fear about job loss, which I think is already happening because
273
00:30:10.830 --> 00:30:13.869
if a CFO can cut payroll
274
00:30:13.870 --> 00:30:32.519
even by the perception that AI is going to change things, they're they're gonna do it, and they have been doing it. And that part is upsetting and concerning. But from the beginning, they were saying this is going to be bad, it's going to be bad. And I was like, Parts of it might be bad, but a lot of it is going to be a game changer and it's going to help. You know, the WebAIM
275
00:30:32.519 --> 00:30:37.080
million that I mentioned earlier, that report showed that 97%
276
00:30:37.080 --> 00:30:38.679
of the web was inaccessible.
277
00:30:38.840 --> 00:30:40.200
And after six years,
278
00:30:40.519 --> 00:30:42.760
we've now gotten it down to 95
279
00:30:42.760 --> 00:30:44.600
or 94%
280
00:30:44.600 --> 00:30:45.559
inaccessible,
281
00:30:45.559 --> 00:30:46.679
the top million sites.
282
00:30:47.295 --> 00:31:02.979
We have an industry here that's been around for some decades, and that's the best we can do, improve it by a few percent. I mean, the report isn't 100% perfect because some of the issues are not as bad as other issues, but it's still not a great situation. And so if AI
283
00:31:03.059 --> 00:31:08.899
can help improve that, then I'm all for it. And what I try and tell people is you're not gonna stop AI.
284
00:31:08.980 --> 00:31:11.139
It's here, and it's gonna grow.
285
00:31:11.539 --> 00:31:16.634
And if you just leave it to others, you don't have, nobody comes out with a benchmark,
286
00:31:16.794 --> 00:31:19.354
if nobody talks about it, and nobody goes out,
287
00:31:19.755 --> 00:31:25.914
and in essence sells that this is important, right? Because I hate sales
288
00:31:26.235 --> 00:31:32.520
in one way, but in the other way, you have to do it in order to get people to understand your mission and your vision.
289
00:31:33.160 --> 00:31:34.440
And if we don't
290
00:31:34.520 --> 00:31:37.560
get the people building these products to understand
291
00:31:37.720 --> 00:31:44.975
how they impact folks, then we're gonna leave the game up to them. And we already know what happens there. You're gonna have that 97%
292
00:31:44.975 --> 00:31:46.095
inaccessibility.
293
00:31:46.095 --> 00:32:13.674
We have to engage. We have to be a part of this world. We have and we can't go in and just naysay everything. You have to be positive about it, even though there are negative aspects. I'm terrified that that there might be job loss and that this time is different. Technology always created more jobs, and this time feels different. It does. Maybe it'll create more jobs. Maybe it'll take away more jobs. So far, it has probably taken more jobs. Right? You know, I I worked for an organization and
294
00:32:14.075 --> 00:32:15.754
AI decimated their
295
00:32:17.035 --> 00:32:17.674
main
296
00:32:17.915 --> 00:32:19.114
income driver,
297
00:32:19.275 --> 00:32:22.715
and they had mass layoffs, including me. And I had AI in my title.
298
00:32:24.715 --> 00:32:25.355
So there you go.
299
00:32:26.160 --> 00:32:28.560
Yeah. No. It's definitely strange
300
00:32:28.560 --> 00:32:36.320
and interesting time that we live in right now. And I I think that to the point of accessibility in particular,
301
00:32:36.560 --> 00:32:39.360
AI acts as an amplifier
302
00:32:39.360 --> 00:32:50.615
of whatever you put into it. And as we mentioned, a lot of the web is inaccessible, and so the code that we generate from these tools is also inaccessible because we're amplifying the inputs.
303
00:32:50.775 --> 00:32:52.775
And for people who
304
00:32:53.095 --> 00:32:56.110
are invested in improving accessibility,
305
00:32:56.350 --> 00:33:01.710
as you said, they need to make sure that they're guiding these tools in a direction to
306
00:33:01.710 --> 00:33:12.764
enforce some of the accessibility requirements. You mentioned the need for incorporating a linter into that feedback loop for the AIs. What are some of the elements of education and training
307
00:33:12.924 --> 00:33:17.565
and just some of the ways that engineering teams should be thinking about incorporating
308
00:33:17.645 --> 00:33:18.605
accessibility
309
00:33:18.605 --> 00:33:20.124
as a first order
310
00:33:20.365 --> 00:33:25.270
requirement into the work that they're doing as a means of amplifying
311
00:33:25.270 --> 00:33:32.309
its reach and capability as we turn the corner into this new and uncertain era that we're entering?
312
00:33:32.870 --> 00:33:36.550
Number one is is actually working with people with disabilities.
313
00:33:36.985 --> 00:33:49.784
If you actually see what your product looks like from the perspective of someone with a disability, you will never look at your product the same way. I'm officially a public speaker. I've brought people to tears
314
00:33:50.230 --> 00:33:51.269
talking about
315
00:33:51.590 --> 00:33:58.950
I've had moms come to me like crying because of the impact that it's had on their kids and how moved they were about
316
00:33:59.269 --> 00:34:03.830
all of this. But nothing that I've ever done or said compares
317
00:34:04.115 --> 00:34:13.475
to this one time where my co founder of GAD, Jenison Asuncion, who is blind, he went into a travel company, and he is a world traveler.
318
00:34:13.555 --> 00:34:17.635
When I met him, first time, he had just done a red eye
319
00:34:17.890 --> 00:34:18.690
from
320
00:34:18.690 --> 00:34:21.410
Canada. I think it was from Toronto to Vegas,
321
00:34:21.730 --> 00:34:22.850
did a red eye,
322
00:34:23.490 --> 00:34:27.570
went to a wedding of a YouTuber, by the way, Tommy Edison,
323
00:34:27.970 --> 00:34:38.204
the blind film critic. And then he took another red eye the same day. So he didn't even stay overnight at the hotel in Vegas. He went to another city, went right off the plane to Keynote,
324
00:34:38.204 --> 00:34:40.845
and then did sleep over that night.
325
00:34:41.085 --> 00:34:52.300
So it's like the third country, and then he took another trip to Chicago the next day. This is exactly the kind of person that this travel company that's that's their audience.
326
00:34:52.380 --> 00:34:58.685
And when he came in, I I remember we're in a room full of people, of engineers, mobile developers,
327
00:34:58.685 --> 00:35:17.270
and then he got stuck in a keyboard trap just to register. So before he could even do anything else and he was like, I can't I I can't get out of this. What's going on here? I can't see anything. I can't I don't know what's happening. And the entire room went silent, And the impact that he had there is something I could never possibly replicate
328
00:35:17.270 --> 00:35:22.550
because that's your work. And you look at the impact of somebody that's your user,
329
00:35:22.710 --> 00:35:29.715
your core user base. It's not because they're blind, but they're that they can't be perfectly the persona that you are after.
330
00:35:30.035 --> 00:35:32.755
Yeah. That's a very powerful
331
00:35:32.915 --> 00:35:34.435
vignette and representation,
332
00:35:34.435 --> 00:35:38.515
and I think that it is those personal stories that have the
333
00:35:38.675 --> 00:35:49.570
most lasting impact because it's all well and good to say, oh, you need to make sure your website's accessible because you're losing out on x percent of your audience or potential revenue streams.
334
00:35:49.890 --> 00:35:51.010
But everybody,
335
00:35:51.010 --> 00:36:03.445
I'm sure, has either has someone in their day to day orbit who has some form of impairment, whether it's something that you're aware of or not, or you have at least encountered somebody who has those impairments.
336
00:36:04.005 --> 00:36:08.724
Can I I add you since you bring that up? I I I wanna add something here. Well, first of
337
00:36:09.420 --> 00:36:12.780
accessibility team then told us that they heard from their team,
338
00:36:13.099 --> 00:36:31.515
Oh, I used to not really think much of those accessibility bugs and kind of ignored them, but now I get it, and now I'm going to pay attention to it. So that impact was so important. But I wanted to also mention that I had somebody that worked for me who was a designer who was color blind. And it took years for me to find out, even though it's an accessibility company,
339
00:36:31.515 --> 00:36:44.490
but no designer wants to get known, Oh, I'm the color blind designer, because it's gonna impact their career. So you just never know what people are facing in their day to day life. Right? And he was a great designer, by the
340
00:36:45.130 --> 00:36:56.365
And the other piece of it too is one of the common follow ons from all these accessibility conversations that I've experienced is that beyond it opening up the possibility
341
00:36:56.365 --> 00:37:03.085
of a certain portion of your audience to be able to interact with your properties, it also improves
342
00:37:03.300 --> 00:37:05.140
the user experience
343
00:37:05.300 --> 00:37:13.060
for non impaired people. So one of the examples that I've often heard is making it easier to do keyboard navigation,
344
00:37:13.060 --> 00:37:24.365
makes it simpler for the busy parent who's juggling a kid in one arm and a phone or tablet in the other or, you know, laptop in the other trying to get things done. It makes it so that their life is easier.
345
00:37:25.325 --> 00:37:30.285
Yeah. And and you really hit the nail on the head there too because with the AI coding,
346
00:37:30.285 --> 00:37:35.860
the little secret that people are beginning to realize is that CLIs are better than APIs.
347
00:37:36.100 --> 00:37:38.820
If you can build a CLI first,
348
00:37:39.140 --> 00:37:41.860
you're going to be able to interact with your agent much better.
349
00:37:42.260 --> 00:37:47.540
And this, because computers are kind of looking at it the same way as
350
00:37:47.915 --> 00:37:49.355
humans do with a screen
351
00:37:49.835 --> 00:37:50.475
reader.
352
00:37:50.714 --> 00:38:03.355
So essentially people are like, Oh, wait, the computer needs to understand what's going on and the CLIs are not very accessible. And it's like the first time that people started to pay attention to this because it's their agents that are communicating
353
00:38:03.810 --> 00:38:06.690
and they benefit from that very
354
00:38:07.330 --> 00:38:10.530
same case. I didn't explain that very well, but hopefully
355
00:38:10.930 --> 00:38:12.770
all the words eventually
356
00:38:13.010 --> 00:38:23.365
arrange themselves well enough to be clear. That's a very valid point too, is that making it more accessible for humans also makes it easier for the computer to introspect
357
00:38:23.365 --> 00:38:25.125
and be able to
358
00:38:25.445 --> 00:38:26.325
improve
359
00:38:26.405 --> 00:38:31.365
the inputs to its feedback loop to make sure that it's producing appropriate outputs.
360
00:38:31.365 --> 00:38:32.725
And so
361
00:38:32.260 --> 00:38:36.020
I was curious as well, what are some of the areas
362
00:38:36.260 --> 00:38:52.685
of improvement that you would like to see in terms of the types of tooling for doing some of that measurement of accessibility where you mentioned axe core as a means of determining is this website accessible, but what are some of the elements that still require
363
00:38:52.685 --> 00:38:58.285
human judgment to be able to make an appropriate determination of accessibility,
364
00:38:58.285 --> 00:39:01.325
and what are some of the ways that we can improve
365
00:39:01.325 --> 00:39:01.724
the
366
00:39:02.410 --> 00:39:08.570
tools that are available to these agents to accelerate their feedback loops and make accessibility
367
00:39:08.570 --> 00:39:12.410
in a broader scale much more part of that inner loop?
368
00:39:12.970 --> 00:39:15.050
The biggest gap is
369
00:39:15.210 --> 00:39:16.010
the
370
00:39:16.175 --> 00:39:17.055
computers
371
00:39:17.055 --> 00:39:20.335
being able to launch a screen reader or any
372
00:39:20.415 --> 00:39:23.615
assistive technology. There is no cloud platform
373
00:39:23.695 --> 00:39:36.920
that has all the major screen readers, for example. And I think that there's some licensing issues that will stop that from ever happening. But even if you could have, you buy the licenses for yourself or for your organization,
374
00:39:37.160 --> 00:39:41.240
right, and you allow an agent to do that testing
375
00:39:41.905 --> 00:39:42.785
directly,
376
00:39:43.105 --> 00:39:53.425
that would be a game changer, I think, in terms of automating the testing. You still need that human being to look at, well, what does this sound like to a human being?
377
00:39:54.039 --> 00:39:57.160
What is it? How does it come off as? Can't really
378
00:39:57.559 --> 00:39:58.200
I mean,
379
00:39:59.000 --> 00:40:00.759
accessibility professionals,
380
00:40:00.759 --> 00:40:19.194
they can get a lot of the way there, but it's not the same if you're able to navigate without a screen reader. If you're a 100% a screen reader user and you have no choice to turn that off and understand what is going on without it, you're gonna catch things that you just won't catch because you're popping up a screen reader for the purpose of testing.
381
00:40:19.730 --> 00:40:26.609
And so that human element, I don't know if it will ever go away. But again, are we gonna hit a Rubicon moment?
382
00:40:26.849 --> 00:40:42.984
I'm not supposed to say this, but we are. We are. There will be a time where it's gonna be able to where AI will be able to run every single screen reader as long as you bought a copy for yourself and look at the code from every single angle and identify the differences
383
00:40:43.065 --> 00:40:43.785
between
384
00:40:43.944 --> 00:40:49.210
every type of AT. 'll get better than the human at doing that, and then we're gonna be in a much better place.
385
00:40:50.170 --> 00:40:54.490
And I know that there are a number of standards for
386
00:40:54.650 --> 00:40:56.170
the web for
387
00:40:56.410 --> 00:40:58.730
various accessibility technologies.
388
00:40:58.810 --> 00:41:02.755
I'm curious what you're seeing in other
389
00:41:02.915 --> 00:41:04.435
digital arenas,
390
00:41:04.435 --> 00:41:09.235
whether that is things such as augmented reality or virtual reality.
391
00:41:09.315 --> 00:41:12.675
Mobile devices are obviously a a key area.
392
00:41:12.995 --> 00:41:15.235
But as
393
00:41:14.100 --> 00:41:17.460
the entire world gets more and more digitized,
394
00:41:17.460 --> 00:41:20.820
what are some of the areas where we're lacking appropriate
395
00:41:20.820 --> 00:41:23.540
or widely adopted standards for
396
00:41:23.700 --> 00:41:24.900
the accessibility
397
00:41:24.900 --> 00:41:26.740
technologies to be able to interact with?
398
00:41:28.175 --> 00:41:44.070
Well, when it comes to XR, you have a group called XR Access that are really good. And I don't know that we're we're ready for standards at this point because it just I don't know. Is is XR ever gonna take off? I have every single version of the Quest,
399
00:41:44.230 --> 00:41:51.510
and the few times that I've used it, I love it. But that that big thing on me just is not comfortable.
400
00:41:51.589 --> 00:41:55.715
The the what was the the Apple product called again? The Vision
401
00:41:56.355 --> 00:42:04.115
Pro. I don't even remember the I think it's something like that. Yeah, I'm not sure. I tried it in the Apple store. It's an amazing
402
00:42:04.115 --> 00:42:04.995
technology.
403
00:42:05.235 --> 00:42:08.060
These things are just fantastic,
404
00:42:08.060 --> 00:42:13.020
but I don't I don't know how anybody's gonna use them. Even the RayBans,
405
00:42:13.180 --> 00:42:14.300
they're really cool,
406
00:42:14.700 --> 00:42:24.945
but you have the weirdness factor from the people on the other side who don't wanna be filmed by everybody all the time. It's certainly helpful if you can okay. Let me do some visual recognition
407
00:42:25.185 --> 00:42:34.385
and then go to LinkedIn and find the entire history of the person that I'm looking at. That can be useful. Like, I don't you know, as a public speaker, you often you meet so many people,
408
00:42:35.090 --> 00:43:03.105
and they often will come up to you and you can't tell, Is this somebody that I should know and should be embarrassed that I don't know? Or did I once meet them ten years ago because they gave a talk? And so it'd be super useful to be able to do that visual recognition, pull up the LinkedIn, but it's also super creepy. And we've come to accept a lot of things I never thought we would, but that one's a tough one. And in a way, I I hope we don't accept it. And I don't know. I don't know if this technology is going anywhere. Blockchain was fascinating,
409
00:43:03.345 --> 00:43:04.950
but all I've seen
410
00:43:05.270 --> 00:43:19.990
are people getting honest people getting cheated a lot. I have no interest in it. I I know some people that love it that are really good at it technically, but I I can't stand it. So I just don't engage in that so much. So I I don't know that I can give you
411
00:43:19.224 --> 00:43:21.625
a good answer as you might want. Sorry.
412
00:43:22.425 --> 00:43:25.785
Oh, that's absolutely fine. And then another
413
00:43:25.944 --> 00:43:32.905
dimension of this as well is that the ability for software engineers and technologists
414
00:43:33.150 --> 00:43:33.869
to
415
00:43:34.190 --> 00:43:35.710
improve the accessibility
416
00:43:35.710 --> 00:43:36.590
of the
417
00:43:37.230 --> 00:43:40.750
products that they're creating because of these LLMs
418
00:43:40.750 --> 00:43:41.630
accelerating
419
00:43:41.630 --> 00:43:43.630
the development cycles.
420
00:43:43.710 --> 00:43:47.549
It also lowers the barrier to entry for nonengineers
421
00:43:47.549 --> 00:43:53.445
to actually get involved in the creation process. And I'm curious how you're seeing accessibility
422
00:43:53.445 --> 00:43:54.405
professionals,
423
00:43:54.405 --> 00:43:57.445
people who are more on the testing and validation side,
424
00:43:57.605 --> 00:43:58.805
how is the
425
00:43:58.965 --> 00:44:00.085
improved
426
00:44:00.165 --> 00:44:01.365
accessibility
427
00:44:01.500 --> 00:44:02.620
to misappropriate
428
00:44:02.620 --> 00:44:12.300
the word of these code generation tools, improving their ability to get more involved in the creation process and the remediation
429
00:44:12.300 --> 00:44:14.940
process of the accessibility shortcomings?
430
00:44:16.335 --> 00:44:33.330
I I don't know too many people that are that are really doing much. I know maybe two, three, four people. I haven't seen it have the impact that I'd like it to, but it's still early. I I think the problem is you have to put in the hours. Have to put in the miles, and if you don't, then you can't steer these models.
431
00:44:33.490 --> 00:44:34.850
It took me forever
432
00:44:35.090 --> 00:44:38.610
to get it to a point where I could ask a question
433
00:44:39.010 --> 00:44:40.690
about, is this accessible,
434
00:44:40.690 --> 00:44:56.925
and steer it to hopefully give me a good answer. And for example, I got Claude Co. To the point where I told it which sources I wanted it to look up. Anytime I asked the question, I said, I want you to look it up, scrape an article from these sources,
435
00:44:57.520 --> 00:45:02.400
and then tell me whether this is accessible or not, or how to build it accessibly
436
00:45:02.720 --> 00:45:05.680
or whatnot. And it worked for like two weeks,
437
00:45:06.080 --> 00:45:19.015
and then something changed, and a lot of people have complained about Cloud Code's quality dipping. But all of a sudden, they started to quote all of these sources, but they lied. They literally would consistently
438
00:45:19.175 --> 00:45:27.330
it wanted to defend the code that was there. It didn't wanna change anything. It didn't want to evaluate if it was or was not accessible,
439
00:45:27.650 --> 00:45:31.330
so it would change what was on the page and then say,
440
00:45:31.890 --> 00:45:33.330
this is what it says.
441
00:45:33.650 --> 00:45:37.890
And I said so I would keep steering it more and more and say, give me a direct quote.
442
00:45:38.130 --> 00:45:50.405
You've lied to me so often that I'm going to double check it, and it would still lie. Right? And then I started to move to Codex recently, especially after I saw the how well OpenAI did, and that's not happening on Codex.
443
00:45:50.565 --> 00:45:53.125
I'm actually getting really solid answers,
444
00:45:53.365 --> 00:45:57.525
but you have to put in the miles of playing with all these different models.
445
00:45:58.000 --> 00:46:14.565
And I don't know too many people who are, or if they are, they're not being very loud about it. So I wouldn't say that I've seen the impact yet, but now that I've gotten this benchmark where I want it to be, I definitely have some ideas for products that either I'll open source or depending on on what it does,
446
00:46:15.125 --> 00:46:21.205
sell it and try and help people get there. But they'll get there. I think it's another year or two, and then it'll it'll start to happen.
447
00:46:23.140 --> 00:46:24.820
And so for
448
00:46:24.980 --> 00:46:26.820
people who are in
449
00:46:26.980 --> 00:46:30.180
engineering teams, particularly in leadership roles,
450
00:46:30.260 --> 00:46:31.140
what are
451
00:46:31.380 --> 00:46:35.780
some of the ways that they should be thinking about accessibility,
452
00:46:36.020 --> 00:46:36.660
their
453
00:46:37.140 --> 00:46:37.940
responsibilities
454
00:46:38.285 --> 00:46:39.005
for
455
00:46:39.245 --> 00:46:41.645
improving it, their investment in the
456
00:46:42.205 --> 00:46:43.565
core technologies
457
00:46:43.565 --> 00:46:46.765
that they should be investing in to help improve
458
00:46:46.845 --> 00:46:53.725
the ability for as broad of an audience as possible to interact with what they're building, particularly in the context of
459
00:46:54.050 --> 00:46:55.970
the various AI
460
00:46:55.970 --> 00:46:56.770
capabilities
461
00:46:56.770 --> 00:47:00.290
that are being sprinkled through virtually everything these days?
462
00:47:01.410 --> 00:47:06.050
The every organization has their own quirks and their own data,
463
00:47:06.210 --> 00:47:12.665
and one of their differentiators is the data. So if I were part of an engineering organization
464
00:47:12.745 --> 00:47:15.945
and my task was to focus on accessibility,
465
00:47:16.265 --> 00:47:17.945
I would start with taking
466
00:47:17.945 --> 00:47:20.825
the entire corpus of data, of code,
467
00:47:21.480 --> 00:47:24.440
and the accessibility bugs that are being tracked,
468
00:47:24.760 --> 00:47:29.160
and then generate these data sets that show for my particular use case,
469
00:47:29.560 --> 00:47:42.305
what does the before and after look like? What is good code? What is bad code for my code base? And then use that to train models or to fine tune models and improve whatever they put in front of the engineers.
470
00:47:42.305 --> 00:47:51.320
And then, of course, another aspect is you do need to have some kind of chatbot that is going to give you some good direction. You should be able to ask your chatbot accessibility
471
00:47:51.320 --> 00:47:54.920
questions and have it answer it. Most of the organizations
472
00:47:54.920 --> 00:47:56.840
that are doing accessibility
473
00:47:56.840 --> 00:48:10.725
testing and such definitely have some chatbots. The one that I have access to is Deque, and it does pretty good. They have, I think they're called the Deque Bot. And if you buy a subscription, it's total if you're a developer and you wanna learn accessibility,
474
00:48:10.965 --> 00:48:13.365
I would definitely go to Deque University.
475
00:48:13.365 --> 00:48:23.520
WebAIM has some good articles. You know, w three c, there's there's several organizations with good articles, but DQ University will take you through that in a thorough manner, and they've got a really good chatbot,
476
00:48:23.680 --> 00:48:46.244
which is probably for sale at an organization. So you need to be able to have on Slack or whatever tool you use or Teams. You need to be able to ask a chatbot questions and get really reasonably good answers. And then you need to have some agents that are sitting on your code base that are looking for accessibility bugs on every commit that are paying attention to the user
477
00:48:46.720 --> 00:48:52.560
issues that come up and try and vet and validate them and automatically do a PR.
478
00:48:52.880 --> 00:48:57.920
And it really requires some effort to be robust and to steer the model well,
479
00:48:58.000 --> 00:49:08.565
but it's definitely doable. I think we finally reached the point. We're just talking the last few weeks. We finally reached a point where I think you can build really good bots that will write the
480
00:49:09.445 --> 00:49:14.005
code to fix it as soon as the bug comes out. It shouldn't be that hard.
481
00:49:14.405 --> 00:49:19.340
And what I have in mind is building something that that really sources
482
00:49:19.819 --> 00:49:23.740
what they say and proving it. So you say, okay. Here's a quote,
483
00:49:23.980 --> 00:49:26.940
but it's a real quote, not a lie. This is the source.
484
00:49:27.180 --> 00:49:35.135
This is the fix. All you have to do is vet it and accept it. And once you know how to fix that kind of bug, you're not gonna have to look it up again.
485
00:49:35.935 --> 00:49:38.015
And as you have been
486
00:49:38.415 --> 00:49:43.055
working in this overall space of accessibility technologies
487
00:49:42.900 --> 00:49:45.539
and in particular investing in
488
00:49:45.539 --> 00:49:47.220
the benchmarking
489
00:49:47.299 --> 00:49:56.020
of these code generation agents and just experimenting with the various ways that AI can improve the accessibility of digital experiences.
490
00:49:56.020 --> 00:50:00.235
What is one of the most interesting or unexpected or challenging lessons that you've learned in the process?
491
00:50:01.355 --> 00:50:03.035
It's really about the culture.
492
00:50:03.275 --> 00:50:08.155
You can sit and come up with a regulation that you have to be accessible,
493
00:50:08.155 --> 00:50:25.690
and then the organization will focus around the legality of it and try and hit the letter of the law, or you can focus on the culture. And there are two aspects to it. If you can get buy in, top down buy in, and bottom up and middle out to reference
494
00:50:25.690 --> 00:50:30.545
that wonderful TV show Silicon Valley, You suddenly can have an organization
495
00:50:30.704 --> 00:50:31.345
that
496
00:50:31.505 --> 00:50:35.025
will operate on all cylinders, but in most organizations,
497
00:50:35.025 --> 00:50:38.785
don't have the finance department behind you. You are underfunded.
498
00:50:38.785 --> 00:50:42.305
You've got one to 10 people supporting an organization,
499
00:50:42.385 --> 00:50:43.505
often 100,000
500
00:50:43.505 --> 00:50:49.610
and more, and end users sometimes in the millions, and you've got a handful of people. That is overwhelming.
501
00:50:49.610 --> 00:50:54.090
So what you have to do is you have to focus on the culture. And the example I love to give
502
00:50:54.330 --> 00:50:58.035
is if you're a designer and you can't get a budget for accessibility,
503
00:50:58.115 --> 00:51:00.355
but if you do the learning yourself
504
00:51:00.355 --> 00:51:07.155
and you care about your craft, then you're gonna know if you're designing a chat interface that has an online offline indicator.
505
00:51:07.155 --> 00:51:25.540
And if you make the the online indicator a button that's green and an offline indicator that's red, you have now made something that is gray and gray to a color blind user. If instead you have that green and red, but you also write online offline because you don't use color alone to convey information,
506
00:51:25.780 --> 00:51:35.965
now you're good at your craft. Craft. And because you're good at your craft, your users are going to benefit. And you do not need to ask the CFO for permission to write the word online offline
507
00:51:36.045 --> 00:51:42.685
while you're designing because that's part of your culture, that you care about your craft, and you're gonna build something beautiful
508
00:51:42.920 --> 00:51:44.200
that also works.
509
00:51:44.440 --> 00:51:47.000
That is something that goes far beyond
510
00:51:47.160 --> 00:51:50.200
trying to convince people, because everybody asks me,
511
00:51:50.520 --> 00:51:52.920
How do you prove that there's an ROI?
512
00:51:53.000 --> 00:51:57.400
Well, I don't know. How do you prove that usability is gonna improve your company? It's so obvious,
513
00:51:57.945 --> 00:52:05.065
but talk to the CFO about a usability budget. It's really hard. They don't care. They really don't care.
514
00:52:05.465 --> 00:52:09.385
But the people that are writing software and the people that are designing software,
515
00:52:09.545 --> 00:52:18.000
most of them care. And if they can understand the impact of what they're doing and that it costs no money to learn how to do it well, then you're gonna have an impact.
516
00:52:18.880 --> 00:52:19.760
Absolutely.
517
00:52:19.760 --> 00:52:20.880
And particularly
518
00:52:20.880 --> 00:52:21.600
when
519
00:52:21.680 --> 00:52:30.535
code generation is becoming the way that software is being written and there is effectively a zero marginal cost for adding those accessibility features,
520
00:52:30.535 --> 00:52:37.494
then it makes it even more of a no brainer to just start including it as a de facto standard. This is true.
521
00:52:38.870 --> 00:52:44.150
Are there any other aspects of the work that you're doing on accessibility awareness,
522
00:52:44.310 --> 00:52:45.590
your AI
523
00:52:45.590 --> 00:52:46.870
model benchmarks,
524
00:52:46.870 --> 00:52:56.805
or just the overall role of generative AI in this space of improving the utility of digital experiences that we didn't discuss yet that you'd like to cover before we close out the show?
525
00:52:57.445 --> 00:53:01.205
Yes. This is this is a message to the AI practitioners,
526
00:53:01.205 --> 00:53:02.645
artificial intelligence.
527
00:53:03.069 --> 00:53:11.230
What is it we're trying to do? We're trying to emulate human beings. And how do you emulate a human being? You have to get personal.
528
00:53:11.309 --> 00:53:12.830
And as you're getting personal,
529
00:53:13.069 --> 00:53:14.670
you have to understand the differences,
530
00:53:15.605 --> 00:53:17.204
what our capabilities
531
00:53:17.204 --> 00:53:17.765
are,
532
00:53:18.005 --> 00:53:33.110
and if you want to build the best AI models, focus on the edge cases. And where are you gonna find the edge cases when it comes to human beings? People with disabilities. I guarantee you, if you make something usable to someone who's blind, to someone who's deaf, to someone who's deaf blind,
533
00:53:33.270 --> 00:53:38.230
you are going to make a way better product. You're gonna solve problems that you didn't even know existed
534
00:53:38.310 --> 00:53:42.645
because you solved for those edge cases. You are going to be the best AI
535
00:53:42.645 --> 00:53:44.405
researcher and engineer
536
00:53:44.565 --> 00:53:52.405
if you would pay attention, if you go to the edge cases. And I always love the people that that look for the edge cases. They're always the best at what they do.
537
00:53:52.885 --> 00:53:53.765
Absolutely.
538
00:53:53.765 --> 00:54:09.260
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today.
539
00:54:10.415 --> 00:54:15.135
That's a big question. I mean, if we're just purely looking at the benchmark,
540
00:54:15.135 --> 00:54:17.695
color contrast is the number one currently.
541
00:54:17.775 --> 00:54:18.415
But I
542
00:54:18.735 --> 00:54:27.000
think the biggest gap is there because it's expensive and it's hard, and that is trying to get testing with people with disabilities.
543
00:54:27.000 --> 00:54:28.840
There's a great organization called
544
00:54:29.080 --> 00:54:29.560
that
545
00:54:29.880 --> 00:54:31.800
will give you that feedback.
546
00:54:31.880 --> 00:54:49.825
And once you've seen someone with a disability at an organization like Fable go through your website or your mobile app or whatever your product is and show you the gaps, you just can never look at things the same way. But it costs money. I think once you do it, you feel like you haven't fully gotten
547
00:54:49.905 --> 00:55:14.135
your product to where it could be, and you'll always want to have that as long as there's a budget for it. So it's not an easy answer. I wish I wish that I could come up with an easy answer. Like, AI will just solve this for you, but you really need to you need to speak to your users, and that that has not changed since the dawn of commerce. Right? You need to speak to your users. And there are a lot of users with disabilities, so you need to speak to them and get their feedback.
548
00:55:15.175 --> 00:55:24.930
Alright. Well, thank you very much for taking the time today to join me and share the work that you're doing on improving the awareness of accessibility challenges
549
00:55:24.930 --> 00:55:33.170
and your work on helping to push the industry further into investing in that space. It's definitely a very important mission.
550
00:55:33.170 --> 00:55:37.555
So I appreciate all of the time and effort that you're putting into that, and I hope enjoy the rest of your day.
551
00:55:37.955 --> 00:55:40.515
Well, thank you so much. I really enjoyed our conversation.
552
00:55:42.035 --> 00:55:54.540
Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast dot in it covers the Python language, its community, and the innovative ways it is being used.
553
00:55:54.780 --> 00:56:05.381
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com
554
00:56:05.381 --> 00:56:06.101
with your story.