Generative AI Meets Accessibility: Benchmarks, Breakthroughs, and Blind Spots with Joe Devon의 전체 녹취록

--> 00:04:35.865 To get to the bank took all day, and the bank's website was inaccessible. 54 00:04:35.865 --> 00:04:54.645 And it sort of felt like a human rights thing to me because even though banking is a business, can't really survive in today's society without banking. And there was no reason that my dad, who spoke 11 languages and was brilliant, should be locked out of it. And so it's really about making technology work for people like my dad, 55 00:04:54.805 --> 00:04:55.605 for people 56 00:04:55.765 --> 00:05:00.405 that might be blind and they can't see the mouse on the screen, so they can't use a mouse. 57 00:05:00.725 --> 00:05:11.570 They can't see the screen, so they need a screen reader to speak it out to them. If you're deaf, you need captions. If you're deaf blind, you use a braille interpreter, and and so you need a transcript 58 00:05:11.650 --> 00:05:12.930 for the braille display. 59 00:05:13.170 --> 00:05:15.650 And then you have color contrast and 60 00:05:15.970 --> 00:05:26.395 visual clarity for people that are getting older or that just have low vision. So it runs the gamut, but it's really about making sure that technology is usable. It's usability, but for more people. 61 00:05:27.435 --> 00:05:32.875 Yeah. Absolutely. I mean, it's definitely one of those terms that becomes all encompassing, 62 00:05:33.169 --> 00:05:48.050 but it has a lot of breadth and it's obviously a very important aspect, actually, for my day job work in higher education. So every piece of technology that we produce is federally mandated to have a high degree of accessibility, so it's something that I deal with quite frequently. 63 00:05:48.565 --> 00:05:55.125 And I know that you mentioned some of the different types of disabilities that will be a hindrance to 64 00:05:55.285 --> 00:05:57.285 interacting with digital technologies, 65 00:05:57.285 --> 00:06:00.245 and also there is a lot of affordances 66 00:06:00.245 --> 00:06:04.770 for people with varying levels of capability in the physical world as well, 67 00:06:05.169 --> 00:06:07.970 notably curb cuts when you're going through a crosswalk. 68 00:06:07.970 --> 00:06:13.825 But I'm wondering if you can talk to some of the ways that having some of those 69 00:06:13.985 --> 00:06:16.385 handicaps or hindrances 70 00:06:16.465 --> 00:06:28.305 to experience interacting with digital technology, how that can act as a substantial hindrance to day to day activity for people who especially people as the world continues to be digitized. 71 00:06:29.000 --> 00:06:38.040 Well, let's take a simple example of if you're deaf, right? And think about the time of COVID where everybody moves to doing things 72 00:06:38.280 --> 00:06:39.960 on Zoom or one of these 73 00:06:41.405 --> 00:06:45.565 screen sharing. I can't remember the word, but you know what I'm talking about, the 74 00:06:46.685 --> 00:06:48.365 virtual meetings. 75 00:06:48.525 --> 00:06:53.005 If you are deaf, you would probably prefer email, 76 00:06:53.005 --> 00:06:53.725 writing, 77 00:06:54.205 --> 00:07:04.420 sending things that you can read as opposed to people speaking on a screen. So even if you can read lips well, the best you'll get is maybe 70%, 78 00:07:04.420 --> 00:07:06.100 and that it'll depend on the person. 79 00:07:06.500 --> 00:07:08.420 So you need to have these captions. 80 00:07:08.420 --> 00:07:20.665 And if the captions aren't good, then you're really out of luck because all of a sudden, you went from your way of communicating to everybody being on a screen. And now imagine that you're you're reading the captions, 81 00:07:20.745 --> 00:07:46.355 right, and you've got five, six, seven people on the screen. Who is speaking? Did the captions come out right, especially live captions, or were they more like craptions, as they're affectionately called in the business? And imagine you've just had a one hour meeting and you had to read a book, like literally read the text of a book because people are speaking so quickly. So the quality of those captions is really vital, 82 00:07:46.675 --> 00:08:06.840 and it really can impact your day to day activity. And I specifically brought up somebody who's blind because, not only because I have friends who have told me about this experience, I didn't make up that word reading a book. That was something that a friend of mine and former employee of mine said. But most people tend to think of screen readers, and they don't even realize that something like Zoom 83 00:08:07.294 --> 00:08:20.895 can be really difficult for for somebody who's deaf, let alone if you're deaf blind and you need to read a transcript because you have a refreshable braille display. So I I'd say that's just an example, but you can take that to any any individual disability. 84 00:08:21.650 --> 00:08:25.490 Your point of some of these video conferencing utilities 85 00:08:25.490 --> 00:08:29.889 as one of the major touch points for the need for accessibility 86 00:08:29.889 --> 00:08:30.530 is 87 00:08:30.690 --> 00:08:46.825 very enlightening because most of the time when people think accessibility beyond the physical world, they're probably thinking about a website. And I'm wondering if you can just talk to some of the various types of interfaces and experiences that are digitally mediated 88 00:08:46.825 --> 00:08:52.150 that do need this level of investment in accessibility improvements 89 00:08:52.150 --> 00:08:57.590 to allow people to actually take part in these various experiences that are so broadly shared? 90 00:08:58.070 --> 00:09:07.075 Yeah. So let's let's think about mobile apps. Right? With the mobile app, it's empowered so many activities that were previously impossible. 91 00:09:07.555 --> 00:09:25.329 And take for an example, and I use this one because I was at a restaurant, we had a whole big party for GAP, Global Accessibility Awareness Day, which from now on, I'll say GATT for short. And there were a whole bunch of people that

were blind there, and they were all taking Ubers at the end. And we were out on a street that was very dodgy, 92 00:09:25.665 --> 00:09:30.545 and I wanted to make sure everybody got in their Uber. So imagine that you're at a restaurant, 93 00:09:30.865 --> 00:09:52.280 and before you have Uber or Lyft or one of these apps, you had to trust somebody to call up the cab company. You had to know when the cab was there, so you had to trust a stranger to tell you that they were there. You had to trust the cab driver. You had to trust them with the money that you were paying them. And there's so many stories of people that were abused in various ways as a result of this situation, 94 00:09:52.520 --> 00:10:01.595 and the mobile app changed all of that. And now imagine that you're in a restaurant and there was an update to the software that made it inaccessible. 95 00:10:01.595 --> 00:10:05.995 Five minutes ago, it was accessible and you got to the restaurant, but now you're sitting outside, 96 00:10:06.154 --> 00:10:08.875 the restaurant's closing, you're in a bad part of town, 97 00:10:09.459 --> 00:10:14.580 and your mobile app doesn't work anymore. That is a life threatening situation, 98 00:10:15.060 --> 00:10:27.385 and that's why this is so important, and it might not be obvious if you've never sort of been in that situation. But let's take another example just to give you Oh, and let me just add one more thing there. My mentee, Ben Ogilvie, 99 00:10:27.385 --> 00:10:30.345 who heads up accessibility for ArchTouch, 100 00:10:30.345 --> 00:10:33.225 created the state of mobile app accessibility report, 101 00:10:33.385 --> 00:10:34.985 where they took from 102 00:10:35.145 --> 00:10:45.480 about five different industries, they took the top mobile apps and they graded it for the most common user journeys. We're not talking about edge cases here. And 72% 103 00:10:45.480 --> 00:10:47.320 of the common user journeys 104 00:10:47.480 --> 00:11:05.390 had a poor or fail rate in their report. So those numbers are are pretty bad. And for the web, it's even worse. So this is not theoretical. These these are real problems. And now let's take another example since I know you've got a more technical audience. I created this benchmark 105 00:11:05.390 --> 00:11:17.790 for accessibility called AMAC, and I built an entire CLI for it because I was sure on the command line it would be accessible. Why wouldn't it be accessible? You just have words on a screen. There's so little 106 00:11:18.415 --> 00:11:19.295 visual 107 00:11:19.295 --> 00:11:23.775 stuff that should trip you up. So after building this entire CLI, 108 00:11:23.775 --> 00:11:26.335 I stupidly waited till the end to 109 00:11:26.895 --> 00:11:36.770 turn on a voiceover or a screen reader to test it, and I was shocked that almost nothing came out of it. I used all the common tools people used to build those CLIs, 110 00:11:37.010 --> 00:11:41.250 and literally, you could hear nothing, and I had to

throw out the entire CLI 111 00:11:41.330 --> 00:11:43.410 and all of that work because 112 00:11:43.570 --> 00:12:16.575 there's almost no CLI that's accessible. There's like one app in Go that will help you sort of build something somewhat accessible. And Cloud Code, while I'm doing testing and I'm on Cloud Code so much, I have to turn it off because it's just all so much noise. And I have a friend who's blind who does code with Cloud Code, but he says it's very inaccessible and hard, but he's able to do it. I don't know how, but he's obviously better at screen reader. So those are just a couple of different examples of the variety of ways that that the different technologies 113 00:12:16.815 --> 00:12:18.735 interface with people with disabilities. 114 00:12:19.135 --> 00:12:42.965 On the command line utility, I also wonder if it's a factor of which terminal you're using as well because so many different terminals have varying levels of support and features even for some of the various control codes. And then you get into some of the fancy terminals such as warp that even changes a lot of the paradigms that we're used to from the v t 100 terms that we've been using for the past, I don't know, four or five decades. 115 00:12:43.365 --> 00:12:51.690 Yeah. No doubt that that's a big part of it. But in I've used Ghosty, the new one, and that's just a passion project from Mitchell Hashimoto, 116 00:12:51.690 --> 00:12:52.090 who 117 00:12:52.650 --> 00:13:04.250 came out with HashiCorp. And I kind of touched base with him about seeing if it could become a little more accessible, but it's very hard to ask somebody who's just building a passion project in their spare time to, 118 00:13:04.685 --> 00:13:05.644 there's 119 00:13:05.644 --> 00:13:15.565 no monetary thing. So like, what lever do you really have there to push it too hard? But he said he'd like to see it get there. But honestly, like, they're all a mess. 120 00:13:16.045 --> 00:13:28.230 So now digging into some of the role of AI in this overall accessibility space, you touched briefly on the the idea of captions or transcripts for virtual meetings. 121 00:13:28.470 --> 00:13:32.070 Obviously, there are closed captioning for commercially produced 122 00:13:32.545 --> 00:13:33.345 media, 123 00:13:33.425 --> 00:13:44.065 but not necessarily for some of the more mass media. So thinking in terms of YouTube, TikTok, all these various video platforms. I know that YouTube will auto generate transcripts, 124 00:13:44.065 --> 00:13:59.720 but they're obviously going to be of highly variable quality, especially depending on the level of sound quality and the recording, etcetera. Curious if you can just talk to some of the ways that you see AI playing a role in helping to improve the accessibility 125 00:13:59.720 --> 00:14:00.120 of 126 00:14:00.545 --> 00:14:11.985 digital properties as well as helping to circumvent some of the shortcomings of accessibility for some of those same media, whether that's websites, mobile apps, etcetera? 127 00:14:12.465 --> 00:14:15.090 Yeah. So I call this crossing 128 00:14:15.090 --> 00:14:16.210 the Rubicon 129 00:14:16.290 --> 00:14:17.090 AI. 130 00:14:17.090 --> 00:14:21.250 And essentially in any field, when AI gets better at something than a human, 131 00:14:21.490 --> 00:14:23.010 it crossed that Rubicon. 132 00:14:23.010 --> 00:14:25.090 And when it comes to captions, 133 00:14:25.170 --> 00:14:26.690 it's about when 134 00:14:26.930 --> 00:14:39.654 is AI going to do it better than humans? And I don't know if we're there or we're close to there, but I I think I could build within a day or so a system that'll probably be better than more most humans because what I would do very simply 135 00:14:39.894 --> 00:14:40.535 is 136 00:14:40.935 --> 00:14:42.375 try five different, 137 00:14:42.535 --> 00:14:43.495 you know, ASR 138 00:14:44.200 --> 00:14:46.120 tools to create captions, 139 00:14:46.279 --> 00:14:47.720 have them time coded, 140 00:14:48.120 --> 00:14:54.120 and then compare the captions. And then if you compare the captions across three or five different systems 141 00:14:54.200 --> 00:14:55.959 and you see where they agree or disagree, 142 00:14:56.334 --> 00:15:24.010 and then you pull out the diffs where they disagree and maybe make it semi automated, you'll probably get almost perfect captions. But even if you automate it and use an LLM as judge at the end to decide between the differences, you can probably get really far in terms of automated captions, but that's just you hacking it up yourself. Right? But it can go so much deeper than that. So I I have a story where I had a a group chat with two friends who 143 00:15:24.495 --> 00:15:29.855 are both blind, and I had three videos I wanted to show them that were in a series, and they literally 144 00:15:30.095 --> 00:15:37.855 only had visuals. There was no sound in it at all, so it would have been completely useless to send to them. So I'm like, Oh, why don't I try to Vibe Code a solution? 145 00:15:38.420 --> 00:15:45.780 I I Vibe code So because, again, I hadn't touched code in fifteen years. So I just used think it was Cloud Code to have FFmpeg 146 00:15:45.780 --> 00:15:55.395 download the video and then sent it to another model to analyze every frame and describe it, and then had another model summarize 147 00:15:55.715 --> 00:16:00.515 all of those frames and then turned it into an audio description track. 148 00:16:00.755 --> 00:16:07.740 Then I used AI to another model for speech to text that turned that into an audio file. 149 00:16:07.899 --> 00:16:10.620 And then again, used AI with FFmpeg, 150 00:16:10.620 --> 00:16:31.825 which in the past would have taken forever to learn exactly the right commands and all that, to stitch the audio together with the video. And an hour later, I had three videos with a perfect audio description. It was just sublime. And now it'd take me about five minutes to get that script to take a

new video. And this was, like, six months ago. Now the models could have probably done it in ten minutes. 151 00:16:32.740 --> 00:16:38.100 And one of the, I think, main driving factors of why accessibility 152 00:16:38.259 --> 00:16:43.220 on the web, on mobile devices, etcetera, is generally so poor is because 153 00:16:43.300 --> 00:16:53.455 it is a non zero amount of effort and investment to add and support those various accessibility features. I know on the web, there are things like ARIA tags to 154 00:16:53.775 --> 00:17:01.230 assist the screen reader and having appropriate descriptions for some of the different content, whether that's for images for people who are visually impaired. 155 00:17:01.310 --> 00:17:05.150 And there are also issues as far as keyboard navigability. 156 00:17:05.150 --> 00:17:22.955 So making sure that the tab ordering on the various elements is correct so that it maps to the user experience. And because of the fact that a lot of these visual mediums are going to be changing so rapidly as designers come up with new layouts, as you add new pages, etcetera, there's a decent amount of effort involved. 157 00:17:23.115 --> 00:17:28.440 But with the fact that these LLMs have gotten so good at code generation, 158 00:17:28.840 --> 00:17:32.519 I'm wondering how you're seeing that change some of that calculus for 159 00:17:32.520 --> 00:17:33.960 developers and 160 00:17:34.120 --> 00:17:38.040 engineers and some of the people who own these businesses, 161 00:17:38.794 --> 00:17:47.355 how that changes the ways that they think about that investment and whether you're seeing any overall improvement in the level of accessibility across the board. 162 00:17:48.155 --> 00:18:17.284 Definitely not across the board. I would say that it's generally pretty bad if you just ask it without any kind of steering to just write some code for you. It's pretty good at alt text. It actually gets that right pretty often, most of the time, but it'll make all kinds of mistakes just like humans do and just like most of the code that it's trained on. And as a result of that, as I had been doing all of this Vibe coding, I really realized that this was going to be a make or break moment for accessibility. 163 00:18:17.365 --> 00:18:21.365 If we got the AI model companies to pay attention to accessibility, 164 00:18:22.159 --> 00:18:28.320 then we would have a much more accessible web and world. And if we don't, then it's just gonna 165 00:18:28.720 --> 00:18:30.559 be an order of magnitude worse. 166 00:18:30.880 --> 00:18:31.759 And so 167 00:18:32.000 --> 00:18:59.630 that's why I created AMAC, which is the AI model accessibility checker. I vibe coded the first version in three days, which I'm sure was very inaccessible, but it did get it did get the concept across, which was ask the top AI models to generate a web page and then use axe core, which is an automated testing tool, to grade all of them and then see how well all of the AI models

did. So that first version was easy to write, but very 168 00:19:00.030 --> 00:19:06.864 hard to make accessible, and there were lots of bugs in it. And as anybody that's been playing with these AI coding tools knows, 169 00:19:06.945 --> 00:19:09.744 the models keep getting better, but those first versions, 170 00:19:10.065 --> 00:19:13.104 every time I fixed the bug, it introduced two more bugs. 171 00:19:13.184 --> 00:19:17.585 And before releasing it publicly, I found so many fundamental 172 00:19:17.585 --> 00:19:18.625 errors 173 00:19:18.430 --> 00:19:24.590 that it really required becoming a coder again and really looking at the data model especially. Once 174 00:19:24.750 --> 00:19:25.309 I 175 00:19:25.710 --> 00:19:55.710 sort of defined the data model myself and did not allow the AI to decide it, and I also sort of pushed everything to the database because I was more of a database guy back in the day, then I was able to sort of steer it. But as time goes on, if you don't learn how to steer the model, you're not gonna get good results. So now it's great to see who does well and who does who does poorly, and I was I was really surprised with the results. And now you mentioned this AMAC or AI model accessibility 176 00:19:55.710 --> 00:19:56.509 checker 177 00:19:56.590 --> 00:19:59.229 utility that you've built and wondering 178 00:19:59.230 --> 00:20:07.675 if you can give an overview about what your goals are with that utility and some of the ways that you're to see it used to improve 179 00:20:07.675 --> 00:20:09.755 the distribution of accessibility 180 00:20:09.755 --> 00:20:12.794 across the web and various other digital properties? 181 00:20:13.755 --> 00:20:20.875 Yeah. Well, literally just came out with the new version which come which compared all of the top models. And 182 00:20:21.409 --> 00:20:26.849 my goal is that the model companies take a look at the results, and since they compete on benchmarks, 183 00:20:27.010 --> 00:20:31.729 to have a benchmark that's noticed out there that gets picked up by them. 184 00:20:32.770 --> 00:20:47.144 And then they're going to compete on it, and they're going to do a great job because it's not that hard if you do the training for it. The data is known, the mistakes are known. This is not something that should be all that difficult for them to achieve compared to everything else they've done. 185 00:20:47.544 --> 00:20:55.910 And I bring up the results only because it will help illustrate why this is gonna make a difference. So OpenAI killed the competition. 186 00:20:56.070 --> 00:20:59.270 The top four models are all GPTs, 187 00:20:59.270 --> 00:21:04.310 and they've got five models in the top 10, and their open source model came in 11. 188 00:21:04.870 --> 00:21:10.215 So they have a lot to be proud of. And then you've got MiniMax M1 189 00:21:10.215 --> 00:21:16.054 is tied for fourth place with Codex Mini. So here you have a I think it's a Chinese model 190 00:21:16.215 --> 00:21:30.240 that is a small model that is doing extremely well and way better than the other models that cost a lot more and that are really well known. Even Quen got two models in the top 10, and Kimi came in ninth, and Anthropic 191 00:21:30.240 --> 00:21:35.760 only got to the tenth spot. So here you have an expensive model that everybody's talking about for accessibility, 192 00:21:36.175 --> 00:21:38.255 and I think it was Haiku that got 193 00:21:38.575 --> 00:21:54.340 into, just squeaked into the top 10, and Opus 4.5 is something like 19 on the benchmark, and it's very expensive. But the big surprise is Google came in with three point zero Pro. They came in dead last out of 36 models. And I think that when something like that happens, 194 00:21:54.820 --> 00:22:06.260 all of a sudden, it takes the shine off of an amazing model. And I don't think that they're gonna come in last again. So the real goal here is that AI model companies pay attention to accessibility. 195 00:22:06.715 --> 00:22:09.274 Honestly, as much work as I put into 196 00:22:09.595 --> 00:22:11.034 this benchmark, 197 00:22:11.115 --> 00:22:13.594 which I should mention is a GAD Foundation 198 00:22:13.914 --> 00:22:14.634 activity 199 00:22:14.794 --> 00:22:16.234 and that we partnered 200 00:22:17.115 --> 00:22:25.590 with ServiceNow to come out with it. I should at least give them props. They're not gonna make this mistake again. And our mission is to make sure that digital technology is accessible. 201 00:22:25.830 --> 00:22:27.830 So what better way to do it than this? 202 00:22:28.470 --> 00:22:36.885 I think it's interesting too that all of these models, especially the big frontier models have been trained on effectively the entire Internet. 203 00:22:36.965 --> 00:22:52.839 And the fact that a large portion of that Internet is not accessible is likely one of the contributing factors to why a lot of these frontier models do so poorly in generating accessible code because they don't have enough in their training corpus or at least a large enough portion for that to be the 204 00:22:53.000 --> 00:22:55.879 de facto means of code generation. 205 00:22:55.880 --> 00:23:04.284 And I'm curious, as you have gone through this work of benchmarking and published the results, what are some of the ways that you can imagine, 206 00:23:04.765 --> 00:23:17.340 whether it's the large model providers or somebody fine tuning an open weights model, building up that training corpora to provide the good examples of how best to achieve accessible interfaces 207 00:23:17.340 --> 00:23:25.019 and just some of the outcomes that you're hoping to see from the result of this benchmark and some of the ways that you're hoping to maintain 208 00:23:25.179 --> 00:23:29.740 that set of benchmarks and results as these models do continue to evolve and improve? 209 00:23:30.475 --> 00:23:37.994 Well, one thing that's interesting is WebAIM does a yearly survey of the top million websites on the

Internet, just their homepages. 210 00:23:38.154 --> 00:23:48.370 And the data there, at least in terms of the the worst errors that come out, come from the color contrast. Something like 80% of WebAIM's 211 00:23:48.370 --> 00:23:49.330 top issues 212 00:23:49.570 --> 00:23:52.530 websites' top issues was color contrast, 213 00:23:52.770 --> 00:24:03.584 and we're finding the same thing. Maybe even a higher percentage of the of the issues are color contrast. If all of them got color contrast right, they would be well on their way to way more accessible code. 214 00:24:03.825 --> 00:24:10.464 So that's super interesting. But what I forgot to add earlier was this is just the first benchmark, 215 00:24:10.465 --> 00:24:15.200 which is very easy to do. If they go in in their training and they put in linters, 216 00:24:15.280 --> 00:24:19.279 axe core at the base, and they have a linter for accessibility, 217 00:24:19.520 --> 00:24:49.929 then all the automated stuff, they should all get a perfect score. This is not difficult for them to set up. They just have to actually try it. Right? And Google has Lighthouse. Right? They literally in their developer tools, they use the Lighthouse is based on X score. They literally have the tools to do all of this. They just need to put the attention on. Right? But in the future versions, we're going to do some manual testing and we're going to make it a little bit harder. But in terms of them having enough data, they have the data. It's all in the latent space. They have every piece of data they need. 218 00:24:50.170 --> 00:24:51.130 They just need to 219 00:24:51.715 --> 00:25:00.275 steer the model toward the right corners of the code the same way that they're gonna steer the models to the right corners of the web that they've all 220 00:25:00.435 --> 00:25:06.515 slurped up. They clean it up, and you've just gotta clean up that data. And I did hear the other day that 221 00:25:07.370 --> 00:25:08.730 there's a corpus of data, 222 00:25:08.970 --> 00:25:12.890 which I think what they've done is they took the accessibility 223 00:25:12.890 --> 00:25:24.034 bugs for all the open source projects on GitHub, and they took the before and after. So you have a ton of data that shows you what the fixes look like. So this is not a very challenging 224 00:25:24.274 --> 00:25:29.794 problem to solve. You just need some attention to it, and then we're gonna be in a way better place. 225 00:25:30.355 --> 00:25:35.794 One of the other interesting aspects of thinking about AI in the context of accessibility 226 00:25:35.875 --> 00:25:36.595 is 227 00:25:36.870 --> 00:25:44.309 the ways that some of the tools that are powered by AI and how accessible they are. So for some of these various chatbots, 228 00:25:44.550 --> 00:25:47.350 are they accessible? Do they work well with screen readers 229 00:25:47.590 --> 00:25:51.975 for some of the AI coding tools? How do they rank in terms of accessibility? 230 00:25:51.975 --> 00:25:53.335 You mentioned ClaudeCode. 231 00:25:53.575 --> 00:25:57.574 One of your friends was having a poor time with that. And I'm curious how 232 00:25:57.894 --> 00:26:03.654 you're seeing some of these model providers and and various models and coding tools 233 00:26:03.820 --> 00:26:13.260 either lean into or fail to lean into some of those questions of accessibility so that they can be used by as broad of an audience as possible? 234 00:26:13.660 --> 00:26:36.640 Yeah. I haven't personally tested it, so it's hard for me to to to give a robust answer, but just generally from my friends in the space. And I've co presented at GitHub Universe with some folks at GitHub. I know them pretty well. So I know that there are accessibility people on the teams that are building some of their AI products. And from what I understand, if you use Copilot on GitHub, 235 00:26:36.640 --> 00:26:42.640 it is accessible. I just know the people that have built that, so they assure me it is. So I don't think they would lie about that. 236 00:26:43.335 --> 00:26:55.335 And from what I've heard from some others, most of the others aren't, or they are to various degrees. But I assume that people have figured out how to get around it because most of my friends that are blind or have different disabilities 237 00:26:55.810 --> 00:26:59.169 are using ChatGPT and some of these other consumer 238 00:26:59.650 --> 00:27:04.049 products, so they found ways to do it. And then you also have to think about 239 00:27:04.530 --> 00:27:05.490 specialized 240 00:27:05.490 --> 00:27:07.330 services like Aira and 241 00:27:07.890 --> 00:27:10.775 Be My Eyes, which came out with a Be My AI 242 00:27:11.015 --> 00:27:14.135 product. And they were a launch partner with 243 00:27:14.535 --> 00:27:15.495 OpenAI, 244 00:27:15.495 --> 00:27:21.015 which is probably why OpenAI did so well in the benchmark, to tell you the truth, because 245 00:27:21.289 --> 00:27:26.169 I know that they've got a relationship there and that they're paying attention to accessibility, 246 00:27:26.409 --> 00:27:34.169 at least to some degree. And it's also been my experience using their models that it's been a lot easier to steer them. But with BeMyAI, 247 00:27:34.330 --> 00:27:36.649 what it is, is it started out as 248 00:27:36.895 --> 00:27:43.534 a volunteer group where people who can see would connect up with people who are blind, who say, okay, 249 00:27:43.775 --> 00:27:52.830 here's my camera. They can see the screen and then say, is what I'm wearing, does it match? Or, Can you find my keys on the floor? Or whatever they wanted. 250 00:27:53.150 --> 00:27:55.630 And the selling point was the anonymity. 251 00:27:55.630 --> 00:28:00.429 Whereas you have, Aira was another company and they went the other way where they trained people 252 00:28:00.750 --> 00:28:04.225 to be professional about how they are being helpful, 253 00:28:04.385 --> 00:28:14.145 which is just another way to go. So some people

don't like the fact that they might know the person, sort of know the person on the other end that acts as their eyes, whereas in the other group, they want that professionality. 254 00:28:14.145 --> 00:28:21.399 So it all depends. But I think, don't I know about Eyre, but I think they've also started to build an AI product. But Be My AI, 255 00:28:21.559 --> 00:28:29.799 now you don't even need a human for a lot of the tasks. You have an AI that's looking at it. And I got a demo where, when it first came out, 256 00:28:30.715 --> 00:28:56.370 where a friend of mine who's blind, I said, tell me what's on this menu. Like, let's say I wanna get a lemon meringue pie. Can I check if that's on the menu? And took a picture of it and showed it right away. Today, I think we're already far enough along where that's not that impressive, but at the time it was like, oh my. And so if you're blind and you're using these tools, it's really big. And the Meta Ray Ban glasses, I know a lot of people that use it especially for airports, 257 00:28:56.765 --> 00:28:57.565 because 258 00:28:57.725 --> 00:29:08.525 at the airport, you can have an AI tell you where to go, where the gate is, you know, where to sit, like everything. It really can kind of direct you pretty well. So it's it's it's been a bit of a game changer for some. 259 00:29:09.419 --> 00:29:11.979 And so digging now into 260 00:29:12.220 --> 00:29:13.820 some of the perception 261 00:29:13.820 --> 00:29:15.580 of accessibility, 262 00:29:15.659 --> 00:29:19.179 the role of AI there, but also just generally the 263 00:29:19.340 --> 00:29:21.260 importance of accessibility. 264 00:29:21.260 --> 00:29:24.234 I know that you also work in the podcast space. 265 00:29:24.794 --> 00:29:31.994 And so I'm wondering, as you've been doing your own podcast and discussing some of these aspects of accessibility, 266 00:29:32.075 --> 00:29:35.674 the role of AI, what are some of the misconceptions 267 00:29:35.674 --> 00:29:36.154 or 268 00:29:36.600 --> 00:29:37.400 falsehoods 269 00:29:37.400 --> 00:29:38.600 that you've encountered 270 00:29:38.600 --> 00:29:42.039 as you've discussed some of these topic areas with folks? 271 00:29:42.679 --> 00:30:04.830 Well, I mean, nobody that comes on the podcast is really a doomsayer when it comes to AI, at least not yet. But, in the industry, there's definitely been plenty of doomsayers from the very beginning, and I still see a lot of it around. And I'm told sometimes not to say what I believe is is gonna come because it's not very popular in the accessibility space. I think part of it is just fear, 272 00:30:05.150 --> 00:30:10.669 and there's definitely legitimate fear about job loss, which I think is already happening because 273 00:30:10.830 --> 00:30:13.869 if a CFO can cut payroll 274 00:30:13.870 --> 00:30:32.519 even by the perception that AI is going to change things, they're they're gonna do it, and they have been doing it. And that part is upsetting and concerning. But from the beginning, they were saying this is going to be bad, it's going to be bad. And I was like, Parts of it might be bad, but a lot of it is going to be a game changer and it's going to help. You know, the WebAIM 275 00:30:32.519 --> 00:30:37.080 million that I mentioned earlier, that report showed that 97% 276 00:30:37.080 --> 00:30:38.679 of the web was inaccessible. 277 00:30:38.840 --> 00:30:40.200 And after six years, 278 00:30:40.519 --> 00:30:42.760 we've now gotten it down to 95 279 00:30:42.760 --> 00:30:44.600 or 94% 280 00:30:44.600 --> 00:30:45.559 inaccessible, 281 00:30:45.559 --> 00:30:46.679 the top million sites. 282 00:30:47.295 --> 00:31:02.979 We have an industry here that's been around for some decades, and that's the best we can do, improve it by a few percent. I mean, the report isn't 100% perfect because some of the issues are not as bad as other issues, but it's still not a great situation. And so if AI 283 00:31:03.059 --> 00:31:08.899 can help improve that, then I'm all for it. And what I try and tell people is you're not gonna stop AI. 284 00:31:08.980 --> 00:31:11.139 It's here, and it's gonna grow. 285 00:31:11.539 --> 00:31:16.634 And if you just leave it to others, you don't have, nobody comes out with a benchmark, 286 00:31:16.794 --> 00:31:19.354 if nobody talks about it, and nobody goes out, 287 00:31:19.755 --> 00:31:25.914 and in essence sells that this is important, right? Because I hate sales 288 00:31:26.235 --> 00:31:32.520 in one way, but in the other way, you have to do it in order to get people to understand your mission and your vision. 289 00:31:33.160 --> 00:31:34.440 And if we don't 290 00:31:34.520 --> 00:31:37.560 get the people building these products to understand 291 00:31:37.720 --> 00:31:44.975 how they impact folks, then we're gonna leave the game up to them. And we already know what happens there. You're gonna have that 97% 292 00:31:44.975 --> 00:31:46.095 inaccessibility. 293 00:31:46.095 --> 00:32:13.674 We have to engage. We have to be a part of this world. We have and we can't go in and just naysay everything. You have to be positive about it, even though there are negative aspects. I'm terrified that that there might be job loss and that this time is different. Technology always created more jobs, and this time feels different. It does. Maybe it'll create more jobs. Maybe it'll take away more jobs. So far, it has probably taken more jobs. Right? You know, I I worked for an organization and 294 00:32:14.075 --> 00:32:15.754 AI decimated their 295 00:32:17.035 --> 00:32:17.674 main 296 00:32:17.915 --> 00:32:19.114 income driver, 297 00:32:19.275 --> 00:32:22.715 and they had mass layoffs, including me. And I had AI in my title. 298 00:32:24.715 --> 00:32:25.355 So there you go. 299 00:32:26.160 --> 00:32:28.560 Yeah. No. It's definitely strange 300 00:32:28.560 --> 00:32:36.320 and interesting time that we live in right now. And I I think that to the point of

accessibility in particular, 301 00:32:36.560 --> 00:32:39.360 AI acts as an amplifier 302 00:32:39.360 --> 00:32:50.615 of whatever you put into it. And as we mentioned, a lot of the web is inaccessible, and so the code that we generate from these tools is also inaccessible because we're amplifying the inputs. 303 00:32:50.775 --> 00:32:52.775 And for people who 304 00:32:53.095 --> 00:32:56.110 are invested in improving accessibility, 305 00:32:56.350 --> 00:33:01.710 as you said, they need to make sure that they're guiding these tools in a direction to 306 00:33:01.710 --> 00:33:12.764 enforce some of the accessibility requirements. You mentioned the need for incorporating a linter into that feedback loop for the AIs. What are some of the elements of education and training 307 00:33:12.924 --> 00:33:17.565 and just some of the ways that engineering teams should be thinking about incorporating 308 00:33:17.645 --> 00:33:18.605 accessibility 309 00:33:18.605 --> 00:33:20.124 as a first order 310 00:33:20.365 --> 00:33:25.270 requirement into the work that they're doing as a means of amplifying 311 00:33:25.270 --> 00:33:32.309 its reach and capability as we turn the corner into this new and uncertain era that we're entering? 312 00:33:32.870 --> 00:33:36.550 Number one is is actually working with people with disabilities. 313 00:33:36.985 --> 00:33:49.784 If you actually see what your product looks like from the perspective of someone with a disability, you will never look at your product the same way. I'm officially a public speaker. I've brought people to tears 314 00:33:50.230 --> 00:33:51.269 talking about 315 00:33:51.590 --> 00:33:58.950 I've had moms come to me like crying because of the impact that it's had on their kids and how moved they were about 316 00:33:59.269 --> 00:34:03.830 all of this. But nothing that I've ever done or said compares 317 00:34:04.115 --> 00:34:13.475 to this one time where my co founder of GAD, Jenison Asuncion, who is blind, he went into a travel company, and he is a world traveler. 318 00:34:13.555 --> 00:34:17.635 When I met him, first time, he had just done a red eye 319 00:34:17.890 --> 00:34:18.690 from 320 00:34:18.690 --> 00:34:21.410 Canada. I think it was from Toronto to Vegas, 321 00:34:21.730 --> 00:34:22.850 did a red eye, 322 00:34:23.490 --> 00:34:27.570 went to a wedding of a YouTuber, by the way, Tommy Edison, 323 00:34:27.970 --> 00:34:38.204 the blind film critic. And then he took another red eye the same day. So he didn't even stay overnight at the hotel in Vegas. He went to another city, went right off the plane to Keynote, 324 00:34:38.204 --> 00:34:40.845 and then did sleep over that night. 325 00:34:41.085 --> 00:34:52.300 So it's like the third country, and then he took another trip to Chicago the next day. This is exactly the kind of person that this travel company that's that's their audience. 326 00:34:52.380 --> 00:34:58.685 And when he came in, I I remember we're in a room full of

people, of engineers, mobile developers, 327 00:34:58.685 --> 00:35:17.270 and then he got stuck in a keyboard trap just to register. So before he could even do anything else and he was like, I can't I I can't get out of this. What's going on here? I can't see anything. I can't I don't know what's happening. And the entire room went silent, And the impact that he had there is something I could never possibly replicate 328 00:35:17.270 --> 00:35:22.550 because that's your work. And you look at the impact of somebody that's your user, 329 00:35:22.710 --> 00:35:29.715 your core user base. It's not because they're blind, but they're that they can't be perfectly the persona that you are after. 330 00:35:30.035 --> 00:35:32.755 Yeah. That's a very powerful 331 00:35:32.915 --> 00:35:34.435 vignette and representation, 332 00:35:34.435 --> 00:35:38.515 and I think that it is those personal stories that have the 333 00:35:38.675 --> 00:35:49.570 most lasting impact because it's all well and good to say, oh, you need to make sure your website's accessible because you're losing out on x percent of your audience or potential revenue streams. 334 00:35:49.890 --> 00:35:51.010 But everybody, 335 00:35:51.010 --> 00:36:03.445 I'm sure, has either has someone in their day to day orbit who has some form of impairment, whether it's something that you're aware of or not, or you have at least encountered somebody who has those impairments. 336 00:36:04.005 --> 00:36:08.724 Can I I add you since you bring that up? I I I wanna add something here. Well, first of 337 00:36:09.420 --> 00:36:12.780 accessibility team then told us that they heard from their team, 338 00:36:13.099 --> 00:36:31.515 Oh, I used to not really think much of those accessibility bugs and kind of ignored them, but now I get it, and now I'm going to pay attention to it. So that impact was so important. But I wanted to also mention that I had somebody that worked for me who was a designer who was color blind. And it took years for me to find out, even though it's an accessibility company, 339 00:36:31.515 --> 00:36:44.490 but no designer wants to get known, Oh, I'm the color blind designer, because it's gonna impact their career. So you just never know what people are facing in their day to day life. Right? And he was a great designer, by the 340 00:36:45.130 --> 00:36:56.365 And the other piece of it too is one of the common follow ons from all these accessibility conversations that I've experienced is that beyond it opening up the possibility 341 00:36:56.365 --> 00:37:03.085 of a certain portion of your audience to be able to interact with your properties, it also improves 342 00:37:03.300 --> 00:37:05.140 the user experience 343 00:37:05.300 --> 00:37:13.060 for non impaired people. So one of the examples that I've often heard is making it easier to do keyboard navigation, 344 00:37:13.060 --> 00:37:24.365 makes it simpler for the busy parent who's juggling a kid in one arm and a phone or tablet in the other or, you know, laptop in the other trying to get things done. It makes it so that their life is easier. 345 00:37:25.325 --> 00:37:30.285 Yeah. And and you really hit the nail on the head there too because with the AI coding, 346 00:37:30.285 --> 00:37:35.860 the little secret that people are beginning to realize is that CLIs are better than APIs. 347 00:37:36.100 --> 00:37:38.820 If you can build a CLI first, 348 00:37:39.140 --> 00:37:41.860 you're going to be able to interact with your agent much better. 349 00:37:42.260 --> 00:37:47.540 And this, because computers are kind of looking at it the same way as 350 00:37:47.915 --> 00:37:49.355 humans do with a screen 351 00:37:49.835 --> 00:37:50.475 reader. 352 00:37:50.714 --> 00:38:03.355 So essentially people are like, Oh, wait, the computer needs to understand what's going on and the CLIs are not very accessible. And it's like the first time that people started to pay attention to this because it's their agents that are communicating 353 00:38:03.810 --> 00:38:06.690 and they benefit from that very 354 00:38:07.330 --> 00:38:10.530 same case. I didn't explain that very well, but hopefully 355 00:38:10.930 --> 00:38:12.770 all the words eventually 356 00:38:13.010 --> 00:38:23.365 arrange themselves well enough to be clear. That's a very valid point too, is that making it more accessible for humans also makes it easier for the computer to introspect 357 00:38:23.365 --> 00:38:25.125 and be able to 358 00:38:25.445 --> 00:38:26.325 improve 359 00:38:26.405 --> 00:38:31.365 the inputs to its feedback loop to make sure that it's producing appropriate outputs. 360 00:38:31.365 --> 00:38:32.725 And so 361 00:38:32.260 --> 00:38:36.020 I was curious as well, what are some of the areas 362 00:38:36.260 --> 00:38:52.685 of improvement that you would like to see in terms of the types of tooling for doing some of that measurement of accessibility where you mentioned axe core as a means of determining is this website accessible, but what are some of the elements that still require 363 00:38:52.685 --> 00:38:58.285 human judgment to be able to make an appropriate determination of accessibility, 364 00:38:58.285 --> 00:39:01.325 and what are some of the ways that we can improve 365 00:39:01.325 --> 00:39:01.724 the 366 00:39:02.410 --> 00:39:08.570 tools that are available to these agents to accelerate their feedback loops and make accessibility 367 00:39:08.570 --> 00:39:12.410 in a broader scale much more part of that inner loop? 368 00:39:12.970 --> 00:39:15.050 The biggest gap is 369 00:39:15.210 --> 00:39:16.010 the 370 00:39:16.175 --> 00:39:17.055 computers 371 00:39:17.055 --> 00:39:20.335 being able to launch a screen reader or any 372 00:39:20.415 --> 00:39:23.615 assistive technology. There is no cloud platform 373 00:39:23.695 --> 00:39:36.920 that has all the major screen readers, for example. And I think that there's some licensing issues that will stop that from ever

happening. But even if you could have, you buy the licenses for yourself or for your organization, 374 00:39:37.160 --> 00:39:41.240 right, and you allow an agent to do that testing 375 00:39:41.905 --> 00:39:42.785 directly, 376 00:39:43.105 --> 00:39:53.425 that would be a game changer, I think, in terms of automating the testing. You still need that human being to look at, well, what does this sound like to a human being? 377 00:39:54.039 --> 00:39:57.160 What is it? How does it come off as? Can't really 378 00:39:57.559 --> 00:39:58.200 I mean, 379 00:39:59.000 --> 00:40:00.759 accessibility professionals, 380 00:40:00.759 --> 00:40:19.194 they can get a lot of the way there, but it's not the same if you're able to navigate without a screen reader. If you're a 100% a screen reader user and you have no choice to turn that off and understand what is going on without it, you're gonna catch things that you just won't catch because you're popping up a screen reader for the purpose of testing. 381 00:40:19.730 --> 00:40:26.609 And so that human element, I don't know if it will ever go away. But again, are we gonna hit a Rubicon moment? 382 00:40:26.849 --> 00:40:42.984 I'm not supposed to say this, but we are. We are. There will be a time where it's gonna be able to where AI will be able to run every single screen reader as long as you bought a copy for yourself and look at the code from every single angle and identify the differences 383 00:40:43.065 --> 00:40:43.785 between 384 00:40:43.944 --> 00:40:49.210 every type of AT. 'll get better than the human at doing that, and then we're gonna be in a much better place. 385 00:40:50.170 --> 00:40:54.490 And I know that there are a number of standards for 386 00:40:54.650 --> 00:40:56.170 the web for 387 00:40:56.410 --> 00:40:58.730 various accessibility technologies. 388 00:40:58.810 --> 00:41:02.755 I'm curious what you're seeing in other 389 00:41:02.915 --> 00:41:04.435 digital arenas, 390 00:41:04.435 --> 00:41:09.235 whether that is things such as augmented reality or virtual reality. 391 00:41:09.315 --> 00:41:12.675 Mobile devices are obviously a a key area. 392 00:41:12.995 --> 00:41:15.235 But as 393 00:41:14.100 --> 00:41:17.460 the entire world gets more and more digitized, 394 00:41:17.460 --> 00:41:20.820 what are some of the areas where we're lacking appropriate 395 00:41:20.820 --> 00:41:23.540 or widely adopted standards for 396 00:41:23.700 --> 00:41:24.900 the accessibility 397 00:41:24.900 --> 00:41:26.740 technologies to be able to interact with? 398 00:41:28.175 --> 00:41:44.070 Well, when it comes to XR, you have a group called XR Access that are really good. And I don't know that we're we're ready for standards at this point because it just I don't know. Is is XR ever gonna take off? I have every single version of the Quest, 399 00:41:44.230 --> 00:41:51.510 and the few times that I've used it, I love it. But that that big thing on me just is not comfortable. 400 00:41:51.589

--> 00:41:55.715 The the what was the the Apple product called again? The Vision 401 00:41:56.355 --> 00:42:04.115 Pro. I don't even remember the I think it's something like that. Yeah, I'm not sure. I tried it in the Apple store. It's an amazing 402 00:42:04.115 --> 00:42:04.995 technology. 403 00:42:05.235 --> 00:42:08.060 These things are just fantastic, 404 00:42:08.060 --> 00:42:13.020 but I don't I don't know how anybody's gonna use them. Even the RayBans, 405 00:42:13.180 --> 00:42:14.300 they're really cool, 406 00:42:14.700 --> 00:42:24.945 but you have the weirdness factor from the people on the other side who don't wanna be filmed by everybody all the time. It's certainly helpful if you can okay. Let me do some visual recognition 407 00:42:25.185 --> 00:42:34.385 and then go to LinkedIn and find the entire history of the person that I'm looking at. That can be useful. Like, I don't you know, as a public speaker, you often you meet so many people, 408 00:42:35.090 --> 00:43:03.105 and they often will come up to you and you can't tell, Is this somebody that I should know and should be embarrassed that I don't know? Or did I once meet them ten years ago because they gave a talk? And so it'd be super useful to be able to do that visual recognition, pull up the LinkedIn, but it's also super creepy. And we've come to accept a lot of things I never thought we would, but that one's a tough one. And in a way, I I hope we don't accept it. And I don't know. I don't know if this technology is going anywhere. Blockchain was fascinating, 409 00:43:03.345 --> 00:43:04.950 but all I've seen 410 00:43:05.270 --> 00:43:19.990 are people getting honest people getting cheated a lot. I have no interest in it. I I know some people that love it that are really good at it technically, but I I can't stand it. So I just don't engage in that so much. So I I don't know that I can give you 411 00:43:19.224 --> 00:43:21.625 a good answer as you might want. Sorry. 412 00:43:22.425 --> 00:43:25.785 Oh, that's absolutely fine. And then another 413 00:43:25.944 --> 00:43:32.905 dimension of this as well is that the ability for software engineers and technologists 414 00:43:33.150 --> 00:43:33.869 to 415 00:43:34.190 --> 00:43:35.710 improve the accessibility 416 00:43:35.710 --> 00:43:36.590 of the 417 00:43:37.230 --> 00:43:40.750 products that they're creating because of these LLMs 418 00:43:40.750 --> 00:43:41.630 accelerating 419 00:43:41.630 --> 00:43:43.630 the development cycles. 420 00:43:43.710 --> 00:43:47.549 It also lowers the barrier to entry for nonengineers 421 00:43:47.549 --> 00:43:53.445 to actually get involved in the creation process. And I'm curious how you're seeing accessibility 422 00:43:53.445 --> 00:43:54.405 professionals, 423 00:43:54.405 --> 00:43:57.445 people who are more on the testing and validation side, 424 00:43:57.605 --> 00:43:58.805 how is the 425 00:43:58.965 --> 00:44:00.085 improved 426 00:44:00.165 --> 00:44:01.365 accessibility 427 00:44:01.500 --> 00:44:02.620 to misappropriate 428 00:44:02.620 --> 00:44:12.300 the word of these code generation tools, improving their ability to get more involved in the creation process and the remediation 429 00:44:12.300 --> 00:44:14.940 process of the accessibility shortcomings? 430 00:44:16.335 --> 00:44:33.330 I I don't know too many people that are that are really doing much. I know maybe two, three, four people. I haven't seen it have the impact that I'd like it to, but it's still early. I I think the problem is you have to put in the hours. Have to put in the miles, and if you don't, then you can't steer these models. 431 00:44:33.490 --> 00:44:34.850 It took me forever 432 00:44:35.090 --> 00:44:38.610 to get it to a point where I could ask a question 433 00:44:39.010 --> 00:44:40.690 about, is this accessible, 434 00:44:40.690 --> 00:44:56.925 and steer it to hopefully give me a good answer. And for example, I got Claude Co. To the point where I told it which sources I wanted it to look up. Anytime I asked the question, I said, I want you to look it up, scrape an article from these sources, 435 00:44:57.520 --> 00:45:02.400 and then tell me whether this is accessible or not, or how to build it accessibly 436 00:45:02.720 --> 00:45:05.680 or whatnot. And it worked for like two weeks, 437 00:45:06.080 --> 00:45:19.015 and then something changed, and a lot of people have complained about Cloud Code's quality dipping. But all of a sudden, they started to quote all of these sources, but they lied. They literally would consistently 438 00:45:19.175 --> 00:45:27.330 it wanted to defend the code that was there. It didn't wanna change anything. It didn't want to evaluate if it was or was not accessible, 439 00:45:27.650 --> 00:45:31.330 so it would change what was on the page and then say, 440 00:45:31.890 --> 00:45:33.330 this is what it says. 441 00:45:33.650 --> 00:45:37.890 And I said so I would keep steering it more and more and say, give me a direct quote. 442 00:45:38.130 --> 00:45:50.405 You've lied to me so often that I'm going to double check it, and it would still lie. Right? And then I started to move to Codex recently, especially after I saw the how well OpenAI did, and that's not happening on Codex. 443 00:45:50.565 --> 00:45:53.125 I'm actually getting really solid answers, 444 00:45:53.365 --> 00:45:57.525 but you have to put in the miles of playing with all these different models. 445 00:45:58.000 --> 00:46:14.565 And I don't know too many people who are, or if they are, they're not being very loud about it. So I wouldn't say that I've seen the impact yet, but now that I've gotten this benchmark where I want it to be, I definitely have some ideas for products that either I'll open source or depending on on what it does, 446 00:46:15.125 --> 00:46:21.205 sell it and try and help people get there. But they'll get there. I think it's another year or two, and then it'll it'll start to happen. 447 00:46:23.140

--> 00:46:24.820 And so for 448 00:46:24.980 --> 00:46:26.820 people who are in 449 00:46:26.980 --> 00:46:30.180 engineering teams, particularly in leadership roles, 450 00:46:30.260 --> 00:46:31.140 what are 451 00:46:31.380 --> 00:46:35.780 some of the ways that they should be thinking about accessibility, 452 00:46:36.020 --> 00:46:36.660 their 453 00:46:37.140 --> 00:46:37.940 responsibilities 454 00:46:38.285 --> 00:46:39.005 for 455 00:46:39.245 --> 00:46:41.645 improving it, their investment in the 456 00:46:42.205 --> 00:46:43.565 core technologies 457 00:46:43.565 --> 00:46:46.765 that they should be investing in to help improve 458 00:46:46.845 --> 00:46:53.725 the ability for as broad of an audience as possible to interact with what they're building, particularly in the context of 459 00:46:54.050 --> 00:46:55.970 the various AI 460 00:46:55.970 --> 00:46:56.770 capabilities 461 00:46:56.770 --> 00:47:00.290 that are being sprinkled through virtually everything these days? 462 00:47:01.410 --> 00:47:06.050 The every organization has their own quirks and their own data, 463 00:47:06.210 --> 00:47:12.665 and one of their differentiators is the data. So if I were part of an engineering organization 464 00:47:12.745 --> 00:47:15.945 and my task was to focus on accessibility, 465 00:47:16.265 --> 00:47:17.945 I would start with taking 466 00:47:17.945 --> 00:47:20.825 the entire corpus of data, of code, 467 00:47:21.480 --> 00:47:24.440 and the accessibility bugs that are being tracked, 468 00:47:24.760 --> 00:47:29.160 and then generate these data sets that show for my particular use case, 469 00:47:29.560 --> 00:47:42.305 what does the before and after look like? What is good code? What is bad code for my code base? And then use that to train models or to fine tune models and improve whatever they put in front of the engineers. 470 00:47:42.305 --> 00:47:51.320 And then, of course, another aspect is you do need to have some kind of chatbot that is going to give you some good direction. You should be able to ask your chatbot accessibility 471 00:47:51.320 --> 00:47:54.920 questions and have it answer it. Most of the organizations 472 00:47:54.920 --> 00:47:56.840 that are doing accessibility 473 00:47:56.840 --> 00:48:10.725 testing and such definitely have some chatbots. The one that I have access to is Deque, and it does pretty good. They have, I think they're called the Deque Bot. And if you buy a subscription, it's total if you're a developer and you wanna learn accessibility, 474 00:48:10.965 --> 00:48:13.365 I would definitely go to Deque University. 475 00:48:13.365 --> 00:48:23.520 WebAIM has some good articles. You know, w three c, there's there's several organizations with good articles, but DQ University will take you through that in a thorough manner, and they've got a really good chatbot, 476 00:48:23.680 --> 00:48:46.244 which is probably for sale at an organization. So you need to be able to have on Slack or whatever tool you use or Teams. You need to be able to ask a chatbot questions and get really reasonably good answers. And then you need to have some agents that are sitting on your code base that are looking for accessibility bugs on every commit that are paying attention to the user 477 00:48:46.720 --> 00:48:52.560 issues that come up and try and vet and validate them and automatically do a PR. 478 00:48:52.880 --> 00:48:57.920 And it really requires some effort to be robust and to steer the model well, 479 00:48:58.000 --> 00:49:08.565 but it's definitely doable. I think we finally reached the point. We're just talking the last few weeks. We finally reached a point where I think you can build really good bots that will write the 480 00:49:09.445 --> 00:49:14.005 code to fix it as soon as the bug comes out. It shouldn't be that hard. 481 00:49:14.405 --> 00:49:19.340 And what I have in mind is building something that that really sources 482 00:49:19.819 --> 00:49:23.740 what they say and proving it. So you say, okay. Here's a quote, 483 00:49:23.980 --> 00:49:26.940 but it's a real quote, not a lie. This is the source. 484 00:49:27.180 --> 00:49:35.135 This is the fix. All you have to do is vet it and accept it. And once you know how to fix that kind of bug, you're not gonna have to look it up again. 485 00:49:35.935 --> 00:49:38.015 And as you have been 486 00:49:38.415 --> 00:49:43.055 working in this overall space of accessibility technologies 487 00:49:42.900 --> 00:49:45.539 and in particular investing in 488 00:49:45.539 --> 00:49:47.220 the benchmarking 489 00:49:47.299 --> 00:49:56.020 of these code generation agents and just experimenting with the various ways that AI can improve the accessibility of digital experiences. 490 00:49:56.020 --> 00:50:00.235 What is one of the most interesting or unexpected or challenging lessons that you've learned in the process? 491 00:50:01.355 --> 00:50:03.035 It's really about the culture. 492 00:50:03.275 --> 00:50:08.155 You can sit and come up with a regulation that you have to be accessible, 493 00:50:08.155 --> 00:50:25.690 and then the organization will focus around the legality of it and try and hit the letter of the law, or you can focus on the culture. And there are two aspects to it. If you can get buy in, top down buy in, and bottom up and middle out to reference 494 00:50:25.690 --> 00:50:30.545 that wonderful TV show Silicon Valley, You suddenly can have an organization 495 00:50:30.704 --> 00:50:31.345 that 496 00:50:31.505 --> 00:50:35.025 will operate on all cylinders, but in most organizations, 497 00:50:35.025 --> 00:50:38.785 don't have the finance department behind you. You are underfunded. 498 00:50:38.785 --> 00:50:42.305 You've got one to 10 people supporting an organization, 499 00:50:42.385 --> 00:50:43.505 often 100,000 500 00:50:43.505 --> 00:50:49.610 and more, and end users sometimes in the millions, and you've got a handful of people. That is overwhelming. 501 00:50:49.610 -->

00:50:54.090 So what you have to do is you have to focus on the culture. And the example I love to give 502 00:50:54.330 --> 00:50:58.035 is if you're a designer and you can't get a budget for accessibility, 503 00:50:58.115 --> 00:51:00.355 but if you do the learning yourself 504 00:51:00.355 --> 00:51:07.155 and you care about your craft, then you're gonna know if you're designing a chat interface that has an online offline indicator. 505 00:51:07.155 --> 00:51:25.540 And if you make the the online indicator a button that's green and an offline indicator that's red, you have now made something that is gray and gray to a color blind user. If instead you have that green and red, but you also write online offline because you don't use color alone to convey information, 506 00:51:25.780 --> 00:51:35.965 now you're good at your craft. Craft. And because you're good at your craft, your users are going to benefit. And you do not need to ask the CFO for permission to write the word online offline 507 00:51:36.045 --> 00:51:42.685 while you're designing because that's part of your culture, that you care about your craft, and you're gonna build something beautiful 508 00:51:42.920 --> 00:51:44.200 that also works. 509 00:51:44.440 --> 00:51:47.000 That is something that goes far beyond 510 00:51:47.160 --> 00:51:50.200 trying to convince people, because everybody asks me, 511 00:51:50.520 --> 00:51:52.920 How do you prove that there's an ROI? 512 00:51:53.000 --> 00:51:57.400 Well, I don't know. How do you prove that usability is gonna improve your company? It's so obvious, 513 00:51:57.945 --> 00:52:05.065 but talk to the CFO about a usability budget. It's really hard. They don't care. They really don't care. 514 00:52:05.465 --> 00:52:09.385 But the people that are writing software and the people that are designing software, 515 00:52:09.545 --> 00:52:18.000 most of them care. And if they can understand the impact of what they're doing and that it costs no money to learn how to do it well, then you're gonna have an impact. 516 00:52:18.880 --> 00:52:19.760 Absolutely. 517 00:52:19.760 --> 00:52:20.880 And particularly 518 00:52:20.880 --> 00:52:21.600 when 519 00:52:21.680 --> 00:52:30.535 code generation is becoming the way that software is being written and there is effectively a zero marginal cost for adding those accessibility features, 520 00:52:30.535 --> 00:52:37.494 then it makes it even more of a no brainer to just start including it as a de facto standard. This is true. 521 00:52:38.870 --> 00:52:44.150 Are there any other aspects of the work that you're doing on accessibility awareness, 522 00:52:44.310 --> 00:52:45.590 your AI 523 00:52:45.590 --> 00:52:46.870 model benchmarks, 524 00:52:46.870 --> 00:52:56.805 or just the overall role of generative AI in this space of improving the utility of digital experiences that we didn't discuss yet that you'd like to cover before we close out the show? 525 00:52:57.445 --> 00:53:01.205 Yes. This is

this is a message to the AI practitioners, 526 00:53:01.205 --> 00:53:02.645 artificial intelligence. 527 00:53:03.069 --> 00:53:11.230 What is it we're trying to do? We're trying to emulate human beings. And how do you emulate a human being? You have to get personal. 528 00:53:11.309 --> 00:53:12.830 And as you're getting personal, 529 00:53:13.069 --> 00:53:14.670 you have to understand the differences, 530 00:53:15.605 --> 00:53:17.204 what our capabilities 531 00:53:17.204 --> 00:53:17.765 are, 532 00:53:18.005 --> 00:53:33.110 and if you want to build the best AI models, focus on the edge cases. And where are you gonna find the edge cases when it comes to human beings? People with disabilities. I guarantee you, if you make something usable to someone who's blind, to someone who's deaf, to someone who's deaf blind, 533 00:53:33.270 --> 00:53:38.230 you are going to make a way better product. You're gonna solve problems that you didn't even know existed 534 00:53:38.310 --> 00:53:42.645 because you solved for those edge cases. You are going to be the best AI 535 00:53:42.645 --> 00:53:44.405 researcher and engineer 536 00:53:44.565 --> 00:53:52.405 if you would pay attention, if you go to the edge cases. And I always love the people that that look for the edge cases. They're always the best at what they do. 537 00:53:52.885 --> 00:53:53.765 Absolutely. 538 00:53:53.765 --> 00:54:09.260 Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today. 539 00:54:10.415 --> 00:54:15.135 That's a big question. I mean, if we're just purely looking at the benchmark, 540 00:54:15.135 --> 00:54:17.695 color contrast is the number one currently. 541 00:54:17.775 --> 00:54:18.415 But I 542 00:54:18.735 --> 00:54:27.000 think the biggest gap is there because it's expensive and it's hard, and that is trying to get testing with people with disabilities. 543 00:54:27.000 --> 00:54:28.840 There's a great organization called 544 00:54:29.080 --> 00:54:29.560 that 545 00:54:29.880 --> 00:54:31.800 will give you that feedback. 546 00:54:31.880 --> 00:54:49.825 And once you've seen someone with a disability at an organization like Fable go through your website or your mobile app or whatever your product is and show you the gaps, you just can never look at things the same way. But it costs money. I think once you do it, you feel like you haven't fully gotten 547 00:54:49.905 --> 00:55:14.135 your product to where it could be, and you'll always want to have that as long as there's a budget for it. So it's not an easy answer. I wish I wish that I could come up with an easy answer. Like, AI will just solve this for you, but you really need to you need to speak to your users, and that that has not changed since the dawn of commerce. Right? You need to speak to your users. And there are a lot of users with disabilities, so you need to speak to them and get their feedback. 548 00:55:15.175 --> 00:55:24.930 Alright. Well, thank you very much for taking the time today to join me and share the work that you're doing on improving the awareness of accessibility challenges 549 00:55:24.930 --> 00:55:33.170 and your work on helping to push the industry further into investing in that space. It's definitely a very important mission. 550 00:55:33.170 --> 00:55:37.555 So I appreciate all of the time and effort that you're putting into that, and I hope enjoy the rest of your day. 551 00:55:37.955 --> 00:55:40.515 Well, thank you so much. I really enjoyed our conversation. 552 00:55:42.035 --> 00:55:54.540 Thank you for listening. Don't forget to check out our other shows. The data engineering podcast covers the latest on modern data management, and podcast dot in it covers the Python language, its community, and the innovative ways it is being used. 553 00:55:54.780 --> 00:56:05.381 Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@aiengineeringpodcast.com 554 00:56:05.381 --> 00:56:06.101 with your story.