WEBVTT
00:00:00.000 --> 00:00:03.520
What's the lesson here for people who feel like government is too big and too broken to fix?
00:00:03.759 --> 00:00:10.000
If you can't change the two most obvious things, you have to optimize your change literally everywhere else.
00:00:10.240 --> 00:00:13.599
People are weaponizing a democratic process specifically to slow it down.
00:00:13.839 --> 00:00:19.679
I literally took my laptop and went to the food court and started going up to people.
00:00:20.000 --> 00:00:23.280
Welcome to the Functional Government Podcast.
00:00:23.440 --> 00:00:30.879
Most people have never heard of regulations.gov, but it's where ordinary citizens get to push back on rules that will affect their lives.
00:00:31.120 --> 00:00:34.399
It's been operating since 2003 and its scale is enormous.
00:00:34.640 --> 00:00:39.920
The FCC's net neutrality rulemaking received over 20 million comments.
00:00:40.079 --> 00:00:44.560
Yet of those 22 million, the New York Attorney General labeled 18 million as fakes.
00:00:44.719 --> 00:00:55.840
One college student filed 7.7 million comments in support of keeping the rules, while internet service providers paid consulting firms$8.2 million to generate opposition content.
00:00:56.000 --> 00:01:00.399
The EPA's Clean Power Plan drew 4.3 million comments.
00:01:00.640 --> 00:01:03.359
Canada has its own version of this called the Canada Gazette.
00:01:03.520 --> 00:01:08.239
Most people don't know about it, but publishing something in the Gazette is considered official notice.
00:01:08.480 --> 00:01:15.200
Canada didn't even have online commenting until September 2022, following a 2021 pilot.
00:01:15.439 --> 00:01:18.480
Before that, comments came only by mail or email.
00:01:18.640 --> 00:01:23.519
In fact, it was the Kuzma Trade Agreement that pushed Canada to build what the U.S.
00:01:23.680 --> 00:01:25.280
had had for nearly 20 years.
00:01:25.439 --> 00:01:26.959
The Gazette doesn't get much use.
00:01:27.120 --> 00:01:39.200
The Air Passenger Protection Regulations, which was one of the most consumer-facing high-profile consultations in recent years, received around 300 submissions, which is pretty tiny for a country that spends a lot of time complaining about air travel.
00:01:39.359 --> 00:01:49.040
A fee proposal for air travel complaints received 83 written submissions, 30 from the public, 46 from airlines, six from consumer groups, and one from a territorial government.
00:01:49.200 --> 00:01:51.680
And a six-month consultation on open government.
00:01:51.760 --> 00:01:57.680
A group you'd expect to use a lot of open government tools like the Gazette had 1,451 participants.
00:01:57.840 --> 00:02:03.519
Before the online system, departments were tracking comments using Excel spreadsheets and email folders.
00:02:03.680 --> 00:02:09.599
The project team had to do weekly bilingual training sessions because regulators had no common system.
00:02:09.759 --> 00:02:13.199
But we're in a new world now with automation and AI changing the game.
00:02:13.360 --> 00:02:19.039
Regulations.gov is now grappling with AI-generated comments on top of mass comment campaigns.
00:02:19.280 --> 00:02:19.840
One U.S.
00:02:19.919 --> 00:02:28.159
government study found that from 5 to 30% of people whose emails were attached to public comments said they didn't actually make the comment.
00:02:28.400 --> 00:02:32.240
This is the exact problem our guest and his team were hired to address.
00:02:32.319 --> 00:02:37.120
So please welcome Aaron Myers, the chief data officer of Ishmael Interactive.
00:02:37.360 --> 00:02:37.599
Hello.
00:02:37.840 --> 00:02:38.000
Hi.
00:02:38.159 --> 00:02:39.199
Hey Aaron, how are you?
00:02:39.520 --> 00:02:40.000
Pretty good.
00:02:40.159 --> 00:02:40.560
Pretty good.
00:02:40.639 --> 00:02:41.280
Thanks for having me.
00:02:41.599 --> 00:02:45.680
Can you explain what regulations gov does and why it matters?
00:02:45.919 --> 00:02:49.680
As if you were explaining it to someone who's never interacted with government beyond paying taxes.
00:02:50.000 --> 00:02:51.039
Excellent, excellent.
00:02:51.599 --> 00:03:04.719
So regulations.gov is the idea that as a law gets passed and they have to make rules and regulations to say here's how we follow the law, the government's not going to know everything.
00:03:04.879 --> 00:03:09.280
They usually don't, but there are very few places where the government's going to say that out loud.
00:03:09.599 --> 00:03:17.039
So they put up a draft and they say, here's my list of questions, which would help me make this rule much better.
00:03:17.199 --> 00:03:28.159
And they're hoping people will go there, find it, answer it, fill out the information that they're looking for, give them a little more completeness before that draft becomes final.
00:03:28.560 --> 00:03:35.599
When you first encountered regulations.gov, what was the experience like for an average person trying to do a comment on a proposed rule?
00:03:35.680 --> 00:03:37.360
Like what did they actually have to see and do?
00:03:37.599 --> 00:03:46.400
Well, it's it's a little bit chaos because it is uh very legalistic, very technical, sometimes by design.
00:03:47.039 --> 00:03:53.199
And very often the news cycle didn't follow the regulatory cycle.
00:03:53.439 --> 00:04:03.280
So you would find people who hear about a rule, then they go searching on Google to find what is this rule, how's it happening, and then they discover they can comment on it.
00:04:03.439 --> 00:04:09.599
But sometimes the news article passes, and that's five months after the drafted rule closed.
00:04:09.759 --> 00:04:16.079
So they'll just find any rule by the affected agency or who they think the affected agency will be.
00:04:16.560 --> 00:04:32.240
And whether you're for or against, they'll just go on there and start talking about guns, whether or not we need more or less regulation on guns, or uh something having to do with the retirement age or health care.
00:04:32.639 --> 00:04:39.279
Doesn't matter what the topic is, there's always something, but um, they just sort of stuff the ballot wherever they want.
00:04:39.519 --> 00:04:45.040
And is this um once the law is created before the regulations come out, or is it when they're thinking about a law?
00:04:45.199 --> 00:04:47.199
Like where in the legislative process does this happen?
00:04:47.519 --> 00:04:49.120
Uh law happens.
00:04:49.920 --> 00:04:56.079
Then there's some bidding about to see what agency has the band-aid over that law.
00:04:56.319 --> 00:05:01.439
So, you know, they'll pass the Clean Air Act and there'll be some discussions.
00:05:01.600 --> 00:05:07.759
Yeah, of course the EPA is gonna get it, but there might be some work with the Department of Transportation.
00:05:07.920 --> 00:05:13.120
So there'll be some high-level discussions about what regulations are needed and who should come out.
00:05:13.199 --> 00:05:20.079
And that's this kind of underscores the gap between the idea of a law and then the implementation of that law.
00:05:20.319 --> 00:05:36.079
One of the things we see in digital government is the government will pass a law like uh the Fee Transparency Act in Canada, which requires that any government department that collects a fee has to track whether the government delivered the thing they were supposed to for that fee, and if they didn't, then basically refund some of that fee.
00:05:36.240 --> 00:05:44.399
Now you can pass that law, but it's someone's got to build the tools to actually make that law happen, um, which is where the devil's in the details.
00:05:44.720 --> 00:05:47.839
How close is a law to the regulation?
00:05:48.000 --> 00:05:55.360
Do you have any sense of like the asymmetry between the law that gets passed and then the regulations that flow out from that, how many months it takes to do that?
00:05:55.680 --> 00:06:00.480
So I I almost heard you say years, and sometimes it can take years.
00:06:00.959 --> 00:06:02.879
Sometimes it takes days.
00:06:02.959 --> 00:06:08.319
It depends upon the relative awareness and importance of said law.
00:06:08.720 --> 00:06:17.279
Uh, in fact, if a thing is declared an emergency, I'd have to double check, but I think some of this might have happened in light of 9-11.
00:06:17.439 --> 00:06:20.240
You could skip the comment period altogether.
00:06:20.399 --> 00:06:25.680
Your justification into the politicians is just we need this now.
00:06:25.759 --> 00:06:26.800
There's no time for it.
00:06:27.040 --> 00:06:31.279
So you can actually intentionally bypass the entire regulations.gov process.
00:06:31.600 --> 00:06:36.160
As long as you're willing to justify it, which is usually a very high standard.
00:06:36.399 --> 00:06:36.879
For sure.
00:06:37.199 --> 00:06:43.920
But there is a comment period on most regulations that usually is about 60 to 90 days.
00:06:44.079 --> 00:06:56.720
So at the very least, the time between the draft and the final is so the system uses terms like docket and federal register notice, language that's like second nature inside government, but pretty alienating to anyone else.
00:06:57.279 --> 00:07:01.040
How much of a barrier was that jargon in practice when you started working on the system?
00:07:01.199 --> 00:07:04.399
Were you seeing patterns in like who could participate and who couldn't?
00:07:04.720 --> 00:07:17.519
Huge, huge barrier, big problem because of the language inside the law and the language of how the data and the information is structured, dockets, for example.
00:07:17.839 --> 00:07:40.879
Usually the comments that were being considered, and I separate that from submitted, were from either people who are lawyers, so they're professionals at this, they're from industry associations, they're from nonprofits, some sort of institutional or organizational backing.
00:07:41.120 --> 00:07:42.160
Or people who are just freely.
00:07:48.800 --> 00:07:49.040
Yeah.
00:07:49.120 --> 00:07:51.279
Yeah, it is definitely inside baseball.
00:07:51.360 --> 00:08:11.680
It's gotten a little bit better in the past, but because there's this lack of who really understands the context, I separate whose comments get really deeply considered from whose are submitted, because the average person who submits also doesn't know that it's not a vote.
00:08:11.839 --> 00:08:14.560
And that's usually where the mistake happens.
00:08:14.720 --> 00:08:25.920
So your example of the uh paid opposition eight point something million dollars to uh on on the FCC rule, most of that was actually wasted money.
00:08:26.160 --> 00:08:26.399
Right.
00:08:26.560 --> 00:08:32.559
Because unless it's a really valuable content, the comment that's useful, the reviewers don't they have to look at it, but they don't have to consider it, right?
00:08:32.879 --> 00:08:34.879
The first thing they say is it's not a vote.
00:08:35.039 --> 00:08:41.039
So they'll look at it, they'll say, Oh, this says blah, blah, blah, supports this, has no other information.
00:08:41.600 --> 00:08:41.840
Got it.
00:08:41.919 --> 00:08:46.399
So it doesn't actually provide new information on laws that have already been passed and presumably voted on.
00:08:46.639 --> 00:08:53.919
Or even not passed, it still has to provide new information about the law that they are intending, the regulation that they are intending to amp.
00:08:54.080 --> 00:08:54.320
Right.
00:08:54.480 --> 00:09:00.240
So on the other side, you got a bunch of reviewers, and they're legally required to read every single comment.
00:09:00.480 --> 00:09:00.720
Yes.
00:09:00.960 --> 00:09:05.759
Before you got involved, what did that workload look like when these kind of comment flooding campaigns hit?
00:09:06.000 --> 00:09:08.080
Well, it's incredibly hard.
00:09:08.159 --> 00:09:09.360
Uh, still is.
00:09:09.759 --> 00:09:28.799
You know, this is a constant work in progress, but you would see some organizations who might go out to their membership, and this is how some of the misattribution comments happen, and they'll say, We're doing a letter writing campaign to speak out for or against X.
00:09:29.360 --> 00:09:41.279
What they'll submit is an introductory paragraph about the organization, about 80% of it's the same, and then there's just a name change, and they'll do that a hundred thousand times.
00:09:41.440 --> 00:09:49.039
Sometimes it is six-figure submittals that are 80% it's the same, 20% customized, and a name change.
00:09:49.679 --> 00:09:54.879
And presumably that's about to change with AI when you can have a bunch of different responses, each of which looks unique.
00:09:55.120 --> 00:09:56.000
Right, right.
00:09:56.320 --> 00:09:58.399
Uh, that is very true.
00:09:58.639 --> 00:10:14.639
But the the I the the possibility of having to sort all these things and making sure that in the sorting there wasn't something critical in that 20% was costing agencies millions, tens of millions of dollars in some cases.
00:10:14.799 --> 00:10:15.120
Wow.
00:10:15.279 --> 00:10:16.639
In lawyer fees.
00:10:16.879 --> 00:10:20.000
Because you have to have a lawyer say, yes, this passes mustard.
00:10:20.159 --> 00:10:31.919
You could send it to the subject matter expert now who could actually consider whether Bob Davis's comment about gasoline is useful for the people who are putting in mileage regulations.
00:10:32.320 --> 00:10:38.960
So this sounds like a really great recipe for like a human denial of service attack where the law has already been passed.
00:10:39.120 --> 00:10:42.240
You're just talking about the interpretation of the law to the regulation.
00:10:42.720 --> 00:10:48.399
You flood the rink with a bunch of comments that say the same thing, even though the volume of comments doesn't help.
00:10:48.559 --> 00:10:48.720
Right.
00:10:48.879 --> 00:10:51.440
And then you wind up spending tens of millions more.
00:10:51.600 --> 00:10:59.679
And then, of course, people who want to attack your position can say, well, this regulator, this overregulation is costing us too much money, which becomes a self-fulfilling prophecy.
00:10:59.919 --> 00:11:00.159
Yeah.
00:11:00.320 --> 00:11:10.639
And as you put it with AI, now being having the capacity to do that sort of denial of service attack is pretty cheap and accessible.
00:11:10.879 --> 00:11:12.960
So anybody could do it.
00:11:13.039 --> 00:11:17.200
It's going to be sort of a there's potential there for escalating warfare.
00:11:17.519 --> 00:11:26.720
So as I understand it, this is a system where you're legally barred from changing both the submission workflow and the language of the proposed regulations themselves.
00:11:27.200 --> 00:11:35.600
So, like when you show up and you're told day one, you can't touch the two most obvious things that would actually solve this problem.
00:11:35.759 --> 00:11:37.440
What's the reaction like in that room?
00:11:37.759 --> 00:11:42.960
In the case of the actual language of the rule, that that that seems obvious.
00:11:43.039 --> 00:11:43.679
Who am I?
00:11:43.840 --> 00:11:45.039
I'm I'm nobody.
00:11:45.120 --> 00:11:46.639
I'm not going to change rule language.
00:11:46.720 --> 00:11:48.159
That's that's crazy talk.
00:11:48.559 --> 00:11:54.080
But it does seem like the government is gonna have to deal with what has essentially become a denial of service attack surface.
00:11:54.320 --> 00:11:58.240
Yeah, the the workflow, a little more upsetting, a little more of a surprise.
00:11:58.399 --> 00:12:05.120
So the the attitude is really just we have a lot of work and a lot of human testing to do.
00:12:05.360 --> 00:12:12.480
Because if you can't change the two most obvious things, you have to optimize your change literally everywhere else.
00:12:12.720 --> 00:12:13.039
Wow.
00:12:13.200 --> 00:12:17.360
Had previous teams tried to modernize uh the system before you got there?
00:12:17.679 --> 00:12:31.360
To limited success, this system was created back when a lot of enterprise architecture and a lot of data standards work was uh not really a thing.
00:12:31.600 --> 00:12:35.440
So the system at the time had literally no data standards.
00:12:35.600 --> 00:12:42.240
And I it was by design because this is a shared service for over 200 federal agencies.
00:12:42.399 --> 00:12:47.519
So their idea was, oh, well, we're just keeping it flexible for all of these people.
00:12:47.759 --> 00:12:53.679
Yeah, I I later met the person who instituted that, and I wanted to grab a drink and just ask them why.
00:12:54.000 --> 00:12:57.360
So, what was the trigger that caused them to like reach out to your team?
00:12:57.440 --> 00:12:59.200
And maybe you can talk a little about your team at the time.
00:12:59.360 --> 00:13:00.960
What was the trigger that got them to bring you in?
00:13:01.200 --> 00:13:03.519
Well, this was back in my federal service time.
00:13:03.679 --> 00:13:07.600
It was about eight years, I believe, that I spent with that team.
00:13:07.919 --> 00:13:16.399
And their trigger for modernizing at the time was they were on an edition of Drupal that was about to be deprecated.
00:13:16.480 --> 00:13:17.440
The the content manager.
00:13:17.840 --> 00:13:19.679
Drupal is a CMS for folks that don't know that.
00:13:19.759 --> 00:13:20.000
Yeah.
00:13:20.159 --> 00:13:20.320
Yeah.
00:13:20.480 --> 00:13:25.039
So they basically, the platform it was on, was old enough that it was about to be obsolete.
00:13:25.120 --> 00:13:32.399
And so they knew they had to rebuild something, and they thought, hey, it why if we're rebuilding it, maybe there are things we can fix, but don't touch the regulations and don't touch the process.
00:13:32.720 --> 00:13:32.879
Right.
00:13:32.960 --> 00:13:34.480
It's a great forcing event.
00:13:34.720 --> 00:13:35.440
Okay, fair.
00:13:35.679 --> 00:13:40.240
So can you put a number or at least some kind of shape on what the old system was costing?
00:13:40.399 --> 00:13:43.600
Not just in dollars, but like in democratic participation.
00:13:43.919 --> 00:13:51.440
Uh well, we took a sample of about a 17-month period, and it was just under three million comments at the time.
00:13:51.679 --> 00:14:01.840
Each one represents time put in, cost of just people caring about things and money, several millions of dollars in coordination cost.
00:14:02.000 --> 00:14:08.720
Once you count all of those federal agencies talking to each other, 200 federal agencies, the governance on that is a nightmare.
00:14:09.039 --> 00:14:22.399
But in democratic process, I don't think you can put a number on that because it's it's it's 200 federal agencies.
00:14:22.480 --> 00:14:26.799
It's every it's nearly every rulemaking the federal government has.
00:14:27.039 --> 00:14:34.799
And most people even today, even doing those kind of numbers, do not know that they have a right to add input.
00:14:35.120 --> 00:14:43.120
Yeah, and it seems like, I mean, I look at the feed, you know, the feed being like when you scroll Instagram or X or whatever you're scrolling, YouTube.
00:14:43.440 --> 00:14:46.879
A feed is like a system for curating things at scale.
00:14:47.120 --> 00:14:54.399
You can upvote something, you can retweet something or repost it, you can block it, you can DM the person to narrow the communication.
00:14:54.559 --> 00:14:59.440
But a feed is like a tool for millions of people to decide what's interesting and important.
00:14:59.679 --> 00:15:00.639
Even Reddit, right?
00:15:00.720 --> 00:15:06.879
You just upvote, downvote things that bubble to the top divided by how long they've been there, kind of gives you a sense of what's interesting.
00:15:07.039 --> 00:15:13.840
These are all mechanisms for processing information at scale that are at least somewhat resilient to attack.
00:15:14.159 --> 00:15:14.559
Right.
00:15:14.799 --> 00:15:17.279
But and they're much more accessible to the average person.
00:15:17.440 --> 00:15:27.919
You can imagine that you would go in and you know, say, these are the things I care about and see the laws that are related to what you're doing and you know, retweet or upvote or downvote the comments you like and add your own comments.
00:15:28.320 --> 00:15:31.600
Like we have a mechanism for this stuff that's far more participatory.
00:15:31.759 --> 00:15:35.600
Otherwise, without that kind of system, comment flooding is this crazy anti-pattern.
00:15:35.759 --> 00:15:39.279
People are weaponizing a democratic process specifically to slow it down.
00:15:39.759 --> 00:15:40.720
Absolutely.
00:15:41.039 --> 00:15:43.120
And sometimes it it works.
00:15:43.440 --> 00:15:45.919
So does it actually change outcomes or is it just friction?
00:15:46.240 --> 00:15:48.960
Sometimes it changes outcomes.
00:15:49.120 --> 00:15:50.960
It's not very common.
00:15:51.200 --> 00:16:08.639
Usually you'll see small things like extensions on rulemakings due to uh in a few in a few rare instances, it's been because something didn't publish on time, or they weren't able to put an update out on time, or there was downtime in the system.
00:16:08.799 --> 00:16:12.879
In the history of the system, it's been up maybe 99.9% of the time.
00:16:12.960 --> 00:16:16.879
But when it goes down, you have to legally extend the rulemaking period.
00:16:17.200 --> 00:16:17.679
Interesting.
00:16:17.919 --> 00:16:18.080
Yeah.
00:16:18.240 --> 00:16:28.639
And so your team, you were at like 18F uh government service administration stuff in the federal government in the US, had a particular way of working really focused on like human-centered design in a government context.
00:16:28.799 --> 00:16:28.960
Yes.
00:16:29.200 --> 00:16:37.919
How did that help you uh to figure out ways that you might fix it without changing either the wording, the legislation, or the process?
00:16:38.240 --> 00:16:47.919
So the thing that I did was I literally took my laptop and went to the food court and started going up to people.
00:16:48.080 --> 00:16:50.000
Hey, I'm testing this.
00:16:50.159 --> 00:16:52.000
I want more information about that.
00:16:52.159 --> 00:16:52.960
Can you tell me about it?
00:16:53.200 --> 00:16:57.279
Usually feds, because you can only bother so many public citizens with the same thing.
00:16:57.440 --> 00:17:02.000
I was gonna say, it's like Jonathan Haidt used to say that all psychological research was weird.
00:17:02.240 --> 00:17:09.279
Western, educated, independent, democratic, and rich, because all of the studies done at universities were done on university campuses.
00:17:09.359 --> 00:17:14.079
So he would like try and reproduce a study at a McDonald's three miles from campus and get a very different result.
00:17:14.720 --> 00:17:18.720
Your first round of sampling, you're already talking to people who are like inside the beltway.
00:17:18.799 --> 00:17:21.440
They're already baseball players in your inside baseball game, right?
00:17:21.680 --> 00:17:22.000
Yes.
00:17:22.240 --> 00:17:23.759
That happened a few times.
00:17:23.920 --> 00:17:30.160
Then I would go to conferences and sort of walk out from the hotel there and try to meet people.
00:17:30.400 --> 00:17:38.240
But the other part of that is when you talk about federal employees and other people you could test with, it doesn't have to all be beltway people.
00:17:38.400 --> 00:17:40.079
Make sure you get the security guard.
00:17:40.160 --> 00:17:44.480
Make sure you get the lady who's literally at the register of the food court.
00:17:44.640 --> 00:17:51.519
Still counts, still good information, still the exact kind of demo that has every right to use this tool.
00:17:51.759 --> 00:17:57.599
And so you would you would walk up and like open your laptop and say, hey, here's a potential regulation on something.
00:17:57.839 --> 00:17:59.039
How would you like to comment?
00:17:59.119 --> 00:18:00.160
Is that is that the approach?
00:18:00.480 --> 00:18:11.920
I would give them scenarios that are more along the lines of you just watched a news report and it mentions this regulation that is going to happen.
00:18:12.160 --> 00:18:16.400
You get curious, you want to add your input to it, you start off at Google.
00:18:16.480 --> 00:18:18.480
Sometimes I would start from the search engine.
00:18:18.640 --> 00:18:22.000
Find out where to comment, how you want to comment.
00:18:22.160 --> 00:18:23.440
Let me watch what you're doing.
00:18:23.759 --> 00:18:28.960
So you're even starting at the long funnel of just I care about how many guns there are, one way or another.
00:18:29.359 --> 00:18:30.960
And then what do you do about it?
00:18:31.039 --> 00:18:32.640
Did you know there's a thing you could do about it?
00:18:32.960 --> 00:18:34.000
I would vary it up.
00:18:34.160 --> 00:18:38.480
Sometimes it would be at the long funnel, sometimes it would be at the webpage itself.
00:18:38.720 --> 00:18:41.599
I want to mix the data, but that that would be the general idea.
00:18:41.920 --> 00:18:48.319
So let's talk about your two fixes because they're both pretty clever workarounds that fit within these tight legal constraints.
00:18:48.559 --> 00:18:48.799
Yes.
00:18:48.960 --> 00:18:52.480
You said your first one was bulk comment downloads for reviewers.
00:18:52.720 --> 00:18:53.200
Yes.
00:18:54.079 --> 00:18:54.960
Sounds pretty simple.
00:18:55.119 --> 00:18:57.839
Like, like, what does it take to ship that inside a federal system?
00:18:57.920 --> 00:19:00.240
Because bulk downloads sounds like a couple lines of code.
00:19:00.480 --> 00:19:00.720
Yeah.
00:19:00.880 --> 00:19:03.839
Uh ordinarily, it would be pretty simple.
00:19:04.000 --> 00:19:06.480
I I don't want to take sole credit for this idea.
00:19:06.720 --> 00:19:11.519
It was an idea that a lot of people needed and a lot of people requested.
00:19:11.599 --> 00:19:17.599
And at a certain critical depth, you just say, if this many people are asking for it, it must be for a reason.
00:19:17.759 --> 00:19:24.559
And the the bulk download of comments during the Drupal modernization is a little more difficult.
00:19:24.880 --> 00:19:25.920
So let me understand.
00:19:26.079 --> 00:19:32.640
The reviewer had to be on the system, look at one thing, then like click close or save or whatever, go to the next one.
00:19:32.720 --> 00:19:35.839
They couldn't look at them outside of the original system.
00:19:35.920 --> 00:19:36.400
Is that right?
00:19:36.559 --> 00:19:37.200
Uh yes.
00:19:37.519 --> 00:19:39.119
How did that affect their productivity?
00:19:39.359 --> 00:19:40.640
Oh, it's it's horrible.
00:19:40.799 --> 00:19:46.000
It's horrible, especially because very often you have related regulations.
00:19:46.160 --> 00:19:52.799
And within one regulation, you can have dozens of documents itself all in that one docket.
00:19:53.440 --> 00:19:58.799
And usually you have the majority of comments on one document, but not necessarily.
00:19:58.960 --> 00:20:04.799
So you have this separation of data where you're trying to find all the comments about rulemaking.
00:20:04.960 --> 00:20:08.480
You might have to go to another rulemaking, you might have to go to multiple documents.
00:20:08.640 --> 00:20:10.720
It becomes sort of a mapping make.
00:20:11.039 --> 00:20:19.839
So it sounds like the tool was made to let you review an individual comment, but not for the analysis of feedback across comments or synthesis of any kind.
00:20:20.160 --> 00:20:30.559
There are some tools that were and are used that help you find the highly copied comments, the ones that are 80% similar.
00:20:30.720 --> 00:20:39.039
You you put a sort of move a slider to 80%, it shows you that 80% language, then it highlights the 20% as you go through it.
00:20:39.200 --> 00:20:48.880
But in terms of getting mass sentiment analysis or breaking things up by topic or categorization, largely weren't there.
00:20:49.119 --> 00:20:56.240
It does seem to me like one opportunity to do this would be, you know, if you can't replace the existing system, create a parallel system.
00:20:56.480 --> 00:21:03.200
And I've seen tools like polis that are sort of designed for at-scale commenting where you pick your position on something or whatever.
00:21:03.359 --> 00:21:08.880
Or uh the Society Library has a thing where they take a topic and they break it apart into all of the different debate structures.
00:21:09.039 --> 00:21:21.359
Did anybody talk about building a parallel system to this one so that you were still respecting the letter of the law with regulations.gov, but then building another more analyzable system to kind of get the zeitgeist or the vibes from people?
00:21:21.599 --> 00:21:23.039
It's been talked about.
00:21:23.200 --> 00:21:27.119
I wouldn't be surprised if even today it was still worked on.
00:21:27.359 --> 00:21:30.720
This is something that's sort of always been in the background of the discussion.
00:21:30.799 --> 00:21:36.000
I'm sure they probably are still at the very least considering it, if not actively building it.
00:21:36.240 --> 00:21:38.799
There's a few difficulties there.
00:21:39.039 --> 00:21:44.640
One difficulty is the fee structure with how they pay for the service.
00:21:44.799 --> 00:21:52.720
There's 200 federal agencies, but the ones who really produce the majority of regulations are very top heavy.
00:21:52.960 --> 00:22:01.680
And when you have a shared service where there's only a few who do most of the demand, and everybody else who might produce One or two regulations.
00:22:02.400 --> 00:22:08.319
Paying for any sort of thing like that always raises the debate of, well, why isn't EPA covering?
00:22:08.640 --> 00:22:08.880
Got it.
00:22:08.960 --> 00:22:17.519
So you have to bill people according to usage, and there are a few big users that get to dictate the roadmap and stuff because that's the service they're paying for.
00:22:17.759 --> 00:22:27.279
Yeah, you either bill them according to usage or you try to spread it out, but you're going to have the complaints there of why am I floating the bill?
00:22:27.359 --> 00:22:29.440
Because EPA produces so many rules.
00:22:29.599 --> 00:22:32.880
I keep mentioning EPA only because they do produce a lot of rules.
00:22:33.039 --> 00:22:35.279
That's an observable fact.
00:22:35.519 --> 00:22:49.759
Um but either way, it makes trying to get additional funding for any fix, any parallel system that isn't going to be shared equally a major debate topic and something that is very often hard to approve.
00:22:50.240 --> 00:22:59.279
So the second fix that you had is a little more subtle: restructuring the underlying data to reduce the exposure to jargon without actually rewriting the regulations themselves.
00:22:59.359 --> 00:23:00.640
So you're not violating the law.
00:23:00.799 --> 00:23:02.079
Can you unpack that a little?
00:23:02.160 --> 00:23:04.559
Like what does restructuring the data mean in plain terms?
00:23:04.880 --> 00:23:07.440
Uh de-emphasizing the word docket.
00:23:07.599 --> 00:23:09.200
That that was a big start.
00:23:09.359 --> 00:23:16.960
Once again, if you don't get involved with court or cases or the legal system too often, you don't know what a docket is.
00:23:17.039 --> 00:23:18.079
It's a folder.
00:23:18.880 --> 00:23:25.279
And really, if I had the magic wand for a day, I would have just called it a folder myself.
00:23:25.519 --> 00:23:31.599
But we have government people who also use the system and they get attached to their vocabulary, so it becomes a big to-do.
00:23:31.839 --> 00:23:34.319
But it's a folder that you put documents in.
00:23:34.559 --> 00:23:36.319
That's more or less it.
00:23:36.880 --> 00:24:02.319
So doing that, making sure that the search is indexed so it's primarily by document and looking up the keywords and the subjects and the metadata of the document as opposed to the docket first is a big step in that direction because it means when you get a search result and you go there, you're in a document, you're in that area, that level of data that you understand.
00:24:02.480 --> 00:24:06.400
And you might not know what a docket is, but you'll understand the UX of it.
00:24:06.640 --> 00:24:08.000
And that'll help you navigate with it.
00:24:08.319 --> 00:24:12.480
You still got to keep the words there, but you can show people something they'll understand and connect with first.
00:24:12.720 --> 00:24:12.960
Yes.
00:24:13.200 --> 00:24:19.519
And you can make it easier for the reviewers to pull down many things at once and analyze them themselves.
00:24:19.599 --> 00:24:22.319
Although it does seem like the analysis tools are kind of far behind.
00:24:22.480 --> 00:24:30.559
There's like an asymmetry here where the commenters are allowed to use whatever automation they want, and the the reviewers have to, you know, pull out a piece of paper and read it.
00:24:30.720 --> 00:24:32.559
So it seems like you're pretty stuck there.
00:24:32.960 --> 00:24:34.640
They're they're behind, but they're getting better.
00:24:34.960 --> 00:24:37.200
Then there's the API layer for outside groups.
00:24:37.279 --> 00:24:39.759
You you have this data is public, right?
00:24:39.920 --> 00:24:45.680
And so while your hands are tied, the hands of civic organizations that care about these things are not.
00:24:45.839 --> 00:24:51.680
So you basically said we can't simplify the language, but we can make it easier for other people to simplify it for their communities.
00:24:51.920 --> 00:24:56.960
Was that something you framed as a workaround from the start, or did it kind of emerge as you were designing this stuff?
00:24:57.359 --> 00:25:06.160
It emerged because we were already getting a lot of headless browser submittals, people who were just sort of automating mass submittals.
00:25:06.480 --> 00:25:08.799
So can you clarify what a headless browser submittal is?
00:25:08.880 --> 00:25:09.119
Uh yeah.
00:25:09.440 --> 00:25:11.440
Because you're doing the inside baseball language there.
00:25:11.599 --> 00:25:12.160
Yes, you're right.
00:25:12.240 --> 00:25:12.559
You're right.
00:25:12.640 --> 00:25:13.440
Sorry about that.
00:25:13.680 --> 00:25:28.079
Uh so basically it is people who would send all of these comments over to us in a very automated and packaged fashion in such a way that it didn't even go over across a browser.
00:25:28.240 --> 00:25:30.079
It's just sort of computer to computer.
00:25:30.480 --> 00:25:35.599
It's a script that's just sending the messages and that that it knows it are needed to connect to the page and do it.
00:25:35.839 --> 00:25:44.079
Just kind of like what we see today with Cloud and Chrome, where Cloud can take over your browser, or just running Python commands that send web requests to a server.
00:25:44.160 --> 00:25:45.599
That stuff's even easier now than it was then.
00:25:45.839 --> 00:25:48.640
Yeah, just an automated script says push it over.
00:25:48.880 --> 00:25:49.200
Wow.
00:25:49.440 --> 00:25:51.279
We were already getting a lot of those.
00:25:51.440 --> 00:25:57.599
So many that you could see the dip in the system's performance or its traffic at times when there's a lot of different things.
00:25:57.759 --> 00:26:02.240
Oh, because they're using a web server that's designed for humans, but they're hammering it with a machine.
00:26:02.319 --> 00:26:02.559
Okay.
00:26:02.799 --> 00:26:03.440
Yes.
00:26:03.839 --> 00:26:11.200
Uh and when you we looked at who was doing it, there's some very legitimate organizations who are doing this.
00:26:11.359 --> 00:26:15.119
They have a legitimate business need to send over many at a time.
00:26:15.359 --> 00:26:16.480
So can you give me an example of that?
00:26:16.559 --> 00:26:21.200
Is like there's that like Ducks Unlimited had a thing on its website and now it wants to upload all those comments to you?
00:26:21.279 --> 00:26:21.920
Or what is that?
00:26:22.240 --> 00:26:22.960
Uh yeah, yeah.
00:26:23.039 --> 00:26:26.079
So we would get a veteran service organization.
00:26:26.319 --> 00:26:38.079
So somebody who's trying to represent veterans uh from the military across the country to make sure that a regulation establishes their correct benefits for healthcare.
00:26:38.240 --> 00:26:49.839
I don't want to name specific names or anything like that, but what they would do is they would collect information from their membership, which is tens of thousands, if not hundreds of thousands, of people.
00:26:50.000 --> 00:26:52.960
And they would say, okay, we're gonna send you out this form.
00:26:53.839 --> 00:26:59.119
They're going to change your prescription costs, throwing out hypotheticals here.
00:26:59.359 --> 00:27:04.160
Fill out this information in this form, add your name, we'll push it forward.
00:27:04.319 --> 00:27:12.480
And they want to make sure these people have a maximum amount of time to fill everything out, to put in the data that's required and to send it to them.
00:27:12.640 --> 00:27:21.119
And if they're gonna squeeze as much time out of that as they can, they need the quickest and most automated way to then send it to regulation stuff.
00:27:21.680 --> 00:27:27.200
I think the normal reaction would be hey, I've got this website, machines are using it, it's designed for humans.
00:27:27.359 --> 00:27:29.920
Let's make it as hard as possible for machines to use it.
00:27:30.000 --> 00:27:38.160
And websites today have these things like the robots.txt file that are basically instructions that say, hey, you're a computer, you're a Google crawler or whatever, you're not a human.
00:27:38.240 --> 00:27:40.400
You may not use this website in the way a human would.
00:27:40.640 --> 00:27:42.319
So there's a bit of an arms race, right?
00:27:42.480 --> 00:27:42.720
Yes.
00:27:42.880 --> 00:27:56.880
Um, I know that Cloudflare, one of the big content delivery companies that protects sites, has a thing they call a bot labyrinth, which is quite literally when it thinks you're a bot, it sends you into a bunch of generated pages full of nonsense to confuse you, right?
00:27:57.039 --> 00:28:02.880
So there's this very adversarial relationship between websites intended for humans and machines.
00:28:03.359 --> 00:28:12.559
You showed up and said counterintuitively, we think we should make it easier for machines to use this because they're gonna do it anyway, and this will solve some of our performance and load problems.
00:28:12.640 --> 00:28:13.039
Is that right?
00:28:13.279 --> 00:28:14.960
They're gonna do it anyways.
00:28:15.279 --> 00:28:19.119
For most of these anti-bot steps you can take.
00:28:19.279 --> 00:28:24.160
There's very publicly searchable and findable ways to beat them.
00:28:24.319 --> 00:28:38.079
So if we start implementing, you know, some sort of scheme to capture, click on all of these buses in this picture or something like that, we add accessibility problems and we only put up a problem that the cheaters could solve.
00:28:38.319 --> 00:28:38.559
Interesting.
00:28:38.640 --> 00:28:38.880
Yeah.
00:28:38.960 --> 00:28:42.640
So you put up you make it so hard for a machine that you've made it hard for humans.
00:28:42.880 --> 00:28:44.160
So let's skip that.
00:28:44.240 --> 00:28:44.559
Yeah.
00:28:44.640 --> 00:28:44.720
Yeah.
00:28:44.960 --> 00:28:45.519
How did that?
00:28:45.599 --> 00:28:46.880
I mean, that sounds counterintuitive.
00:28:46.960 --> 00:28:51.440
What was the initial reaction to please let's welcome our robot overlords to the conversation?
00:28:51.759 --> 00:29:07.519
Well, once we started getting the data on who was doing this and pointing to legitimate business need, legitimate need, these people who are representing real people with real problems and a real right to speak up, it starts to get a little more reasonable.
00:29:07.759 --> 00:29:12.400
Then when we started pointing out it there is a middle ground, you create a whitelist.
00:29:12.799 --> 00:29:23.519
You say, okay, you you advertise it, you say if you want access to this API, you sign up with us, we write down your name, we give you the API, the key, and we track you.
00:29:23.680 --> 00:29:30.960
We know exactly what you're doing, what quantities you're doing it in, and we can tell if it's a denial of shivers attack or content through that way.
00:29:31.279 --> 00:29:39.039
So you're you're giving people the right to register legitimate bots that should then allow that have a username and a password for a machine.
00:29:39.279 --> 00:29:39.680
Yes.
00:29:39.920 --> 00:29:40.079
Yeah.
00:29:40.400 --> 00:29:49.839
This seems like this is some serious foreshadowing for what we're going to need as people start to deploy like clawed code or claw or open open claw and things like that, right?
00:29:50.000 --> 00:29:50.319
Yeah.
00:29:50.640 --> 00:29:55.519
So uh how long did all this take from your first conversation to when it was live in production?
00:29:55.839 --> 00:29:56.480
Years.
00:29:57.680 --> 00:29:59.440
It took years.
00:29:59.680 --> 00:30:06.559
There's a lot of things happening in the middle of the modernization, a lot of budgetary problems.
00:30:06.720 --> 00:30:14.880
And then there was the FCC rule, which created a title shift in how this entire process is seen.
00:30:15.039 --> 00:30:20.880
Because when net neutrality came out, there were people who were not just sending automated comments.
00:30:21.039 --> 00:30:28.319
They were sending automated comments posing as other people in numbers that the government had probably never seen before.
00:30:28.480 --> 00:30:32.480
In some cases, even pretending to be congressmen who were sending in comments.
00:30:32.799 --> 00:30:36.000
I guess when you make a law that nerds care about, nerds are gonna nerd, right?
00:30:36.240 --> 00:30:36.559
Yes.
00:30:36.799 --> 00:30:37.039
Yeah.
00:30:37.279 --> 00:30:37.920
Big time.
00:30:38.079 --> 00:30:38.400
Big time.
00:30:38.880 --> 00:30:41.680
After you change shipped these changes, what did you see in the data?
00:30:41.839 --> 00:30:45.680
Did it change the diversity of commenters or the review cycle speed?
00:30:46.000 --> 00:30:48.640
In some cases, it changed the review cycle speed.
00:30:48.799 --> 00:30:52.720
I think this is still very much a work in progress.
00:30:53.599 --> 00:30:59.599
Uh, in some cases, it made life easier for those organizations who are using the API.
00:30:59.839 --> 00:31:09.279
And in a very non-political way, I can't emphasize that enough because you can't you can't play winners or losers with your API when you're the government.
00:31:09.440 --> 00:31:14.400
You have to make sure you give it to anybody who's just basically not completely misrepresenting the law.
00:31:14.720 --> 00:31:15.599
That's a great line.
00:31:15.759 --> 00:31:19.039
You can't um play winners or losers with API and you can't play politics.
00:31:19.200 --> 00:31:28.960
You know, the data, especially when it's data that's mandated for publication is part of the democratic process, that data and those those uh platforms, you have to be okay with your worst political enemy using this.
00:31:29.279 --> 00:31:30.799
Yes, absolutely.
00:31:31.039 --> 00:31:50.319
Um but putting it out there, making their life easier on both sides of the aisle, on everybody else, it's it's a continuation of that original idea behind regulations.gov, which is the government isn't gonna have all the answers or all parts of the solution.
00:31:50.960 --> 00:32:04.240
If I want to do a law and uh regulation and sub-nonprofit out there is gonna make 200,000 people more aware of this regulation, at some point the government should just say thank you.
00:32:04.480 --> 00:32:04.720
Yeah.
00:32:04.799 --> 00:32:11.680
I mean, it really is the democratic process in the digital age, but it's obviously been used in a way that is devastating for a lot of these systems.
00:32:11.839 --> 00:32:12.079
Yeah.
00:32:12.160 --> 00:32:12.400
Yeah.
00:32:12.640 --> 00:32:12.960
Okay.
00:32:13.119 --> 00:32:19.759
So uh for the nonprofit and advocacy groups that did use the API, did it actually change their participation?
00:32:19.920 --> 00:32:23.200
Do you see a rise in the proportion of comments that came from those groups?
00:32:23.519 --> 00:32:26.480
I believe so, but that is very hard to validate.
00:32:26.559 --> 00:32:34.559
It needs a long timeline because this is also subject to what laws and regulations that happen to be coming out.
00:32:35.359 --> 00:32:45.839
So uh a net neutrality is going to cause uh an ups it an uptick in that contribution of comments, irregardless.
00:32:46.160 --> 00:32:46.319
Sure.
00:32:46.559 --> 00:32:49.680
Anything that surprised you about the results and the impact?
00:32:50.079 --> 00:33:05.039
I think the thing that surprised me most is and I think you latched onto this very quickly, is the need for some sort of parallel system or better analysis tool to keep going.
00:33:05.759 --> 00:33:09.440
That this isn't this is by far not a solved or done problem.
00:33:09.759 --> 00:33:10.559
Well, it used to be Mr.
00:33:10.640 --> 00:33:12.160
Smith was going to go to Washington, right?
00:33:12.240 --> 00:33:16.640
Or Ottawa or wherever he's going, and now it's everyone is in their phones doing it.
00:33:16.880 --> 00:33:24.880
So much of government was created with this idea that the citizen would connect to a government person on an individual basis.
00:33:25.039 --> 00:33:31.680
There was never a sense of scale, there was another sense of automation, there was never a sense that the citizen would have access to these things themselves.
00:33:31.839 --> 00:33:46.480
I mean, it seems like this is the kind of thing where in a in a reasonably advanced democracy, if you cared about, for example, childcare, you could set an alert to find out about any time that any legislation was being passed where the regulations around childcare might change, and then you could go and comment on them, right?
00:33:46.640 --> 00:33:47.440
Yeah, yeah.
00:33:48.319 --> 00:34:01.519
And what really sort of then surprises me what fell out of this when we started taking a critical look at what goes into the system, what people are submitting for is also the federal land.
00:34:02.799 --> 00:34:15.840
And there's very often a disconnect between the laws and the regulations that come up and the stock of infrastructure that a government has.
00:34:16.159 --> 00:34:16.480
Right.
00:34:16.719 --> 00:34:22.719
So you'll see a lot of laws and regulations that say, oh, you have to notify the people.
00:34:22.880 --> 00:34:34.719
So there's a bunch of things that end up in the system that are not necessarily in the scope of a regulatory making, but it's a system and it counts as technically notifying the people.
00:34:35.039 --> 00:34:36.239
So it's more of an ass covering.
00:34:36.320 --> 00:34:37.599
You're like, hey, I did, I told them.
00:34:37.679 --> 00:34:37.760
Yeah.
00:34:38.960 --> 00:34:39.119
Yeah.
00:34:39.199 --> 00:34:39.360
Okay.
00:34:39.679 --> 00:34:40.000
Yes.
00:34:40.079 --> 00:34:48.400
And so separating the notification that something is happening from the I want to have a reason debate and see if I've missed anything intense is kind of huge.
00:34:48.880 --> 00:34:49.679
Enormous.
00:34:50.000 --> 00:34:50.559
Enormous.
00:34:50.639 --> 00:34:50.880
Yeah.
00:34:51.199 --> 00:35:18.079
It does seem to me like it would be fairly easy, given that you have an API, for someone to register as a legitimate connector via the API, for them to create a website where any person could go and say, I'm interested in these things, for them to scrape the dockets, match them to things people are interested in, send those people a message, have them click to the website, show them just the things they want, maybe even translate it out of jargon, fill in the stuff, and then send it back to your website.
00:35:18.320 --> 00:35:20.320
Like we don't have to wait for the government to do that.
00:35:20.480 --> 00:35:26.079
I wrote a thing a couple of weeks ago called the Machine Fights Back about how humans used to have to grind up against the machinery government.
00:35:26.159 --> 00:35:34.639
It seems like it'd be pretty easy to build a publish and subscribe model for people that just made these things transparent and let you comment on them.
00:35:34.880 --> 00:35:35.519
Yes.
00:35:35.840 --> 00:35:37.760
Uh 100% agreed.
00:35:37.920 --> 00:35:39.440
It's a good business model.
00:35:39.760 --> 00:35:40.239
Business model.
00:35:40.559 --> 00:35:41.599
These days you can just do it for free.
00:35:41.679 --> 00:35:42.960
Like I don't want to make money from it.
00:35:43.280 --> 00:35:51.280
Well, it's a good when I say business model, it's more in terms of it's a good thing you can do to drive a result, money or not.
00:35:51.440 --> 00:36:04.079
I know people who have been looking at doing things like that, lobbyist organizations, nonprofits, who really just want to have a better dashboard of government things that their people can care about.
00:36:04.320 --> 00:36:07.599
And there's no way for me to find out what happened to my comment, right?
00:36:07.920 --> 00:36:20.239
Uh usually what will happen is they produce a final rule and it will to some degree address the comments received.
00:36:20.559 --> 00:36:24.000
Sometimes it'll just say, oh, we got comments and we considered them.
00:36:24.159 --> 00:36:26.480
Sometimes it'll break it out by the individual comment.
00:36:26.639 --> 00:36:29.119
It really depends upon who the rule writer is.
00:36:29.440 --> 00:36:29.599
Okay.
00:36:29.760 --> 00:36:33.360
Let's switch selfishly to Canada now, because we have a similar system here.
00:36:33.440 --> 00:36:35.519
Obviously, the here the Canada Gazette.
00:36:35.599 --> 00:36:40.480
It's where proposed regulations are published and public comment is supposed to be part of the process.
00:36:40.639 --> 00:36:43.199
But in our case, participation is vanishingly small.
00:36:43.360 --> 00:36:43.519
Yes.
00:36:43.760 --> 00:36:48.480
If you were advising a Canadian team facing similar constraints, what's the first thing you'd tell them?
00:36:48.880 --> 00:36:58.000
Uh it it participation in the system is small, but there are a lot of ways to participate in providing input.
00:36:58.159 --> 00:37:00.320
Is it happening somewhere?
00:37:01.199 --> 00:37:03.280
That that's the first thing I would want to know.
00:37:03.440 --> 00:37:13.039
You know, if there's more public meetings that get publicized in the newspaper or the news, or just through an email.
00:37:13.599 --> 00:37:16.000
I don't care how the participation is happening.
00:37:16.079 --> 00:37:22.400
The tool doesn't justify itself unless that unless people gravitate towards participation in that.
00:37:22.800 --> 00:37:23.119
Got it.
00:37:23.280 --> 00:37:33.840
So you spent a decade at the EPA, you spent years at GSA, and now you're at Ishmael Interactive, which was a firm that was built by Ford Former ATF people after Doge shut things down.
00:37:34.000 --> 00:37:39.679
What's been lost with the dismantling of those kinds of human-centered design institutions within government?
00:37:40.079 --> 00:37:41.440
Uh the context.
00:37:41.840 --> 00:37:50.559
The people who would go out and get the context of what government is doing to the people who it's supposed to serve.
00:37:50.719 --> 00:38:02.400
You know, 18F, great office, love their work, but when they made a fix for paying your taxes, they started off by talking to people who were paying their taxes.
00:38:02.559 --> 00:38:02.880
Yeah.
00:38:03.039 --> 00:38:05.519
You can't get that out of just an AI.
00:38:06.079 --> 00:38:07.360
Quite the same.
00:38:07.519 --> 00:38:10.000
Not only that, but just the talent pool.
00:38:10.159 --> 00:38:12.639
You just an incredible talent pool.
00:38:12.800 --> 00:38:14.639
I don't know how they're going to restock.
00:38:15.039 --> 00:38:23.440
I a lot of those changes are also scary just because they did it to such a degree all at once.
00:38:24.320 --> 00:38:35.679
Even if you cut the right people, which personally I think is highly debatable, if you cut X percent of them all at once, it's going to create problems that you have to learn how to bounce back.
00:38:36.079 --> 00:38:39.039
You lose that sort of organizational muscle mass and memory, right?
00:38:39.360 --> 00:38:40.159
Yeah, absolutely.
00:38:40.480 --> 00:38:48.559
So regulations.gov is a case where you couldn't change the law, you couldn't change the language, and you still found a way to, in theory, make democracy more accessible.
00:38:48.800 --> 00:38:52.320
What's the lesson here for people who feel like government is too big and too broken to fix?
00:38:52.559 --> 00:38:53.920
You have a part in it.
00:38:54.079 --> 00:38:57.360
And that's the thing I love about regulations.gov.
00:38:57.679 --> 00:39:08.559
And that's the thing I love about the human testing, is there's usually somebody you could get involved with and relay your experience to.
00:39:08.800 --> 00:39:11.760
If it is going on the gazette, great.
00:39:11.920 --> 00:39:21.840
If it is calling your mayor and saying, I want five minutes of your time, it all starts with transmitting what your personal context is.
00:39:22.000 --> 00:39:23.199
How does this affect me?
00:39:23.519 --> 00:39:30.239
We talked a little bit about AI and automation, and obviously that's something that's happening on a weekly or even daily basis.
00:39:31.119 --> 00:39:40.320
Um, what are your thoughts about what happens when it becomes obvious that the citizens have their own machines and they can point it at government if they don't want it, don't like what it's doing?
00:39:40.800 --> 00:39:42.239
Well, there's two different thoughts.
00:39:42.400 --> 00:39:47.519
One is I'm trying to separate oppositional content from denial of service.
00:39:47.760 --> 00:39:50.639
Denial of service, bad oppositional content.
00:39:50.800 --> 00:39:53.119
As long as it's true, I don't care.
00:39:53.360 --> 00:39:58.320
I'm not supposed to pick sides, government's not supposed to pick sides, that is fine.
00:39:58.559 --> 00:40:08.320
If it is creating a substantive comment or substantive input with actual facts that can be researched, this is no different than using a keyboard instead of a pen.
00:40:08.639 --> 00:40:14.880
The tools that allow scale are separate from the legitimacy of the underlying message or goal.
00:40:15.199 --> 00:40:15.440
That's right.
00:40:15.920 --> 00:40:16.800
It's content first.
00:40:17.119 --> 00:40:27.679
And you also push back against this idea that the website was a reliable source of friction that forced humans to do this firsthand, both as reviewers and as submitters, right?
00:40:27.840 --> 00:40:29.440
Because it was already being circumvented.
00:40:29.599 --> 00:40:32.559
Do you see governments having a similar thought about AI?
00:40:32.639 --> 00:40:33.760
Are they not even thinking about it?
00:40:33.920 --> 00:40:36.079
Are they trying to send bots into labyrinths?
00:40:36.159 --> 00:40:42.239
Or are they now saying, no, we need a skill for the federal government that you can tell Claude about so it knows how to respond?
00:40:42.559 --> 00:40:58.480
I think that this is one of the things the current administration is getting right, is that they're paying more attention to automation and AI in a way that is trying to put it in the workflow, whether they do that successfully or not, it's my government.
00:40:58.559 --> 00:40:59.519
So I hope they do.
00:40:59.760 --> 00:41:02.960
But I think that there's still some fear.
00:41:03.119 --> 00:41:14.480
And I'll I'll I'll take you back to the FCC rule again and the ability to impersonate another person and put your comment onto somebody else's name.
00:41:14.639 --> 00:41:21.920
If you talk to most of the rule writers at the time, the people who actually make the regulations, they didn't care whose name was on it.
00:41:22.079 --> 00:41:25.760
You were committing fraud for something that had no effect.
00:41:26.159 --> 00:41:28.480
Was it part of the consideration set?
00:41:28.719 --> 00:41:29.760
It's the content.
00:41:29.840 --> 00:41:31.679
It can have any name on it.
00:41:31.760 --> 00:41:32.480
It doesn't matter.
00:41:32.800 --> 00:41:33.519
Yeah, it's interesting.
00:41:33.599 --> 00:41:35.840
It wasn't a vote, so there was no question of legitimacy.
00:41:36.000 --> 00:41:36.239
Exactly.
00:41:36.480 --> 00:41:39.840
It was a comment, and that comment stands on its own regardless of who made it.
00:41:40.079 --> 00:41:40.400
Yeah.
00:41:40.639 --> 00:41:45.519
Well, you know, Canada's just launched online commenting in September 2022.
00:41:45.599 --> 00:41:53.519
We're pretty far behind where the US was, but we're not seeing anywhere near the numbers you saw, you know, around 1,500 people for an open government debate.
00:41:53.760 --> 00:41:57.760
What do you think it is that's gonna make Canadians realize they too have a voice?
00:41:58.159 --> 00:42:00.559
So I'm not sure that they don't realize this.
00:42:00.719 --> 00:42:05.440
They might be getting their civic participation and their civic input somewhere else.
00:42:05.679 --> 00:42:10.800
So the first thing I would do is get a Canadian sit down and say, How do you participate?
00:42:12.159 --> 00:42:16.559
If they're not participating through the Gazette, I don't know what it might be.
00:42:16.719 --> 00:42:29.599
Usually it is, but usually when I see a bump up in people who start to then comment to an agency or a group of agencies or on a topic, it takes one big law.
00:42:30.320 --> 00:42:46.480
One big earth-shattering law that gets in the news, gets on your talk shows, then people come in, they find it, they comment, and then the week after they remember, oh wait, there might be other rules and other regulations about this stuff.
00:42:46.719 --> 00:42:48.239
So I'm just gonna go once a week.
00:42:48.480 --> 00:42:52.320
Well, you've kind of made me want to go build the social network for the Canada Gazette now.
00:42:52.400 --> 00:42:55.039
So uh maybe when I have a week off, I'll go try and build that.
00:42:55.280 --> 00:42:57.440
Aaron, thank you so much for sharing this story with us.
00:42:57.599 --> 00:43:08.639
It's uh it's a fascinating example of something that is like you're trying to make this thing work, but your hands are tied in the ways that most people would see are obvious.
00:43:08.800 --> 00:43:19.840
So you're like doing what you can, but my my developer brain just gets furious at the at how outdated this system is when we, you know, Reddit is better at handling comments than this system.
00:43:20.079 --> 00:43:20.880
Yeah, excellent.
00:43:20.960 --> 00:43:21.199
Thank you.
00:43:21.440 --> 00:43:22.159
Thanks a lot, Aaron.
00:43:22.320 --> 00:43:22.639
Awesome.
00:43:22.800 --> 00:43:25.920
Thanks for joining us on the Functional Government Podcast.
00:00:00.000 --> 00:00:03.520
What's the lesson here for people who feel like government is too big and too broken to fix?
00:00:03.759 --> 00:00:10.000
If you can't change the two most obvious things, you have to optimize your change literally everywhere else.
00:00:10.240 --> 00:00:13.599
People are weaponizing a democratic process specifically to slow it down.
00:00:13.839 --> 00:00:19.679
I literally took my laptop and went to the food court and started going up to people.
00:00:20.000 --> 00:00:23.280
Welcome to the Functional Government Podcast.
00:00:23.440 --> 00:00:30.879
Most people have never heard of regulations.gov, but it's where ordinary citizens get to push back on rules that will affect their lives.
00:00:31.120 --> 00:00:34.399
It's been operating since 2003 and its scale is enormous.
00:00:34.640 --> 00:00:39.920
The FCC's net neutrality rulemaking received over 20 million comments.
00:00:40.079 --> 00:00:44.560
Yet of those 22 million, the New York Attorney General labeled 18 million as fakes.
00:00:44.719 --> 00:00:55.840
One college student filed 7.7 million comments in support of keeping the rules, while internet service providers paid consulting firms$8.2 million to generate opposition content.
00:00:56.000 --> 00:01:00.399
The EPA's Clean Power Plan drew 4.3 million comments.
00:01:00.640 --> 00:01:03.359
Canada has its own version of this called the Canada Gazette.
00:01:03.520 --> 00:01:08.239
Most people don't know about it, but publishing something in the Gazette is considered official notice.
00:01:08.480 --> 00:01:15.200
Canada didn't even have online commenting until September 2022, following a 2021 pilot.
00:01:15.439 --> 00:01:18.480
Before that, comments came only by mail or email.
00:01:18.640 --> 00:01:23.519
In fact, it was the Kuzma Trade Agreement that pushed Canada to build what the U.S.
00:01:23.680 --> 00:01:25.280
had had for nearly 20 years.
00:01:25.439 --> 00:01:26.959
The Gazette doesn't get much use.
00:01:27.120 --> 00:01:39.200
The Air Passenger Protection Regulations, which was one of the most consumer-facing high-profile consultations in recent years, received around 300 submissions, which is pretty tiny for a country that spends a lot of time complaining about air travel.
00:01:39.359 --> 00:01:49.040
A fee proposal for air travel complaints received 83 written submissions, 30 from the public, 46 from airlines, six from consumer groups, and one from a territorial government.
00:01:49.200 --> 00:01:51.680
And a six-month consultation on open government.
00:01:51.760 --> 00:01:57.680
A group you'd expect to use a lot of open government tools like the Gazette had 1,451 participants.
00:01:57.840 --> 00:02:03.519
Before the online system, departments were tracking comments using Excel spreadsheets and email folders.
00:02:03.680 --> 00:02:09.599
The project team had to do weekly bilingual training sessions because regulators had no common system.
00:02:09.759 --> 00:02:13.199
But we're in a new world now with automation and AI changing the game.
00:02:13.360 --> 00:02:19.039
Regulations.gov is now grappling with AI-generated comments on top of mass comment campaigns.
00:02:19.280 --> 00:02:19.840
One U.S.
00:02:19.919 --> 00:02:28.159
government study found that from 5 to 30% of people whose emails were attached to public comments said they didn't actually make the comment.
00:02:28.400 --> 00:02:32.240
This is the exact problem our guest and his team were hired to address.
00:02:32.319 --> 00:02:37.120
So please welcome Aaron Myers, the chief data officer of Ishmael Interactive.
00:02:37.360 --> 00:02:37.599
Hello.
00:02:37.840 --> 00:02:38.000
Hi.
00:02:38.159 --> 00:02:39.199
Hey Aaron, how are you?
00:02:39.520 --> 00:02:40.000
Pretty good.
00:02:40.159 --> 00:02:40.560
Pretty good.
00:02:40.639 --> 00:02:41.280
Thanks for having me.
00:02:41.599 --> 00:02:45.680
Can you explain what regulations gov does and why it matters?
00:02:45.919 --> 00:02:49.680
As if you were explaining it to someone who's never interacted with government beyond paying taxes.
00:02:50.000 --> 00:02:51.039
Excellent, excellent.
00:02:51.599 --> 00:03:04.719
So regulations.gov is the idea that as a law gets passed and they have to make rules and regulations to say here's how we follow the law, the government's not going to know everything.
00:03:04.879 --> 00:03:09.280
They usually don't, but there are very few places where the government's going to say that out loud.
00:03:09.599 --> 00:03:17.039
So they put up a draft and they say, here's my list of questions, which would help me make this rule much better.
00:03:17.199 --> 00:03:28.159
And they're hoping people will go there, find it, answer it, fill out the information that they're looking for, give them a little more completeness before that draft becomes final.
00:03:28.560 --> 00:03:35.599
When you first encountered regulations.gov, what was the experience like for an average person trying to do a comment on a proposed rule?
00:03:35.680 --> 00:03:37.360
Like what did they actually have to see and do?
00:03:37.599 --> 00:03:46.400
Well, it's it's a little bit chaos because it is uh very legalistic, very technical, sometimes by design.
00:03:47.039 --> 00:03:53.199
And very often the news cycle didn't follow the regulatory cycle.
00:03:53.439 --> 00:04:03.280
So you would find people who hear about a rule, then they go searching on Google to find what is this rule, how's it happening, and then they discover they can comment on it.
00:04:03.439 --> 00:04:09.599
But sometimes the news article passes, and that's five months after the drafted rule closed.
00:04:09.759 --> 00:04:16.079
So they'll just find any rule by the affected agency or who they think the affected agency will be.
00:04:16.560 --> 00:04:32.240
And whether you're for or against, they'll just go on there and start talking about guns, whether or not we need more or less regulation on guns, or uh something having to do with the retirement age or health care.
00:04:32.639 --> 00:04:39.279
Doesn't matter what the topic is, there's always something, but um, they just sort of stuff the ballot wherever they want.
00:04:39.519 --> 00:04:45.040
And is this um once the law is created before the regulations come out, or is it when they're thinking about a law?
00:04:45.199 --> 00:04:47.199
Like where in the legislative process does this happen?
00:04:47.519 --> 00:04:49.120
Uh law happens.
00:04:49.920 --> 00:04:56.079
Then there's some bidding about to see what agency has the band-aid over that law.
00:04:56.319 --> 00:05:01.439
So, you know, they'll pass the Clean Air Act and there'll be some discussions.
00:05:01.600 --> 00:05:07.759
Yeah, of course the EPA is gonna get it, but there might be some work with the Department of Transportation.
00:05:07.920 --> 00:05:13.120
So there'll be some high-level discussions about what regulations are needed and who should come out.
00:05:13.199 --> 00:05:20.079
And that's this kind of underscores the gap between the idea of a law and then the implementation of that law.
00:05:20.319 --> 00:05:36.079
One of the things we see in digital government is the government will pass a law like uh the Fee Transparency Act in Canada, which requires that any government department that collects a fee has to track whether the government delivered the thing they were supposed to for that fee, and if they didn't, then basically refund some of that fee.
00:05:36.240 --> 00:05:44.399
Now you can pass that law, but it's someone's got to build the tools to actually make that law happen, um, which is where the devil's in the details.
00:05:44.720 --> 00:05:47.839
How close is a law to the regulation?
00:05:48.000 --> 00:05:55.360
Do you have any sense of like the asymmetry between the law that gets passed and then the regulations that flow out from that, how many months it takes to do that?
00:05:55.680 --> 00:06:00.480
So I I almost heard you say years, and sometimes it can take years.
00:06:00.959 --> 00:06:02.879
Sometimes it takes days.
00:06:02.959 --> 00:06:08.319
It depends upon the relative awareness and importance of said law.
00:06:08.720 --> 00:06:17.279
Uh, in fact, if a thing is declared an emergency, I'd have to double check, but I think some of this might have happened in light of 9-11.
00:06:17.439 --> 00:06:20.240
You could skip the comment period altogether.
00:06:20.399 --> 00:06:25.680
Your justification into the politicians is just we need this now.
00:06:25.759 --> 00:06:26.800
There's no time for it.
00:06:27.040 --> 00:06:31.279
So you can actually intentionally bypass the entire regulations.gov process.
00:06:31.600 --> 00:06:36.160
As long as you're willing to justify it, which is usually a very high standard.
00:06:36.399 --> 00:06:36.879
For sure.
00:06:37.199 --> 00:06:43.920
But there is a comment period on most regulations that usually is about 60 to 90 days.
00:06:44.079 --> 00:06:56.720
So at the very least, the time between the draft and the final is so the system uses terms like docket and federal register notice, language that's like second nature inside government, but pretty alienating to anyone else.
00:06:57.279 --> 00:07:01.040
How much of a barrier was that jargon in practice when you started working on the system?
00:07:01.199 --> 00:07:04.399
Were you seeing patterns in like who could participate and who couldn't?
00:07:04.720 --> 00:07:17.519
Huge, huge barrier, big problem because of the language inside the law and the language of how the data and the information is structured, dockets, for example.
00:07:17.839 --> 00:07:40.879
Usually the comments that were being considered, and I separate that from submitted, were from either people who are lawyers, so they're professionals at this, they're from industry associations, they're from nonprofits, some sort of institutional or organizational backing.
00:07:41.120 --> 00:07:42.160
Or people who are just freely.
00:07:48.800 --> 00:07:49.040
Yeah.
00:07:49.120 --> 00:07:51.279
Yeah, it is definitely inside baseball.
00:07:51.360 --> 00:08:11.680
It's gotten a little bit better in the past, but because there's this lack of who really understands the context, I separate whose comments get really deeply considered from whose are submitted, because the average person who submits also doesn't know that it's not a vote.
00:08:11.839 --> 00:08:14.560
And that's usually where the mistake happens.
00:08:14.720 --> 00:08:25.920
So your example of the uh paid opposition eight point something million dollars to uh on on the FCC rule, most of that was actually wasted money.
00:08:26.160 --> 00:08:26.399
Right.
00:08:26.560 --> 00:08:32.559
Because unless it's a really valuable content, the comment that's useful, the reviewers don't they have to look at it, but they don't have to consider it, right?
00:08:32.879 --> 00:08:34.879
The first thing they say is it's not a vote.
00:08:35.039 --> 00:08:41.039
So they'll look at it, they'll say, Oh, this says blah, blah, blah, supports this, has no other information.
00:08:41.600 --> 00:08:41.840
Got it.
00:08:41.919 --> 00:08:46.399
So it doesn't actually provide new information on laws that have already been passed and presumably voted on.
00:08:46.639 --> 00:08:53.919
Or even not passed, it still has to provide new information about the law that they are intending, the regulation that they are intending to amp.
00:08:54.080 --> 00:08:54.320
Right.
00:08:54.480 --> 00:09:00.240
So on the other side, you got a bunch of reviewers, and they're legally required to read every single comment.
00:09:00.480 --> 00:09:00.720
Yes.
00:09:00.960 --> 00:09:05.759
Before you got involved, what did that workload look like when these kind of comment flooding campaigns hit?
00:09:06.000 --> 00:09:08.080
Well, it's incredibly hard.
00:09:08.159 --> 00:09:09.360
Uh, still is.
00:09:09.759 --> 00:09:28.799
You know, this is a constant work in progress, but you would see some organizations who might go out to their membership, and this is how some of the misattribution comments happen, and they'll say, We're doing a letter writing campaign to speak out for or against X.
00:09:29.360 --> 00:09:41.279
What they'll submit is an introductory paragraph about the organization, about 80% of it's the same, and then there's just a name change, and they'll do that a hundred thousand times.
00:09:41.440 --> 00:09:49.039
Sometimes it is six-figure submittals that are 80% it's the same, 20% customized, and a name change.
00:09:49.679 --> 00:09:54.879
And presumably that's about to change with AI when you can have a bunch of different responses, each of which looks unique.
00:09:55.120 --> 00:09:56.000
Right, right.
00:09:56.320 --> 00:09:58.399
Uh, that is very true.
00:09:58.639 --> 00:10:14.639
But the the I the the possibility of having to sort all these things and making sure that in the sorting there wasn't something critical in that 20% was costing agencies millions, tens of millions of dollars in some cases.
00:10:14.799 --> 00:10:15.120
Wow.
00:10:15.279 --> 00:10:16.639
In lawyer fees.
00:10:16.879 --> 00:10:20.000
Because you have to have a lawyer say, yes, this passes mustard.
00:10:20.159 --> 00:10:31.919
You could send it to the subject matter expert now who could actually consider whether Bob Davis's comment about gasoline is useful for the people who are putting in mileage regulations.
00:10:32.320 --> 00:10:38.960
So this sounds like a really great recipe for like a human denial of service attack where the law has already been passed.
00:10:39.120 --> 00:10:42.240
You're just talking about the interpretation of the law to the regulation.
00:10:42.720 --> 00:10:48.399
You flood the rink with a bunch of comments that say the same thing, even though the volume of comments doesn't help.
00:10:48.559 --> 00:10:48.720
Right.
00:10:48.879 --> 00:10:51.440
And then you wind up spending tens of millions more.
00:10:51.600 --> 00:10:59.679
And then, of course, people who want to attack your position can say, well, this regulator, this overregulation is costing us too much money, which becomes a self-fulfilling prophecy.
00:10:59.919 --> 00:11:00.159
Yeah.
00:11:00.320 --> 00:11:10.639
And as you put it with AI, now being having the capacity to do that sort of denial of service attack is pretty cheap and accessible.
00:11:10.879 --> 00:11:12.960
So anybody could do it.
00:11:13.039 --> 00:11:17.200
It's going to be sort of a there's potential there for escalating warfare.
00:11:17.519 --> 00:11:26.720
So as I understand it, this is a system where you're legally barred from changing both the submission workflow and the language of the proposed regulations themselves.
00:11:27.200 --> 00:11:35.600
So, like when you show up and you're told day one, you can't touch the two most obvious things that would actually solve this problem.
00:11:35.759 --> 00:11:37.440
What's the reaction like in that room?
00:11:37.759 --> 00:11:42.960
In the case of the actual language of the rule, that that that seems obvious.
00:11:43.039 --> 00:11:43.679
Who am I?
00:11:43.840 --> 00:11:45.039
I'm I'm nobody.
00:11:45.120 --> 00:11:46.639
I'm not going to change rule language.
00:11:46.720 --> 00:11:48.159
That's that's crazy talk.
00:11:48.559 --> 00:11:54.080
But it does seem like the government is gonna have to deal with what has essentially become a denial of service attack surface.
00:11:54.320 --> 00:11:58.240
Yeah, the the workflow, a little more upsetting, a little more of a surprise.
00:11:58.399 --> 00:12:05.120
So the the attitude is really just we have a lot of work and a lot of human testing to do.
00:12:05.360 --> 00:12:12.480
Because if you can't change the two most obvious things, you have to optimize your change literally everywhere else.
00:12:12.720 --> 00:12:13.039
Wow.
00:12:13.200 --> 00:12:17.360
Had previous teams tried to modernize uh the system before you got there?
00:12:17.679 --> 00:12:31.360
To limited success, this system was created back when a lot of enterprise architecture and a lot of data standards work was uh not really a thing.
00:12:31.600 --> 00:12:35.440
So the system at the time had literally no data standards.
00:12:35.600 --> 00:12:42.240
And I it was by design because this is a shared service for over 200 federal agencies.
00:12:42.399 --> 00:12:47.519
So their idea was, oh, well, we're just keeping it flexible for all of these people.
00:12:47.759 --> 00:12:53.679
Yeah, I I later met the person who instituted that, and I wanted to grab a drink and just ask them why.
00:12:54.000 --> 00:12:57.360
So, what was the trigger that caused them to like reach out to your team?
00:12:57.440 --> 00:12:59.200
And maybe you can talk a little about your team at the time.
00:12:59.360 --> 00:13:00.960
What was the trigger that got them to bring you in?
00:13:01.200 --> 00:13:03.519
Well, this was back in my federal service time.
00:13:03.679 --> 00:13:07.600
It was about eight years, I believe, that I spent with that team.
00:13:07.919 --> 00:13:16.399
And their trigger for modernizing at the time was they were on an edition of Drupal that was about to be deprecated.
00:13:16.480 --> 00:13:17.440
The the content manager.
00:13:17.840 --> 00:13:19.679
Drupal is a CMS for folks that don't know that.
00:13:19.759 --> 00:13:20.000
Yeah.
00:13:20.159 --> 00:13:20.320
Yeah.
00:13:20.480 --> 00:13:25.039
So they basically, the platform it was on, was old enough that it was about to be obsolete.
00:13:25.120 --> 00:13:32.399
And so they knew they had to rebuild something, and they thought, hey, it why if we're rebuilding it, maybe there are things we can fix, but don't touch the regulations and don't touch the process.
00:13:32.720 --> 00:13:32.879
Right.
00:13:32.960 --> 00:13:34.480
It's a great forcing event.
00:13:34.720 --> 00:13:35.440
Okay, fair.
00:13:35.679 --> 00:13:40.240
So can you put a number or at least some kind of shape on what the old system was costing?
00:13:40.399 --> 00:13:43.600
Not just in dollars, but like in democratic participation.
00:13:43.919 --> 00:13:51.440
Uh well, we took a sample of about a 17-month period, and it was just under three million comments at the time.
00:13:51.679 --> 00:14:01.840
Each one represents time put in, cost of just people caring about things and money, several millions of dollars in coordination cost.
00:14:02.000 --> 00:14:08.720
Once you count all of those federal agencies talking to each other, 200 federal agencies, the governance on that is a nightmare.
00:14:09.039 --> 00:14:22.399
But in democratic process, I don't think you can put a number on that because it's it's it's 200 federal agencies.
00:14:22.480 --> 00:14:26.799
It's every it's nearly every rulemaking the federal government has.
00:14:27.039 --> 00:14:34.799
And most people even today, even doing those kind of numbers, do not know that they have a right to add input.
00:14:35.120 --> 00:14:43.120
Yeah, and it seems like, I mean, I look at the feed, you know, the feed being like when you scroll Instagram or X or whatever you're scrolling, YouTube.
00:14:43.440 --> 00:14:46.879
A feed is like a system for curating things at scale.
00:14:47.120 --> 00:14:54.399
You can upvote something, you can retweet something or repost it, you can block it, you can DM the person to narrow the communication.
00:14:54.559 --> 00:14:59.440
But a feed is like a tool for millions of people to decide what's interesting and important.
00:14:59.679 --> 00:15:00.639
Even Reddit, right?
00:15:00.720 --> 00:15:06.879
You just upvote, downvote things that bubble to the top divided by how long they've been there, kind of gives you a sense of what's interesting.
00:15:07.039 --> 00:15:13.840
These are all mechanisms for processing information at scale that are at least somewhat resilient to attack.
00:15:14.159 --> 00:15:14.559
Right.
00:15:14.799 --> 00:15:17.279
But and they're much more accessible to the average person.
00:15:17.440 --> 00:15:27.919
You can imagine that you would go in and you know, say, these are the things I care about and see the laws that are related to what you're doing and you know, retweet or upvote or downvote the comments you like and add your own comments.
00:15:28.320 --> 00:15:31.600
Like we have a mechanism for this stuff that's far more participatory.
00:15:31.759 --> 00:15:35.600
Otherwise, without that kind of system, comment flooding is this crazy anti-pattern.
00:15:35.759 --> 00:15:39.279
People are weaponizing a democratic process specifically to slow it down.
00:15:39.759 --> 00:15:40.720
Absolutely.
00:15:41.039 --> 00:15:43.120
And sometimes it it works.
00:15:43.440 --> 00:15:45.919
So does it actually change outcomes or is it just friction?
00:15:46.240 --> 00:15:48.960
Sometimes it changes outcomes.
00:15:49.120 --> 00:15:50.960
It's not very common.
00:15:51.200 --> 00:16:08.639
Usually you'll see small things like extensions on rulemakings due to uh in a few in a few rare instances, it's been because something didn't publish on time, or they weren't able to put an update out on time, or there was downtime in the system.
00:16:08.799 --> 00:16:12.879
In the history of the system, it's been up maybe 99.9% of the time.
00:16:12.960 --> 00:16:16.879
But when it goes down, you have to legally extend the rulemaking period.
00:16:17.200 --> 00:16:17.679
Interesting.
00:16:17.919 --> 00:16:18.080
Yeah.
00:16:18.240 --> 00:16:28.639
And so your team, you were at like 18F uh government service administration stuff in the federal government in the US, had a particular way of working really focused on like human-centered design in a government context.
00:16:28.799 --> 00:16:28.960
Yes.
00:16:29.200 --> 00:16:37.919
How did that help you uh to figure out ways that you might fix it without changing either the wording, the legislation, or the process?
00:16:38.240 --> 00:16:47.919
So the thing that I did was I literally took my laptop and went to the food court and started going up to people.
00:16:48.080 --> 00:16:50.000
Hey, I'm testing this.
00:16:50.159 --> 00:16:52.000
I want more information about that.
00:16:52.159 --> 00:16:52.960
Can you tell me about it?
00:16:53.200 --> 00:16:57.279
Usually feds, because you can only bother so many public citizens with the same thing.
00:16:57.440 --> 00:17:02.000
I was gonna say, it's like Jonathan Haidt used to say that all psychological research was weird.
00:17:02.240 --> 00:17:09.279
Western, educated, independent, democratic, and rich, because all of the studies done at universities were done on university campuses.
00:17:09.359 --> 00:17:14.079
So he would like try and reproduce a study at a McDonald's three miles from campus and get a very different result.
00:17:14.720 --> 00:17:18.720
Your first round of sampling, you're already talking to people who are like inside the beltway.
00:17:18.799 --> 00:17:21.440
They're already baseball players in your inside baseball game, right?
00:17:21.680 --> 00:17:22.000
Yes.
00:17:22.240 --> 00:17:23.759
That happened a few times.
00:17:23.920 --> 00:17:30.160
Then I would go to conferences and sort of walk out from the hotel there and try to meet people.
00:17:30.400 --> 00:17:38.240
But the other part of that is when you talk about federal employees and other people you could test with, it doesn't have to all be beltway people.
00:17:38.400 --> 00:17:40.079
Make sure you get the security guard.
00:17:40.160 --> 00:17:44.480
Make sure you get the lady who's literally at the register of the food court.
00:17:44.640 --> 00:17:51.519
Still counts, still good information, still the exact kind of demo that has every right to use this tool.
00:17:51.759 --> 00:17:57.599
And so you would you would walk up and like open your laptop and say, hey, here's a potential regulation on something.
00:17:57.839 --> 00:17:59.039
How would you like to comment?
00:17:59.119 --> 00:18:00.160
Is that is that the approach?
00:18:00.480 --> 00:18:11.920
I would give them scenarios that are more along the lines of you just watched a news report and it mentions this regulation that is going to happen.
00:18:12.160 --> 00:18:16.400
You get curious, you want to add your input to it, you start off at Google.
00:18:16.480 --> 00:18:18.480
Sometimes I would start from the search engine.
00:18:18.640 --> 00:18:22.000
Find out where to comment, how you want to comment.
00:18:22.160 --> 00:18:23.440
Let me watch what you're doing.
00:18:23.759 --> 00:18:28.960
So you're even starting at the long funnel of just I care about how many guns there are, one way or another.
00:18:29.359 --> 00:18:30.960
And then what do you do about it?
00:18:31.039 --> 00:18:32.640
Did you know there's a thing you could do about it?
00:18:32.960 --> 00:18:34.000
I would vary it up.
00:18:34.160 --> 00:18:38.480
Sometimes it would be at the long funnel, sometimes it would be at the webpage itself.
00:18:38.720 --> 00:18:41.599
I want to mix the data, but that that would be the general idea.
00:18:41.920 --> 00:18:48.319
So let's talk about your two fixes because they're both pretty clever workarounds that fit within these tight legal constraints.
00:18:48.559 --> 00:18:48.799
Yes.
00:18:48.960 --> 00:18:52.480
You said your first one was bulk comment downloads for reviewers.
00:18:52.720 --> 00:18:53.200
Yes.
00:18:54.079 --> 00:18:54.960
Sounds pretty simple.
00:18:55.119 --> 00:18:57.839
Like, like, what does it take to ship that inside a federal system?
00:18:57.920 --> 00:19:00.240
Because bulk downloads sounds like a couple lines of code.
00:19:00.480 --> 00:19:00.720
Yeah.
00:19:00.880 --> 00:19:03.839
Uh ordinarily, it would be pretty simple.
00:19:04.000 --> 00:19:06.480
I I don't want to take sole credit for this idea.
00:19:06.720 --> 00:19:11.519
It was an idea that a lot of people needed and a lot of people requested.
00:19:11.599 --> 00:19:17.599
And at a certain critical depth, you just say, if this many people are asking for it, it must be for a reason.
00:19:17.759 --> 00:19:24.559
And the the bulk download of comments during the Drupal modernization is a little more difficult.
00:19:24.880 --> 00:19:25.920
So let me understand.
00:19:26.079 --> 00:19:32.640
The reviewer had to be on the system, look at one thing, then like click close or save or whatever, go to the next one.
00:19:32.720 --> 00:19:35.839
They couldn't look at them outside of the original system.
00:19:35.920 --> 00:19:36.400
Is that right?
00:19:36.559 --> 00:19:37.200
Uh yes.
00:19:37.519 --> 00:19:39.119
How did that affect their productivity?
00:19:39.359 --> 00:19:40.640
Oh, it's it's horrible.
00:19:40.799 --> 00:19:46.000
It's horrible, especially because very often you have related regulations.
00:19:46.160 --> 00:19:52.799
And within one regulation, you can have dozens of documents itself all in that one docket.
00:19:53.440 --> 00:19:58.799
And usually you have the majority of comments on one document, but not necessarily.
00:19:58.960 --> 00:20:04.799
So you have this separation of data where you're trying to find all the comments about rulemaking.
00:20:04.960 --> 00:20:08.480
You might have to go to another rulemaking, you might have to go to multiple documents.
00:20:08.640 --> 00:20:10.720
It becomes sort of a mapping make.
00:20:11.039 --> 00:20:19.839
So it sounds like the tool was made to let you review an individual comment, but not for the analysis of feedback across comments or synthesis of any kind.
00:20:20.160 --> 00:20:30.559
There are some tools that were and are used that help you find the highly copied comments, the ones that are 80% similar.
00:20:30.720 --> 00:20:39.039
You you put a sort of move a slider to 80%, it shows you that 80% language, then it highlights the 20% as you go through it.
00:20:39.200 --> 00:20:48.880
But in terms of getting mass sentiment analysis or breaking things up by topic or categorization, largely weren't there.
00:20:49.119 --> 00:20:56.240
It does seem to me like one opportunity to do this would be, you know, if you can't replace the existing system, create a parallel system.
00:20:56.480 --> 00:21:03.200
And I've seen tools like polis that are sort of designed for at-scale commenting where you pick your position on something or whatever.
00:21:03.359 --> 00:21:08.880
Or uh the Society Library has a thing where they take a topic and they break it apart into all of the different debate structures.
00:21:09.039 --> 00:21:21.359
Did anybody talk about building a parallel system to this one so that you were still respecting the letter of the law with regulations.gov, but then building another more analyzable system to kind of get the zeitgeist or the vibes from people?
00:21:21.599 --> 00:21:23.039
It's been talked about.
00:21:23.200 --> 00:21:27.119
I wouldn't be surprised if even today it was still worked on.
00:21:27.359 --> 00:21:30.720
This is something that's sort of always been in the background of the discussion.
00:21:30.799 --> 00:21:36.000
I'm sure they probably are still at the very least considering it, if not actively building it.
00:21:36.240 --> 00:21:38.799
There's a few difficulties there.
00:21:39.039 --> 00:21:44.640
One difficulty is the fee structure with how they pay for the service.
00:21:44.799 --> 00:21:52.720
There's 200 federal agencies, but the ones who really produce the majority of regulations are very top heavy.
00:21:52.960 --> 00:22:01.680
And when you have a shared service where there's only a few who do most of the demand, and everybody else who might produce One or two regulations.
00:22:02.400 --> 00:22:08.319
Paying for any sort of thing like that always raises the debate of, well, why isn't EPA covering?
00:22:08.640 --> 00:22:08.880
Got it.
00:22:08.960 --> 00:22:17.519
So you have to bill people according to usage, and there are a few big users that get to dictate the roadmap and stuff because that's the service they're paying for.
00:22:17.759 --> 00:22:27.279
Yeah, you either bill them according to usage or you try to spread it out, but you're going to have the complaints there of why am I floating the bill?
00:22:27.359 --> 00:22:29.440
Because EPA produces so many rules.
00:22:29.599 --> 00:22:32.880
I keep mentioning EPA only because they do produce a lot of rules.
00:22:33.039 --> 00:22:35.279
That's an observable fact.
00:22:35.519 --> 00:22:49.759
Um but either way, it makes trying to get additional funding for any fix, any parallel system that isn't going to be shared equally a major debate topic and something that is very often hard to approve.
00:22:50.240 --> 00:22:59.279
So the second fix that you had is a little more subtle: restructuring the underlying data to reduce the exposure to jargon without actually rewriting the regulations themselves.
00:22:59.359 --> 00:23:00.640
So you're not violating the law.
00:23:00.799 --> 00:23:02.079
Can you unpack that a little?
00:23:02.160 --> 00:23:04.559
Like what does restructuring the data mean in plain terms?
00:23:04.880 --> 00:23:07.440
Uh de-emphasizing the word docket.
00:23:07.599 --> 00:23:09.200
That that was a big start.
00:23:09.359 --> 00:23:16.960
Once again, if you don't get involved with court or cases or the legal system too often, you don't know what a docket is.
00:23:17.039 --> 00:23:18.079
It's a folder.
00:23:18.880 --> 00:23:25.279
And really, if I had the magic wand for a day, I would have just called it a folder myself.
00:23:25.519 --> 00:23:31.599
But we have government people who also use the system and they get attached to their vocabulary, so it becomes a big to-do.
00:23:31.839 --> 00:23:34.319
But it's a folder that you put documents in.
00:23:34.559 --> 00:23:36.319
That's more or less it.
00:23:36.880 --> 00:24:02.319
So doing that, making sure that the search is indexed so it's primarily by document and looking up the keywords and the subjects and the metadata of the document as opposed to the docket first is a big step in that direction because it means when you get a search result and you go there, you're in a document, you're in that area, that level of data that you understand.
00:24:02.480 --> 00:24:06.400
And you might not know what a docket is, but you'll understand the UX of it.
00:24:06.640 --> 00:24:08.000
And that'll help you navigate with it.
00:24:08.319 --> 00:24:12.480
You still got to keep the words there, but you can show people something they'll understand and connect with first.
00:24:12.720 --> 00:24:12.960
Yes.
00:24:13.200 --> 00:24:19.519
And you can make it easier for the reviewers to pull down many things at once and analyze them themselves.
00:24:19.599 --> 00:24:22.319
Although it does seem like the analysis tools are kind of far behind.
00:24:22.480 --> 00:24:30.559
There's like an asymmetry here where the commenters are allowed to use whatever automation they want, and the the reviewers have to, you know, pull out a piece of paper and read it.
00:24:30.720 --> 00:24:32.559
So it seems like you're pretty stuck there.
00:24:32.960 --> 00:24:34.640
They're they're behind, but they're getting better.
00:24:34.960 --> 00:24:37.200
Then there's the API layer for outside groups.
00:24:37.279 --> 00:24:39.759
You you have this data is public, right?
00:24:39.920 --> 00:24:45.680
And so while your hands are tied, the hands of civic organizations that care about these things are not.
00:24:45.839 --> 00:24:51.680
So you basically said we can't simplify the language, but we can make it easier for other people to simplify it for their communities.
00:24:51.920 --> 00:24:56.960
Was that something you framed as a workaround from the start, or did it kind of emerge as you were designing this stuff?
00:24:57.359 --> 00:25:06.160
It emerged because we were already getting a lot of headless browser submittals, people who were just sort of automating mass submittals.
00:25:06.480 --> 00:25:08.799
So can you clarify what a headless browser submittal is?
00:25:08.880 --> 00:25:09.119
Uh yeah.
00:25:09.440 --> 00:25:11.440
Because you're doing the inside baseball language there.
00:25:11.599 --> 00:25:12.160
Yes, you're right.
00:25:12.240 --> 00:25:12.559
You're right.
00:25:12.640 --> 00:25:13.440
Sorry about that.
00:25:13.680 --> 00:25:28.079
Uh so basically it is people who would send all of these comments over to us in a very automated and packaged fashion in such a way that it didn't even go over across a browser.
00:25:28.240 --> 00:25:30.079
It's just sort of computer to computer.
00:25:30.480 --> 00:25:35.599
It's a script that's just sending the messages and that that it knows it are needed to connect to the page and do it.
00:25:35.839 --> 00:25:44.079
Just kind of like what we see today with Cloud and Chrome, where Cloud can take over your browser, or just running Python commands that send web requests to a server.
00:25:44.160 --> 00:25:45.599
That stuff's even easier now than it was then.
00:25:45.839 --> 00:25:48.640
Yeah, just an automated script says push it over.
00:25:48.880 --> 00:25:49.200
Wow.
00:25:49.440 --> 00:25:51.279
We were already getting a lot of those.
00:25:51.440 --> 00:25:57.599
So many that you could see the dip in the system's performance or its traffic at times when there's a lot of different things.
00:25:57.759 --> 00:26:02.240
Oh, because they're using a web server that's designed for humans, but they're hammering it with a machine.
00:26:02.319 --> 00:26:02.559
Okay.
00:26:02.799 --> 00:26:03.440
Yes.
00:26:03.839 --> 00:26:11.200
Uh and when you we looked at who was doing it, there's some very legitimate organizations who are doing this.
00:26:11.359 --> 00:26:15.119
They have a legitimate business need to send over many at a time.
00:26:15.359 --> 00:26:16.480
So can you give me an example of that?
00:26:16.559 --> 00:26:21.200
Is like there's that like Ducks Unlimited had a thing on its website and now it wants to upload all those comments to you?
00:26:21.279 --> 00:26:21.920
Or what is that?
00:26:22.240 --> 00:26:22.960
Uh yeah, yeah.
00:26:23.039 --> 00:26:26.079
So we would get a veteran service organization.
00:26:26.319 --> 00:26:38.079
So somebody who's trying to represent veterans uh from the military across the country to make sure that a regulation establishes their correct benefits for healthcare.
00:26:38.240 --> 00:26:49.839
I don't want to name specific names or anything like that, but what they would do is they would collect information from their membership, which is tens of thousands, if not hundreds of thousands, of people.
00:26:50.000 --> 00:26:52.960
And they would say, okay, we're gonna send you out this form.
00:26:53.839 --> 00:26:59.119
They're going to change your prescription costs, throwing out hypotheticals here.
00:26:59.359 --> 00:27:04.160
Fill out this information in this form, add your name, we'll push it forward.
00:27:04.319 --> 00:27:12.480
And they want to make sure these people have a maximum amount of time to fill everything out, to put in the data that's required and to send it to them.
00:27:12.640 --> 00:27:21.119
And if they're gonna squeeze as much time out of that as they can, they need the quickest and most automated way to then send it to regulation stuff.
00:27:21.680 --> 00:27:27.200
I think the normal reaction would be hey, I've got this website, machines are using it, it's designed for humans.
00:27:27.359 --> 00:27:29.920
Let's make it as hard as possible for machines to use it.
00:27:30.000 --> 00:27:38.160
And websites today have these things like the robots.txt file that are basically instructions that say, hey, you're a computer, you're a Google crawler or whatever, you're not a human.
00:27:38.240 --> 00:27:40.400
You may not use this website in the way a human would.
00:27:40.640 --> 00:27:42.319
So there's a bit of an arms race, right?
00:27:42.480 --> 00:27:42.720
Yes.
00:27:42.880 --> 00:27:56.880
Um, I know that Cloudflare, one of the big content delivery companies that protects sites, has a thing they call a bot labyrinth, which is quite literally when it thinks you're a bot, it sends you into a bunch of generated pages full of nonsense to confuse you, right?
00:27:57.039 --> 00:28:02.880
So there's this very adversarial relationship between websites intended for humans and machines.
00:28:03.359 --> 00:28:12.559
You showed up and said counterintuitively, we think we should make it easier for machines to use this because they're gonna do it anyway, and this will solve some of our performance and load problems.
00:28:12.640 --> 00:28:13.039
Is that right?
00:28:13.279 --> 00:28:14.960
They're gonna do it anyways.
00:28:15.279 --> 00:28:19.119
For most of these anti-bot steps you can take.
00:28:19.279 --> 00:28:24.160
There's very publicly searchable and findable ways to beat them.
00:28:24.319 --> 00:28:38.079
So if we start implementing, you know, some sort of scheme to capture, click on all of these buses in this picture or something like that, we add accessibility problems and we only put up a problem that the cheaters could solve.
00:28:38.319 --> 00:28:38.559
Interesting.
00:28:38.640 --> 00:28:38.880
Yeah.
00:28:38.960 --> 00:28:42.640
So you put up you make it so hard for a machine that you've made it hard for humans.
00:28:42.880 --> 00:28:44.160
So let's skip that.
00:28:44.240 --> 00:28:44.559
Yeah.
00:28:44.640 --> 00:28:44.720
Yeah.
00:28:44.960 --> 00:28:45.519
How did that?
00:28:45.599 --> 00:28:46.880
I mean, that sounds counterintuitive.
00:28:46.960 --> 00:28:51.440
What was the initial reaction to please let's welcome our robot overlords to the conversation?
00:28:51.759 --> 00:29:07.519
Well, once we started getting the data on who was doing this and pointing to legitimate business need, legitimate need, these people who are representing real people with real problems and a real right to speak up, it starts to get a little more reasonable.
00:29:07.759 --> 00:29:12.400
Then when we started pointing out it there is a middle ground, you create a whitelist.
00:29:12.799 --> 00:29:23.519
You say, okay, you you advertise it, you say if you want access to this API, you sign up with us, we write down your name, we give you the API, the key, and we track you.
00:29:23.680 --> 00:29:30.960
We know exactly what you're doing, what quantities you're doing it in, and we can tell if it's a denial of shivers attack or content through that way.
00:29:31.279 --> 00:29:39.039
So you're you're giving people the right to register legitimate bots that should then allow that have a username and a password for a machine.
00:29:39.279 --> 00:29:39.680
Yes.
00:29:39.920 --> 00:29:40.079
Yeah.
00:29:40.400 --> 00:29:49.839
This seems like this is some serious foreshadowing for what we're going to need as people start to deploy like clawed code or claw or open open claw and things like that, right?
00:29:50.000 --> 00:29:50.319
Yeah.
00:29:50.640 --> 00:29:55.519
So uh how long did all this take from your first conversation to when it was live in production?
00:29:55.839 --> 00:29:56.480
Years.
00:29:57.680 --> 00:29:59.440
It took years.
00:29:59.680 --> 00:30:06.559
There's a lot of things happening in the middle of the modernization, a lot of budgetary problems.
00:30:06.720 --> 00:30:14.880
And then there was the FCC rule, which created a title shift in how this entire process is seen.
00:30:15.039 --> 00:30:20.880
Because when net neutrality came out, there were people who were not just sending automated comments.
00:30:21.039 --> 00:30:28.319
They were sending automated comments posing as other people in numbers that the government had probably never seen before.
00:30:28.480 --> 00:30:32.480
In some cases, even pretending to be congressmen who were sending in comments.
00:30:32.799 --> 00:30:36.000
I guess when you make a law that nerds care about, nerds are gonna nerd, right?
00:30:36.240 --> 00:30:36.559
Yes.
00:30:36.799 --> 00:30:37.039
Yeah.
00:30:37.279 --> 00:30:37.920
Big time.
00:30:38.079 --> 00:30:38.400
Big time.
00:30:38.880 --> 00:30:41.680
After you change shipped these changes, what did you see in the data?
00:30:41.839 --> 00:30:45.680
Did it change the diversity of commenters or the review cycle speed?
00:30:46.000 --> 00:30:48.640
In some cases, it changed the review cycle speed.
00:30:48.799 --> 00:30:52.720
I think this is still very much a work in progress.
00:30:53.599 --> 00:30:59.599
Uh, in some cases, it made life easier for those organizations who are using the API.
00:30:59.839 --> 00:31:09.279
And in a very non-political way, I can't emphasize that enough because you can't you can't play winners or losers with your API when you're the government.
00:31:09.440 --> 00:31:14.400
You have to make sure you give it to anybody who's just basically not completely misrepresenting the law.
00:31:14.720 --> 00:31:15.599
That's a great line.
00:31:15.759 --> 00:31:19.039
You can't um play winners or losers with API and you can't play politics.
00:31:19.200 --> 00:31:28.960
You know, the data, especially when it's data that's mandated for publication is part of the democratic process, that data and those those uh platforms, you have to be okay with your worst political enemy using this.
00:31:29.279 --> 00:31:30.799
Yes, absolutely.
00:31:31.039 --> 00:31:50.319
Um but putting it out there, making their life easier on both sides of the aisle, on everybody else, it's it's a continuation of that original idea behind regulations.gov, which is the government isn't gonna have all the answers or all parts of the solution.
00:31:50.960 --> 00:32:04.240
If I want to do a law and uh regulation and sub-nonprofit out there is gonna make 200,000 people more aware of this regulation, at some point the government should just say thank you.
00:32:04.480 --> 00:32:04.720
Yeah.
00:32:04.799 --> 00:32:11.680
I mean, it really is the democratic process in the digital age, but it's obviously been used in a way that is devastating for a lot of these systems.
00:32:11.839 --> 00:32:12.079
Yeah.
00:32:12.160 --> 00:32:12.400
Yeah.
00:32:12.640 --> 00:32:12.960
Okay.
00:32:13.119 --> 00:32:19.759
So uh for the nonprofit and advocacy groups that did use the API, did it actually change their participation?
00:32:19.920 --> 00:32:23.200
Do you see a rise in the proportion of comments that came from those groups?
00:32:23.519 --> 00:32:26.480
I believe so, but that is very hard to validate.
00:32:26.559 --> 00:32:34.559
It needs a long timeline because this is also subject to what laws and regulations that happen to be coming out.
00:32:35.359 --> 00:32:45.839
So uh a net neutrality is going to cause uh an ups it an uptick in that contribution of comments, irregardless.
00:32:46.160 --> 00:32:46.319
Sure.
00:32:46.559 --> 00:32:49.680
Anything that surprised you about the results and the impact?
00:32:50.079 --> 00:33:05.039
I think the thing that surprised me most is and I think you latched onto this very quickly, is the need for some sort of parallel system or better analysis tool to keep going.
00:33:05.759 --> 00:33:09.440
That this isn't this is by far not a solved or done problem.
00:33:09.759 --> 00:33:10.559
Well, it used to be Mr.
00:33:10.640 --> 00:33:12.160
Smith was going to go to Washington, right?
00:33:12.240 --> 00:33:16.640
Or Ottawa or wherever he's going, and now it's everyone is in their phones doing it.
00:33:16.880 --> 00:33:24.880
So much of government was created with this idea that the citizen would connect to a government person on an individual basis.
00:33:25.039 --> 00:33:31.680
There was never a sense of scale, there was another sense of automation, there was never a sense that the citizen would have access to these things themselves.
00:33:31.839 --> 00:33:46.480
I mean, it seems like this is the kind of thing where in a in a reasonably advanced democracy, if you cared about, for example, childcare, you could set an alert to find out about any time that any legislation was being passed where the regulations around childcare might change, and then you could go and comment on them, right?
00:33:46.640 --> 00:33:47.440
Yeah, yeah.
00:33:48.319 --> 00:34:01.519
And what really sort of then surprises me what fell out of this when we started taking a critical look at what goes into the system, what people are submitting for is also the federal land.
00:34:02.799 --> 00:34:15.840
And there's very often a disconnect between the laws and the regulations that come up and the stock of infrastructure that a government has.
00:34:16.159 --> 00:34:16.480
Right.
00:34:16.719 --> 00:34:22.719
So you'll see a lot of laws and regulations that say, oh, you have to notify the people.
00:34:22.880 --> 00:34:34.719
So there's a bunch of things that end up in the system that are not necessarily in the scope of a regulatory making, but it's a system and it counts as technically notifying the people.
00:34:35.039 --> 00:34:36.239
So it's more of an ass covering.
00:34:36.320 --> 00:34:37.599
You're like, hey, I did, I told them.
00:34:37.679 --> 00:34:37.760
Yeah.
00:34:38.960 --> 00:34:39.119
Yeah.
00:34:39.199 --> 00:34:39.360
Okay.
00:34:39.679 --> 00:34:40.000
Yes.
00:34:40.079 --> 00:34:48.400
And so separating the notification that something is happening from the I want to have a reason debate and see if I've missed anything intense is kind of huge.
00:34:48.880 --> 00:34:49.679
Enormous.
00:34:50.000 --> 00:34:50.559
Enormous.
00:34:50.639 --> 00:34:50.880
Yeah.
00:34:51.199 --> 00:35:18.079
It does seem to me like it would be fairly easy, given that you have an API, for someone to register as a legitimate connector via the API, for them to create a website where any person could go and say, I'm interested in these things, for them to scrape the dockets, match them to things people are interested in, send those people a message, have them click to the website, show them just the things they want, maybe even translate it out of jargon, fill in the stuff, and then send it back to your website.
00:35:18.320 --> 00:35:20.320
Like we don't have to wait for the government to do that.
00:35:20.480 --> 00:35:26.079
I wrote a thing a couple of weeks ago called the Machine Fights Back about how humans used to have to grind up against the machinery government.
00:35:26.159 --> 00:35:34.639
It seems like it'd be pretty easy to build a publish and subscribe model for people that just made these things transparent and let you comment on them.
00:35:34.880 --> 00:35:35.519
Yes.
00:35:35.840 --> 00:35:37.760
Uh 100% agreed.
00:35:37.920 --> 00:35:39.440
It's a good business model.
00:35:39.760 --> 00:35:40.239
Business model.
00:35:40.559 --> 00:35:41.599
These days you can just do it for free.
00:35:41.679 --> 00:35:42.960
Like I don't want to make money from it.
00:35:43.280 --> 00:35:51.280
Well, it's a good when I say business model, it's more in terms of it's a good thing you can do to drive a result, money or not.
00:35:51.440 --> 00:36:04.079
I know people who have been looking at doing things like that, lobbyist organizations, nonprofits, who really just want to have a better dashboard of government things that their people can care about.
00:36:04.320 --> 00:36:07.599
And there's no way for me to find out what happened to my comment, right?
00:36:07.920 --> 00:36:20.239
Uh usually what will happen is they produce a final rule and it will to some degree address the comments received.
00:36:20.559 --> 00:36:24.000
Sometimes it'll just say, oh, we got comments and we considered them.
00:36:24.159 --> 00:36:26.480
Sometimes it'll break it out by the individual comment.
00:36:26.639 --> 00:36:29.119
It really depends upon who the rule writer is.
00:36:29.440 --> 00:36:29.599
Okay.
00:36:29.760 --> 00:36:33.360
Let's switch selfishly to Canada now, because we have a similar system here.
00:36:33.440 --> 00:36:35.519
Obviously, the here the Canada Gazette.
00:36:35.599 --> 00:36:40.480
It's where proposed regulations are published and public comment is supposed to be part of the process.
00:36:40.639 --> 00:36:43.199
But in our case, participation is vanishingly small.
00:36:43.360 --> 00:36:43.519
Yes.
00:36:43.760 --> 00:36:48.480
If you were advising a Canadian team facing similar constraints, what's the first thing you'd tell them?
00:36:48.880 --> 00:36:58.000
Uh it it participation in the system is small, but there are a lot of ways to participate in providing input.
00:36:58.159 --> 00:37:00.320
Is it happening somewhere?
00:37:01.199 --> 00:37:03.280
That that's the first thing I would want to know.
00:37:03.440 --> 00:37:13.039
You know, if there's more public meetings that get publicized in the newspaper or the news, or just through an email.
00:37:13.599 --> 00:37:16.000
I don't care how the participation is happening.
00:37:16.079 --> 00:37:22.400
The tool doesn't justify itself unless that unless people gravitate towards participation in that.
00:37:22.800 --> 00:37:23.119
Got it.
00:37:23.280 --> 00:37:33.840
So you spent a decade at the EPA, you spent years at GSA, and now you're at Ishmael Interactive, which was a firm that was built by Ford Former ATF people after Doge shut things down.
00:37:34.000 --> 00:37:39.679
What's been lost with the dismantling of those kinds of human-centered design institutions within government?
00:37:40.079 --> 00:37:41.440
Uh the context.
00:37:41.840 --> 00:37:50.559
The people who would go out and get the context of what government is doing to the people who it's supposed to serve.
00:37:50.719 --> 00:38:02.400
You know, 18F, great office, love their work, but when they made a fix for paying your taxes, they started off by talking to people who were paying their taxes.
00:38:02.559 --> 00:38:02.880
Yeah.
00:38:03.039 --> 00:38:05.519
You can't get that out of just an AI.
00:38:06.079 --> 00:38:07.360
Quite the same.
00:38:07.519 --> 00:38:10.000
Not only that, but just the talent pool.
00:38:10.159 --> 00:38:12.639
You just an incredible talent pool.
00:38:12.800 --> 00:38:14.639
I don't know how they're going to restock.
00:38:15.039 --> 00:38:23.440
I a lot of those changes are also scary just because they did it to such a degree all at once.
00:38:24.320 --> 00:38:35.679
Even if you cut the right people, which personally I think is highly debatable, if you cut X percent of them all at once, it's going to create problems that you have to learn how to bounce back.
00:38:36.079 --> 00:38:39.039
You lose that sort of organizational muscle mass and memory, right?
00:38:39.360 --> 00:38:40.159
Yeah, absolutely.
00:38:40.480 --> 00:38:48.559
So regulations.gov is a case where you couldn't change the law, you couldn't change the language, and you still found a way to, in theory, make democracy more accessible.
00:38:48.800 --> 00:38:52.320
What's the lesson here for people who feel like government is too big and too broken to fix?
00:38:52.559 --> 00:38:53.920
You have a part in it.
00:38:54.079 --> 00:38:57.360
And that's the thing I love about regulations.gov.
00:38:57.679 --> 00:39:08.559
And that's the thing I love about the human testing, is there's usually somebody you could get involved with and relay your experience to.
00:39:08.800 --> 00:39:11.760
If it is going on the gazette, great.
00:39:11.920 --> 00:39:21.840
If it is calling your mayor and saying, I want five minutes of your time, it all starts with transmitting what your personal context is.
00:39:22.000 --> 00:39:23.199
How does this affect me?
00:39:23.519 --> 00:39:30.239
We talked a little bit about AI and automation, and obviously that's something that's happening on a weekly or even daily basis.
00:39:31.119 --> 00:39:40.320
Um, what are your thoughts about what happens when it becomes obvious that the citizens have their own machines and they can point it at government if they don't want it, don't like what it's doing?
00:39:40.800 --> 00:39:42.239
Well, there's two different thoughts.
00:39:42.400 --> 00:39:47.519
One is I'm trying to separate oppositional content from denial of service.
00:39:47.760 --> 00:39:50.639
Denial of service, bad oppositional content.
00:39:50.800 --> 00:39:53.119
As long as it's true, I don't care.
00:39:53.360 --> 00:39:58.320
I'm not supposed to pick sides, government's not supposed to pick sides, that is fine.
00:39:58.559 --> 00:40:08.320
If it is creating a substantive comment or substantive input with actual facts that can be researched, this is no different than using a keyboard instead of a pen.
00:40:08.639 --> 00:40:14.880
The tools that allow scale are separate from the legitimacy of the underlying message or goal.
00:40:15.199 --> 00:40:15.440
That's right.
00:40:15.920 --> 00:40:16.800
It's content first.
00:40:17.119 --> 00:40:27.679
And you also push back against this idea that the website was a reliable source of friction that forced humans to do this firsthand, both as reviewers and as submitters, right?
00:40:27.840 --> 00:40:29.440
Because it was already being circumvented.
00:40:29.599 --> 00:40:32.559
Do you see governments having a similar thought about AI?
00:40:32.639 --> 00:40:33.760
Are they not even thinking about it?
00:40:33.920 --> 00:40:36.079
Are they trying to send bots into labyrinths?
00:40:36.159 --> 00:40:42.239
Or are they now saying, no, we need a skill for the federal government that you can tell Claude about so it knows how to respond?
00:40:42.559 --> 00:40:58.480
I think that this is one of the things the current administration is getting right, is that they're paying more attention to automation and AI in a way that is trying to put it in the workflow, whether they do that successfully or not, it's my government.
00:40:58.559 --> 00:40:59.519
So I hope they do.
00:40:59.760 --> 00:41:02.960
But I think that there's still some fear.
00:41:03.119 --> 00:41:14.480
And I'll I'll I'll take you back to the FCC rule again and the ability to impersonate another person and put your comment onto somebody else's name.
00:41:14.639 --> 00:41:21.920
If you talk to most of the rule writers at the time, the people who actually make the regulations, they didn't care whose name was on it.
00:41:22.079 --> 00:41:25.760
You were committing fraud for something that had no effect.
00:41:26.159 --> 00:41:28.480
Was it part of the consideration set?
00:41:28.719 --> 00:41:29.760
It's the content.
00:41:29.840 --> 00:41:31.679
It can have any name on it.
00:41:31.760 --> 00:41:32.480
It doesn't matter.
00:41:32.800 --> 00:41:33.519
Yeah, it's interesting.
00:41:33.599 --> 00:41:35.840
It wasn't a vote, so there was no question of legitimacy.
00:41:36.000 --> 00:41:36.239
Exactly.
00:41:36.480 --> 00:41:39.840
It was a comment, and that comment stands on its own regardless of who made it.
00:41:40.079 --> 00:41:40.400
Yeah.
00:41:40.639 --> 00:41:45.519
Well, you know, Canada's just launched online commenting in September 2022.
00:41:45.599 --> 00:41:53.519
We're pretty far behind where the US was, but we're not seeing anywhere near the numbers you saw, you know, around 1,500 people for an open government debate.
00:41:53.760 --> 00:41:57.760
What do you think it is that's gonna make Canadians realize they too have a voice?
00:41:58.159 --> 00:42:00.559
So I'm not sure that they don't realize this.
00:42:00.719 --> 00:42:05.440
They might be getting their civic participation and their civic input somewhere else.
00:42:05.679 --> 00:42:10.800
So the first thing I would do is get a Canadian sit down and say, How do you participate?
00:42:12.159 --> 00:42:16.559
If they're not participating through the Gazette, I don't know what it might be.
00:42:16.719 --> 00:42:29.599
Usually it is, but usually when I see a bump up in people who start to then comment to an agency or a group of agencies or on a topic, it takes one big law.
00:42:30.320 --> 00:42:46.480
One big earth-shattering law that gets in the news, gets on your talk shows, then people come in, they find it, they comment, and then the week after they remember, oh wait, there might be other rules and other regulations about this stuff.
00:42:46.719 --> 00:42:48.239
So I'm just gonna go once a week.
00:42:48.480 --> 00:42:52.320
Well, you've kind of made me want to go build the social network for the Canada Gazette now.
00:42:52.400 --> 00:42:55.039
So uh maybe when I have a week off, I'll go try and build that.
00:42:55.280 --> 00:42:57.440
Aaron, thank you so much for sharing this story with us.
00:42:57.599 --> 00:43:08.639
It's uh it's a fascinating example of something that is like you're trying to make this thing work, but your hands are tied in the ways that most people would see are obvious.
00:43:08.800 --> 00:43:19.840
So you're like doing what you can, but my my developer brain just gets furious at the at how outdated this system is when we, you know, Reddit is better at handling comments than this system.
00:43:20.079 --> 00:43:20.880
Yeah, excellent.
00:43:20.960 --> 00:43:21.199
Thank you.
00:43:21.440 --> 00:43:22.159
Thanks a lot, Aaron.
00:43:22.320 --> 00:43:22.639
Awesome.
00:43:22.800 --> 00:43:25.920
Thanks for joining us on the Functional Government Podcast.
