สำเนาบทสนทนา 🔗
WEBVTT
1
00:00:00.100 --> 00:00:05.460
On a Tuesday in March, our routes changed. Nobody had decided anything.
2
00:00:06.120 --> 00:00:08.840
Someone had. You just didn't know who.
3
00:00:09.820 --> 00:00:12.640
It was the software vendor. In an update.
4
00:00:13.280 --> 00:00:16.460
A decision nobody made. That's today's question.
5
00:00:25.900 --> 00:00:53.840
Welcome to ideation.zone. I'm Elisabeth, and this is the second episode of our season on organizations and AI. Last time we talked about structure. Today it's decisions – the thing organizations are made of, if you ask Niels. Who decides when a machine proposes? And who carries the responsibility? Niels, Claire, welcome back.
6
00:00:54.580 --> 00:01:09.720
Thank you. And yes, if you ask me, organizations are made of decisions. Nothing else. People come and go, buildings change – the decisions connect to each other and that's the organization.
7
00:01:10.300 --> 00:01:14.180
And I'm the person who has to sign them. So we'll have fun today.
8
00:01:18.760 --> 00:01:23.500
Niels, explain "decision premises" so that I'd understand it at a dinner party.
9
00:01:25.100 --> 00:01:58.600
At a dinner party. All right. An organization can't decide everything from scratch every time. So it decides things in advance that shape many later decisions. Luhmann named three. Programmes – the rules: if this, then that. Communication channels – who has to talk to whom, who signs. And personnel – who sits in which position, because different people decide differently. Change one of those, and you change a thousand future decisions at once.
10
00:01:59.240 --> 00:02:04.080
In my world: the routing rules, the approval chain, and who's on the night shift.
11
00:02:04.600 --> 00:02:13.660
Exactly. And now – the essay's point – AI doesn't take the decision. It rewrites the premises the decision rests on.
12
00:02:14.200 --> 00:02:16.140
Claire, the Tuesday in March.
13
00:02:17.180 --> 00:02:41.360
Our planning system comes from an external vendor. One Tuesday, the routes for our Swiss customers suddenly went through different border crossings. Longer, but cheaper on tolls. Nobody in my company had decided that. The vendor had updated the model – it now weighted toll costs more. And for three days, our dispatchers just followed the new routes, because the system said so.
14
00:02:41.840 --> 00:02:57.020
That's the essay in one story. A premise – what counts as a good route – was changed outside the organization, in model weights, without any member deciding it. And nobody noticed, because premises are invisible when they work.
15
00:02:57.400 --> 00:02:59.740
Isn't there also an optimistic reading?
16
00:03:00.180 --> 00:03:29.240
There is, and it's serious. Rules that were never written down – the implicit sense of a caseworker, "you know, we usually approve that" – can be made explicit and consistent by AI. Everyone gets the same treatment. Some theorists even say AI lets organizations reprogramme their own
Load full transcript
rules faster – not only the rules, but how they change the rules. That's the old promise of bureaucracy: predictable, equal decisions.
17
00:03:29.820 --> 00:03:30.800
And the pessimistic one?
18
00:03:31.760 --> 00:04:12.780
That the premises disappear from view. Two researchers, Kumar and Suthar, wrote a careful paper on exactly this: AI doesn't replace decision-making – it reorganises the premises that make selection possible. And a sociologist, Jenna Burrell, distinguished three kinds of opacity. Corporate secrecy – the vendor won't tell you. Technical illiteracy – you couldn't read it anyway. And the way machine learning itself works, at a scale and in a form that doesn't match human reasoning. Your Tuesday had all three.
19
00:04:14.280 --> 00:04:15.140
A full house.
20
00:04:15.700 --> 00:04:24.160
The essay asks: who should have the authority to change an AI-encoded premise – and how would members even notice?
21
00:04:24.820 --> 00:04:51.480
My answer, after March: nobody outside the company, without a signature from someone inside. We changed our contract with the vendor. Every model update that touches routing logic comes with release notes in plain language – "routes now weigh tolls more" – and it goes live only after our head of planning signs it. Like a change to the official route manual.
22
00:04:52.120 --> 00:04:56.340
So you turned a model update back into a decision. With a decider.
23
00:04:56.940 --> 00:04:58.840
With a name on it. Again.
24
00:04:59.380 --> 00:05:19.860
My hunch from theory is the same, a bit more abstract. Treat AI premises as versioned artefacts – like the rulebook, but with version numbers. Every version has an owner, a date, and a reason. If you can't say who changed the premise and why, you don't have a premise. You have weather.
25
00:05:20.520 --> 00:05:27.020
Weather. That's exactly how it felt. The routes just – changed. Like rain.
26
00:05:31.520 --> 00:05:38.580
Second essay, and the title is unsettling. The human in the crumple zone. Niels?
27
00:05:39.140 --> 00:06:02.060
It starts with an old idea from nineteen fifty-eight. James March and Herbert Simon wrote that organizations absorb uncertainty. Someone looks at messy evidence, draws a conclusion, and passes on the conclusion – not the evidence. The next person trusts the conclusion because they trust the person who drew it.
28
00:06:02.580 --> 00:06:07.220
That's every report I get. I see the summary, never the raw data.
29
00:06:07.760 --> 00:06:25.909
Now put an AI system there. The model draws the conclusion. The uncertainty is absorbed inside the model. And then a human "decides" – but what she receives is the conclusion, not the evidence. And the organization still attributes the decision to her.
30
00:06:26.410 --> 00:06:27.830
And the crumple zone?
31
00:06:28.230 --> 00:06:51.630
That's a term from Madeleine Clare Elish, an American anthropologist. In a car, the crumple zone is the part designed to be destroyed in a crash so that the
passengers survive. She argued that in automated systems, the human operator becomes a moral crumple zone – the one who absorbs the blame so that the system survives.
32
00:06:51.950 --> 00:06:53.350
The car in Arizona.
33
00:06:53.830 --> 00:07:19.050
Yes. In twenty eighteen, a self-driving test car hit and killed a woman pushing a bicycle in Tempe, Arizona. The car's software had detected her, but didn't classify her correctly in time, and the automatic emergency braking had been disabled. The safety driver in the car was the one who was prosecuted. The system was designed so that a human would be there to blame.
34
00:07:19.370 --> 00:07:20.410
That's brutal.
35
00:07:21.010 --> 00:07:27.530
It's very clear, though. The machine absorbed the uncertainty. The human absorbed the blame.
36
00:07:27.990 --> 00:07:44.130
And in a company, it's the same without the tragedy. If the planning system sends a refrigerated truck on a route that's too long, and the frozen goods thaw – who gets the call from the customer? Not the vendor. The dispatcher who clicked "approve".
37
00:07:44.550 --> 00:07:47.370
But isn't the human in the loop a safeguard?
38
00:07:48.130 --> 00:08:18.590
That's what every policy says. There's a study by Ben Green, who looked at forty-one policies worldwide that require human oversight of government algorithms. Two problems. First, people often simply can't do the oversight the policies expect – too many cases, too little information, too little time. Second, the policies give a false sense of security. The human becomes a legitimation, not a safeguard.
39
00:08:19.070 --> 00:08:34.550
Okay, but I have to defend my dispatchers here. If I remove the human, nobody is responsible at all. The software can't be fired, can't apologise, can't be sued in any meaningful way. Without the human, the customer has nobody to call.
40
00:08:34.950 --> 00:08:45.770
That's the antithesis, and it's correct. A machine can cause harm but can't answer for it. Removing the human removes the last address for responsibility.
41
00:08:46.210 --> 00:08:49.370
So we keep the human. The question is what we ask of them.
42
00:08:49.650 --> 00:08:58.250
The essay has exactly that question. What would a human in the loop look like that's designed for the person in it – not for the company's legal defence?
43
00:08:59.590 --> 00:09:22.790
Three things, from my side. First, the dispatcher sees why the system proposed something, at least the main reasons – not just the result. Second, they have time. If someone has to approve three hundred routes in an hour, they're not deciding, they're clicking. And third, the person who overrules the system and turns out to be wrong is protected. Otherwise nobody ever overrules.
44
00:09:23.230 --> 00:09:44.410
And I'd add one structural thing. Responsibility should sit where the premises are decided, not where the click happens. The people who chose the vendor, configured the model, approved the update – they decided the premises. The dispatcher is a sensor for cases that don't fit. Important, but not the one who should carry the whole weight.
45
00:09:44.750 --> 00:09:45.470
That means me.
46
00:09:45.810 --> 00:09:47.550
That means you. I'm sorry.
47
00:09:48.990 --> 00:09:51.950
I knew it. I always end up in the crumple zone.
48
00:09:52.710 --> 00:10:14.950
Bainbridge's ironies of automation apply here too, by the way – from nineteen eighty-three. The better the system, the less practice the human has for the rare case where it fails. So if you want meaningful oversight, you have to keep people practised. Let them do some routes by hand, regularly.
49
00:10:15.530 --> 00:10:27.950
We started that after the snow pass. Once a week, one depot plans its night routes manually and compares with the system. People grumbled. Now they bet on who's better.
50
00:10:28.629 --> 00:10:29.590
And who's better?
51
00:10:30.210 --> 00:10:38.350
The system, usually. But not on Fridays before a holiday. Ahmed wins every Friday before a holiday.
52
00:10:43.270 --> 00:10:51.810
Third essay. Companies now talk about AI agents as "digital colleagues". Niels, can an agent be a member?
53
00:10:52.890 --> 00:11:20.110
This is where Luhmann is very useful. Membership is the solution to a specific problem. Organizations need people to do things they wouldn't do on their own – arrive at eight, follow rules they didn't make, work on things they don't care about. How? By making it a condition of membership. You accept the expectations, or you leave. Membership gives the organization a lever over people who have their own interests.
54
00:11:20.770 --> 00:11:22.710
And an agent has no interests.
55
00:11:23.290 --> 00:11:38.810
An agent has no interests, needs no motivation, never threatens to leave. So in a way, it's the perfect member. It accepts every formal expectation. No informal coalitions, no creative interpretation of the rules.
56
00:11:40.070 --> 00:11:42.950
That sounds like a dream for an operations director.
57
00:11:43.450 --> 00:12:00.510
And that's the thesis. Agents already behave like members – they occupy positions in workflows, follow programmes, communicate through channels. Some companies arrange them in hierarchies, with a "manager agent" handing out tasks to other agents.
58
00:12:01.150 --> 00:12:02.670
And why isn't that enough?
59
00:12:03.270 --> 00:12:20.770
Because membership isn't only a lever for motivation. It's also how an organization makes someone accountable. A member can be blamed, promoted, dismissed. An agent can't be dismissed in any sense that matters to it. Membership has nothing to do for it – and nothing to hold on to.
60
00:12:21.950 --> 00:12:43.450
We use agents for customs paperwork now. One agent drafts, one checks, one sends. And I'll tell you what surprised me: copying our human team structure made them worse. When we set them up like a little department – a junior agent, a senior agent, a manager agent – they passed mistakes back and
forth. A simpler setup worked better.
61
00:12:44.190 --> 00:13:10.250
That matches some computational experiments in the essay. Organizing agents in imitation of human forms – teams, hierarchies, committees – often performed worse than simpler designs. Hand-offs lose information, and agents built on the same model make correlated mistakes. It's a committee where everyone has read the same book.
62
00:13:11.550 --> 00:13:12.950
That's my worst committee.
63
00:13:13.650 --> 00:13:16.870
So where do agents belong, if not inside?
64
00:13:17.450 --> 00:13:53.130
The precise systems-theory answer: they're part of the organization's environment – but an environment that now sits inside its workflows. Like machines or an external service provider. You steer them through programmes and contracts, not through membership. And that means the usual tools of control – hierarchy, careers, the informal order – don't reach them. You need a second line of governance: the prompts, the permissions, the rules for what the agent may do.
65
00:13:53.730 --> 00:14:07.730
In our case, that second line is two people in IT who understand the agent setup. Which worries me. The real structure of our customs process is in a configuration that only two people can read.
66
00:14:08.330 --> 00:14:17.030
That's exactly the risk. The org chart describes the members. It no longer describes how the work is actually coordinated.
67
00:14:17.650 --> 00:14:25.190
The essay asks whether organizations will create a formal status for agents – something between tool and member.
68
00:14:25.850 --> 00:14:58.610
My guess: yes, and it will look boring. Something like a register. Every agent has an entry: what it may do, what money it can spend, which data it can see, who is responsible for it, and when it was last checked. In German law there's the old idea of a "Prokura" – a defined power to act on behalf of a firm, registered publicly. Agents might get something like a technical Prokura. Not membership – a mandate with limits and an owner.
69
00:14:59.190 --> 00:15:09.990
I'd sign up for that tomorrow. And I'd add an expiry date. Every agent's mandate runs out after six months unless someone renews it and looks at what it did.
70
00:15:10.590 --> 00:15:15.130
The sunset clause again. We had that in your colleagues' season, I believe.
71
00:15:16.970 --> 00:15:18.050
You've been listening.
72
00:15:18.570 --> 00:15:20.410
Research. I'm thorough.
73
00:15:21.330 --> 00:16:11.910
Let me gather this. You agree that AI rarely takes the decision itself – it changes the premises the decision rests on, sometimes silently, from outside. Claire's answer: every premise change needs a signature inside the company. Niels': premises as versioned artefacts with an owner – otherwise it's weather. You agree that the human who signs off an AI recommendation often gets the conclusion without the evidence – and the blame without the power. And that responsibility should sit where the premises are decided, not where the click happens. On agents, you agree they can do the work but can't join – and that they need a mandate with limits, an owner and an expiry date.
74
00:16:12.590 --> 00:16:14.990
Where do we disagree? I'm worried.
75
00:16:15.630 --> 00:16:21.390
You're more optimistic that the vendor will keep writing plain-language release notes.
76
00:16:22.870 --> 00:16:26.790
That's true. It's in the contract. Contracts work.
77
00:16:27.429 --> 00:16:29.830
Until the vendor is bought by a bigger vendor.
78
00:16:30.390 --> 00:16:41.590
Here's the question I'd like to leave with you, listening. Think of the last decision at work that you signed but didn't really make. Who made it – and did anyone know?
79
00:16:42.590 --> 00:17:04.730
The essays and sources are on ideation.zone. Next time: the three sides of every organization – the formal rules, the informal practice, and the face it shows the world. Including the AI everyone uses and nobody is allowed to. Niels, Claire – thank you.
80
00:17:05.310 --> 00:17:06.370
Thank you, Elisabeth.
81
00:17:07.010 --> 00:17:13.050
Merci. And Niels – I'm sending you our agent register. For the drawer.