1
00:00:15,042 --> 00:00:18,871
I'm Arshavir Blackwell and this is Inside
the Black Box.
2
00:00:18,871 --> 00:00:22,546
Today's work was done in collaboration
with John Holman.
3
00:00:22,546 --> 00:00:27,225
This is the fourth installment in our
series on what happens inside a
4
00:00:27,225 --> 00:00:29,724
language model when you fine-tune it.
5
00:00:29,724 --> 00:00:32,599
If you're new here, a quick catch-up.
6
00:00:32,599 --> 00:00:36,958
We've been using a technique called LoRA
to teach a language model to
7
00:00:36,958 --> 00:00:42,467
write like specific authors, and then
using tools called sparse autoencoders
8
00:00:42,467 --> 00:00:47,207
to look inside the model and see how it
organized what it learned.
9
00:00:47,207 --> 00:00:50,799
Think of the sparse autoencoder as a
prism.
10
00:00:50,799 --> 00:00:55,342
You point it at the tangled internal
activity of a neural network,
11
00:00:55,342 --> 00:00:59,251
and it breaks that activity into
individual features,
12
00:00:59,251 --> 00:01:06,162
individual colors, each one responding to
something specific in the text.
13
00:00:59,251 --> 00:01:06,162
In
14
00:01:06,162 --> 00:01:10,725
the first three articles, we trained the
model on Marcus Aurelius and
15
00:01:10,725 --> 00:01:12,183
on advertising copy.
16
00:01:12,183 --> 00:01:17,372
We kept finding the same structural
patterns: groups of features that
17
00:01:17,372 --> 00:01:20,242
fire together and matter together,
18
00:01:20,242 --> 00:01:24,652
two different ways the model encodes
information (some groups moving
19
00:01:24,652 --> 00:01:28,452
in lockstep,
others cooperating more loosely),
20
00:01:28,452 --> 00:01:33,399
and a strange phenomenon we called
representation without function.
21
00:01:33,399 --> 00:01:38,579
Clusters of features that exist inside
the model but don't appear to
22
00:01:38,579 --> 00:01:43,757
do anything when you test them.
23
00:01:38,579 --> 00:01:43,757
All of that came from two corpora.
24
00:01:43,757 --> 00:01:45,354
Two types of text.
25
00:01:45,354 --> 00:01:50,034
The question we're asking today is
simple: does any of this hold up when
26
00:01:50,034 --> 00:01:56,562
we try a completely new author?
27
00:01:50,034 --> 00:01:56,562
The answer is yes.
28
00:01:50,034 --> 00:01:56,562
We fine-tuned the same
29
00:01:56,562 --> 00:02:01,360
base model, Llama 3.
30
00:01:56,562 --> 00:02:01,360
2 3B-Instruct,
31
00:02:01,360 --> 00:02:03,034
on the letters of Seneca.
32
00:02:03,034 --> 00:02:09,345
Specifically, the Gummere translation of
his Moral Letters to Lucilius.
33
00:02:09,345 --> 00:02:12,298
Same LoRA method, same settings,
34
00:02:12,298 --> 00:02:14,737
same analysis pipeline.
35
00:02:14,737 --> 00:02:19,075
A completely fresh model,
trained from scratch on Seneca.
36
00:02:19,075 --> 00:02:24,636
No connection whatsoever to the Marcus
model we built earlier.
37
00:02:19,075 --> 00:02:24,636
The corpus
38
00:02:24,636 --> 00:02:33,160
was about 209, 000 words, broken into 1,
139 evaluation chunks.
39
00:02:24,636 --> 00:02:33,160
Now,
40
00:02:33,160 --> 00:02:37,660
in Article 1, we used something called a
crosscoder to compare the base
41
00:02:37,660 --> 00:02:39,187
model and the Marcus model.
42
00:02:39,187 --> 00:02:43,661
I want to explain what that is and why we
stopped using it,
43
00:02:43,661 --> 00:02:48,700
because it matters for everything that
follows.
44
00:02:43,661 --> 00:02:48,700
A crosscoder is a special
45
00:02:48,700 --> 00:02:53,105
type of sparse autoencoder with two input
channels.
46
00:02:53,105 --> 00:02:56,458
Think of it as two doors leading into the
same room.
47
00:02:56,458 --> 00:03:00,215
The base model's activations come in
through one door,
48
00:03:00,215 --> 00:03:03,480
the LoRA model's activations come in
through the other.
49
00:03:03,480 --> 00:03:08,023
The crosscoder knows which door each
activation came through,
50
00:03:08,023 --> 00:03:09,933
and that one bit of information,
51
00:03:09,933 --> 00:03:16,560
base or LoRA, is all it needs to sort
features on its own during training.
52
00:03:16,560 --> 00:03:22,618
Features that only fire on activations
from one door get classified
53
00:03:16,560 --> 00:03:22,618
automatically.
54
00:03:22,618 --> 00:03:27,190
No manual cutoff needed.
55
00:03:22,618 --> 00:03:27,190
That worked great for a single experiment.
56
00:03:27,190 --> 00:03:28,630
But here's the problem.
57
00:03:28,630 --> 00:03:35,521
Each crosscoder learns its own dividing
line between "shared" and "LoRA-specific."
58
00:03:35,521 --> 00:03:38,617
You don't control where that line falls.
59
00:03:38,617 --> 00:03:41,725
Train one on Marcus, it draws one line.
60
00:03:41,725 --> 00:03:45,239
Train another on Seneca,
it draws a different line.
61
00:03:45,239 --> 00:03:50,181
When you want to put three corpora side
by side in a comparison table,
62
00:03:50,181 --> 00:03:53,217
you need the same measuring stick every
time.
63
00:03:53,217 --> 00:03:58,914
The crosscoder can't give you that.
64
00:03:53,217 --> 00:03:58,914
So we switched to what we call a combined
65
00:03:58,914 --> 00:03:59,554
SAE.
66
00:03:59,554 --> 00:04:02,104
And the difference, in one sentence,
67
00:04:02,104 --> 00:04:07,128
is this: the crosscoder trains on paired
vectors with labels,
68
00:04:07,128 --> 00:04:08,807
base or LoRA.
69
00:04:08,807 --> 00:04:15,322
The combined SAE puts everything in one
bucket and sorts after.
70
00:04:08,807 --> 00:04:15,322
Here's
71
00:04:15,322 --> 00:04:16,440
how it works.
72
00:04:16,440 --> 00:04:20,876
We take all 1, 139 text chunks,
73
00:04:20,876 --> 00:04:24,871
run each one through both the base model
and the Seneca LoRA,
74
00:04:24,871 --> 00:04:30,082
and shuffle all the resulting activations
together into one big training
75
00:04:30,082 --> 00:04:30,542
set.
76
00:04:30,542 --> 00:04:32,222
One door.
77
00:04:32,222 --> 00:04:34,767
Everything walks through the same
entrance.
78
00:04:34,767 --> 00:04:39,550
The SAE has no idea which activation came
from which model.
79
00:04:39,550 --> 00:04:44,478
It just learns features from the mixed
pile.
80
00:04:39,550 --> 00:04:44,478
After training is done,
81
00:04:44,478 --> 00:04:48,069
we go back and classify each feature
ourselves.
82
00:04:48,069 --> 00:04:50,466
We ask: when this feature fired,
83
00:04:50,466 --> 00:04:55,583
what fraction of the time was it firing
on LoRA activations versus base
84
00:04:55,583 --> 00:04:56,849
activations?
85
00:04:56,849 --> 00:05:00,594
That gives us a number called the
specificity ratio,
86
00:05:00,594 --> 00:05:07,064
on a scale from zero (fires only on the
base model) to one (fires only
87
00:05:07,064 --> 00:05:08,502
on the LoRA).
88
00:05:08,502 --> 00:05:13,126
Features above 0.
89
00:05:08,502 --> 00:05:13,126
8, we call Seneca-specific.
90
00:05:13,126 --> 00:05:15,720
The LoRA built those.
91
00:05:15,720 --> 00:05:21,189
Features between 0.
92
00:05:15,720 --> 00:05:21,189
2 and 0.
93
00:05:15,720 --> 00:05:21,189
8, we call shared.
94
00:05:21,189 --> 00:05:23,745
Both models use them.
95
00:05:23,745 --> 00:05:30,216
Features below 0.
96
00:05:23,745 --> 00:05:30,216
2 fire mostly on the base model and we
exclude them,
97
00:05:30,216 --> 00:05:33,328
since we're interested in what the LoRA
is doing.
98
00:05:33,328 --> 00:05:38,441
And crucially, we applied the same rule
to all three corpora.
99
00:05:38,441 --> 00:05:42,194
Marcus, Seneca, and ad copy.
100
00:05:42,194 --> 00:05:45,554
Same cutoffs, same measuring stick,
101
00:05:45,554 --> 00:05:50,476
every time.
102
00:05:45,554 --> 00:05:50,476
Out of 12, 000 features in the SAE,
103
00:05:50,476 --> 00:05:55,196
1, 194 were active on Seneca text.
104
00:05:55,196 --> 00:05:58,076
56 were Seneca-specific.
105
00:05:58,076 --> 00:06:01,112
265 were shared.
106
00:06:01,112 --> 00:06:04,951
The rest were base-dominant and got
excluded.
107
00:06:04,951 --> 00:06:07,031
That ratio matters.
108
00:06:07,031 --> 00:06:12,702
The Seneca LoRA is mostly reusing
features the base model already has.
109
00:06:12,702 --> 00:06:15,806
It's only building a small number of new
ones.
110
00:06:15,806 --> 00:06:18,086
Marcus was similar.
111
00:06:18,086 --> 00:06:23,221
Ad copy, which had to handle twelve
wildly different commercial styles,
112
00:06:23,221 --> 00:06:26,842
needed more new machinery.
113
00:06:23,221 --> 00:06:26,842
Here's the catch.
114
00:06:26,842 --> 00:06:32,089
We just said the LoRA-specific features
respond to patterns in Seneca's
115
00:06:32,089 --> 00:06:32,594
prose.
116
00:06:32,594 --> 00:06:35,791
But when we try to interpret them one at
a time,
117
00:06:35,791 --> 00:06:38,904
using a technique called the logit lens,
118
00:06:38,904 --> 00:06:41,219
most of them are incoherent.
119
00:06:41,219 --> 00:06:45,957
You project a feature through the model's
vocabulary to see what words
120
00:06:45,957 --> 00:06:56,510
it points toward,
and you get things like "undy, strup,
121
00:06:45,957 --> 00:06:56,510
Gain." Or "Paren, ethical, Sev, adem."
122
00:06:56,510 --> 00:07:04,355
Or "swer, epad, bins, Buster." These are
not meaningful word lists.
123
00:07:04,355 --> 00:07:08,193
They're noise.
124
00:07:04,355 --> 00:07:08,193
This is the same thing we found in Marcus.
125
00:07:08,193 --> 00:07:11,545
Most individual features, taken alone,
126
00:07:11,545 --> 00:07:16,754
don't map to recognizable concepts.
127
00:07:11,545 --> 00:07:16,754
There are a few exceptions.
128
00:07:16,754 --> 00:07:21,989
A couple of features project cleanly onto
the word "very." One maps to
129
00:07:21,989 --> 00:07:30,483
"running, runs, ran." One shows a mild
signal toward "surroundings, environment,
130
00:07:21,989 --> 00:07:30,483
transitional,"
131
00:07:30,483 --> 00:07:34,434
which has some connection to Seneca's
theme of equanimity.
132
00:07:34,434 --> 00:07:37,149
But these readable features are rare,
133
00:07:37,149 --> 00:07:41,287
and they share a telling property: they
have the highest peak signals
134
00:07:41,287 --> 00:07:43,136
and the least causal weight.
135
00:07:43,136 --> 00:07:47,203
The features you can interpret
individually are the ones that matter
136
00:07:47,203 --> 00:07:51,235
least.
137
00:07:47,203 --> 00:07:51,235
The interpretable unit is not the feature.
138
00:07:51,235 --> 00:07:52,595
It's the cluster.
139
00:07:52,595 --> 00:07:58,171
Think of individual features like
individual pixels in a photograph.
140
00:07:58,171 --> 00:08:03,279
You can stare at one pixel and try to
guess what the picture is,
141
00:08:03,279 --> 00:08:04,237
but you'll fail.
142
00:08:04,237 --> 00:08:11,255
The meaning only emerges when you look at
which pixels light up together.
143
00:08:04,237 --> 00:08:11,255
When
144
00:08:11,255 --> 00:08:13,838
we clustered Seneca's shared features,
145
00:08:13,838 --> 00:08:18,067
five groups,
same method we used on Marcus and ad copy,
146
00:08:18,067 --> 00:08:22,692
we got a silhouette score of 0.
147
00:08:18,067 --> 00:08:22,692
1415.
148
00:08:22,692 --> 00:08:25,246
If you haven't heard that term before,
149
00:08:25,246 --> 00:08:28,592
think of it as a measure of how
well-separated the groups are.
150
00:08:28,592 --> 00:08:32,666
A high score means the clusters are
distinct from each other.
151
00:08:32,666 --> 00:08:36,331
The features in each group really do
belong together,
152
00:08:36,331 --> 00:08:43,551
and really are different from features in
other groups.
153
00:08:36,331 --> 00:08:43,551
0.1415 is higher
154
00:08:43,551 --> 00:08:51,824
than Marcus at 0.
155
00:08:43,551 --> 00:08:51,824
0737 and more than double ad copy at 0.
156
00:08:43,551 --> 00:08:51,824
061.
157
00:08:51,824 --> 00:08:57,580
The model finds more natural groupings in
Seneca's philosophical text
158
00:08:57,580 --> 00:08:59,633
than in anything else we've tested.
159
00:08:59,633 --> 00:09:06,200
Seneca's LoRA produced the cleanest
internal organization we've seen.
160
00:08:59,633 --> 00:09:06,200
The
161
00:09:06,200 --> 00:09:11,414
tight-versus-loose pattern replicates
exactly.
162
00:09:06,200 --> 00:09:11,414
One tight cluster out of
163
00:09:11,414 --> 00:09:11,878
five.
164
00:09:11,878 --> 00:09:19,708
Cluster 4, with 64 features,
where a single axis of variation explains
165
00:09:19,708 --> 00:09:22,178
81% of everything happening in that
cluster.
166
00:09:22,178 --> 00:09:25,294
These features move in near-lockstep.
167
00:09:25,294 --> 00:09:28,009
Like a choir singing in unison.
168
00:09:28,009 --> 00:09:32,401
The remaining four clusters are loose
coalitions.
169
00:09:32,401 --> 00:09:36,064
Features that fire together but with more
independence.
170
00:09:36,064 --> 00:09:41,359
Like a jazz ensemble where everyone is
playing in the same key but improvising
171
00:09:41,359 --> 00:09:45,835
their own parts.
172
00:09:41,359 --> 00:09:45,835
This is the same ratio we found in Marcus.
173
00:09:45,835 --> 00:09:48,313
One tight, four loose.
174
00:09:48,313 --> 00:09:51,431
And the same ratio we found in ad copy.
175
00:09:51,431 --> 00:09:54,231
One tight, four loose.
176
00:09:54,231 --> 00:09:58,544
Three corpora,
three very different types of text,
177
00:09:58,544 --> 00:10:04,406
and the model organizes its adaptation
the same way every time.
178
00:09:58,544 --> 00:10:04,406
A quick
179
00:10:04,406 --> 00:10:05,868
note on methodology here.
180
00:10:05,868 --> 00:10:09,571
The Marcus numbers we're comparing
against are slightly different from
181
00:10:09,571 --> 00:10:11,931
what we reported in Articles 2 and 3.
182
00:10:11,931 --> 00:10:17,268
That's because we went back and applied
the same combined SAE approach
183
00:10:17,268 --> 00:10:20,640
and the same specificity cutoffs to
Marcus.
184
00:10:20,640 --> 00:10:23,833
We had to,
in order to make the comparison fair.
185
00:10:23,833 --> 00:10:27,511
This removed five features that were
base-dominant.
186
00:10:27,511 --> 00:10:30,227
The cluster sizes shifted slightly.
187
00:10:30,227 --> 00:10:36,437
The structural findings stayed the same.
188
00:10:30,227 --> 00:10:36,437
Seneca's Cluster 2 is a beast.
189
00:10:36,437 --> 00:10:41,787
141 features, the largest cluster in any
corpus we've tested.
190
00:10:41,787 --> 00:10:47,842
And a peak causal delta of 2.
191
00:10:41,787 --> 00:10:47,842
9482.
192
00:10:41,787 --> 00:10:47,842
What does that mean?
193
00:10:47,842 --> 00:10:50,318
We take the cluster's features,
194
00:10:50,318 --> 00:10:52,718
zero them out through the SAE,
195
00:10:52,718 --> 00:10:56,948
and measure how much the model's
next-word predictions shift.
196
00:10:56,948 --> 00:11:01,182
The bigger the number, the more the model
depends on that cluster.
197
00:11:01,182 --> 00:11:04,855
Marcus peaked at about 0.
198
00:11:01,182 --> 00:11:04,855
94.
199
00:11:04,855 --> 00:11:07,972
Ad copy peaked at 0.
200
00:11:04,855 --> 00:11:07,972
65.
201
00:11:07,972 --> 00:11:11,570
Seneca's peak is 2.
202
00:11:07,972 --> 00:11:11,570
95.
203
00:11:11,570 --> 00:11:17,618
The Seneca LoRA concentrates more of its
learned adaptation into a single
204
00:11:17,618 --> 00:11:24,291
dominant cluster than either previous
corpus.
205
00:11:17,618 --> 00:11:24,291
What does Cluster 2 actually
206
00:11:24,291 --> 00:11:25,559
respond to?
207
00:11:25,559 --> 00:11:28,118
Philosophical argumentation.
208
00:11:28,118 --> 00:11:32,664
Its top-firing passages are about the
advice of Attalus,
209
00:11:32,664 --> 00:11:35,852
the poetry of Cleanthes addressing
Jupiter,
210
00:11:35,852 --> 00:11:40,083
and a meditation on humanity's
relationship with nature.
211
00:11:40,083 --> 00:11:46,792
The top causal token is "for," the
connective tissue of reasoned argument.
212
00:11:46,792 --> 00:11:54,282
This cluster encodes how Seneca builds
and connects philosophical claims.
213
00:11:46,792 --> 00:11:54,282
When
214
00:11:54,282 --> 00:11:59,074
we measure how much better the Seneca
LoRA's SAE reconstructs Seneca text
215
00:11:59,074 --> 00:12:01,790
compared to the base model's SAE,
216
00:12:01,790 --> 00:12:04,747
on held-out text the model never trained
on,
217
00:12:04,747 --> 00:12:09,762
we get a median gap of 2.
218
00:12:04,747 --> 00:12:09,762
78 times.
219
00:12:04,747 --> 00:12:09,762
That's real.
220
00:12:09,762 --> 00:12:13,921
The LoRA genuinely learned
Seneca-specific structure.
221
00:12:13,921 --> 00:12:17,514
But it's smaller than Marcus at about
five times,
222
00:12:17,514 --> 00:12:23,752
and much smaller than ad copy at 7.
223
00:12:17,514 --> 00:12:23,752
7 times.
224
00:12:17,514 --> 00:12:23,752
The ordering makes sense.
225
00:12:23,752 --> 00:12:28,469
Seneca's philosophical prose is closer to
the kind of text the base model
226
00:12:28,469 --> 00:12:29,929
already handles well.
227
00:12:29,929 --> 00:12:34,407
Ad copy,
with its extreme stylistic variety,
228
00:12:34,407 --> 00:12:39,720
things like "bestie we need to talk about
your jewelry situation," that's
229
00:12:39,720 --> 00:12:41,918
the farthest from standard English,
230
00:12:41,918 --> 00:12:44,316
so the LoRA has to do the most work.
231
00:12:44,316 --> 00:12:46,509
Marcus sits in the middle.
232
00:12:46,509 --> 00:12:51,304
Seneca, whose prose is arguably the most
conventionally readable of the
233
00:12:51,304 --> 00:12:56,375
three,
requires the least new construction.
234
00:12:51,304 --> 00:12:56,375
This confirms something we
235
00:12:56,375 --> 00:12:57,998
suggested in Article 3.
236
00:12:57,998 --> 00:13:02,363
The cross-reconstruction gap isn't just
measuring whether the LoRA learned
237
00:13:02,363 --> 00:13:02,875
something.
238
00:13:02,875 --> 00:13:06,310
It's measuring how much work the LoRA had
to do.
239
00:13:06,310 --> 00:13:10,623
The distance between the target text and
what the model could already
240
00:13:10,623 --> 00:13:19,172
produce.
241
00:13:10,623 --> 00:13:19,172
The Seneca LoRA created 56 purpose-built
features out of 1, 194
242
00:13:19,172 --> 00:13:20,582
active features.
243
00:13:20,582 --> 00:13:23,220
That's about 4.
244
00:13:20,582 --> 00:13:23,220
7%.
245
00:13:23,220 --> 00:13:26,896
Marcus came in at 5.
246
00:13:23,220 --> 00:13:26,896
2%.
247
00:13:26,896 --> 00:13:28,333
Nearly identical.
248
00:13:28,333 --> 00:13:33,379
Both are single philosophical voices that
are relatively close to what
249
00:13:33,379 --> 00:13:37,162
the model already knows.
250
00:13:33,379 --> 00:13:37,162
Ad copy,
251
00:13:37,162 --> 00:13:41,161
with twelve wildly different styles
crammed into one LoRA,
252
00:13:41,161 --> 00:13:44,596
needed 12.
253
00:13:41,161 --> 00:13:44,596
2% purpose-built features.
254
00:13:44,596 --> 00:13:46,074
The pattern is clear.
255
00:13:46,074 --> 00:13:50,670
The more different the target text is
from what the model already knows,
256
00:13:50,670 --> 00:13:53,143
the more new machinery it has to build.
257
00:13:53,143 --> 00:13:58,453
Philosophical text reuses most of the
existing features and only builds
258
00:13:58,453 --> 00:14:00,327
a small number of new ones.
259
00:14:00,327 --> 00:14:05,369
Commercial copy across twelve registers
needs roughly twice the proportion
260
00:14:05,369 --> 00:14:09,992
of dedicated features.
261
00:14:05,369 --> 00:14:09,992
Every structural finding from Marcus and
ad copy
262
00:14:09,992 --> 00:14:11,270
holds for Seneca.
263
00:14:11,270 --> 00:14:14,543
Two encoding regimes: one tight,
264
00:14:14,543 --> 00:14:15,661
four loose.
265
00:14:15,661 --> 00:14:20,291
Cluster-level causal effects that dwarf
individual features.
266
00:14:20,291 --> 00:14:22,045
An inert cluster.
267
00:14:22,045 --> 00:14:26,507
The cross-reconstruction gap scales with
task difficulty.
268
00:14:26,507 --> 00:14:31,559
Feature specificity does too.
269
00:14:26,507 --> 00:14:31,559
Three corpora,
270
00:14:31,559 --> 00:14:34,273
three very different kinds of text,
271
00:14:34,273 --> 00:14:40,302
and the model organizes its adaptation
the same way every time.
272
00:14:34,273 --> 00:14:40,302
That's
273
00:14:40,302 --> 00:14:42,836
the replication story, and it matters,
274
00:14:42,836 --> 00:14:46,430
because a pattern that shows up once
could be an artifact.
275
00:14:46,430 --> 00:14:52,668
A pattern that shows up three times is
structure.
276
00:14:46,430 --> 00:14:52,668
But replication is only
277
00:14:52,668 --> 00:14:54,873
half of what Seneca gave us.
278
00:14:54,873 --> 00:14:58,300
Now that we know the structural findings
are real,
279
00:14:58,300 --> 00:15:03,220
the next question is: what can a second
philosophical corpus teach us
280
00:15:03,220 --> 00:15:04,827
that a single one couldn't?
281
00:15:04,827 --> 00:15:07,544
It turns out, quite a lot.
282
00:15:07,544 --> 00:15:09,780
We'll pick that up next time.
283
00:15:09,780 --> 00:15:15,694
I'm Arshavir Blackwell and this has been
Inside the Black Box.
00:00:15,042 --> 00:00:18,871
I'm Arshavir Blackwell and this is Inside
the Black Box.
2
00:00:18,871 --> 00:00:22,546
Today's work was done in collaboration
with John Holman.
3
00:00:22,546 --> 00:00:27,225
This is the fourth installment in our
series on what happens inside a
4
00:00:27,225 --> 00:00:29,724
language model when you fine-tune it.
5
00:00:29,724 --> 00:00:32,599
If you're new here, a quick catch-up.
6
00:00:32,599 --> 00:00:36,958
We've been using a technique called LoRA
to teach a language model to
7
00:00:36,958 --> 00:00:42,467
write like specific authors, and then
using tools called sparse autoencoders
8
00:00:42,467 --> 00:00:47,207
to look inside the model and see how it
organized what it learned.
9
00:00:47,207 --> 00:00:50,799
Think of the sparse autoencoder as a
prism.
10
00:00:50,799 --> 00:00:55,342
You point it at the tangled internal
activity of a neural network,
11
00:00:55,342 --> 00:00:59,251
and it breaks that activity into
individual features,
12
00:00:59,251 --> 00:01:06,162
individual colors, each one responding to
something specific in the text.
13
00:00:59,251 --> 00:01:06,162
In
14
00:01:06,162 --> 00:01:10,725
the first three articles, we trained the
model on Marcus Aurelius and
15
00:01:10,725 --> 00:01:12,183
on advertising copy.
16
00:01:12,183 --> 00:01:17,372
We kept finding the same structural
patterns: groups of features that
17
00:01:17,372 --> 00:01:20,242
fire together and matter together,
18
00:01:20,242 --> 00:01:24,652
two different ways the model encodes
information (some groups moving
19
00:01:24,652 --> 00:01:28,452
in lockstep,
others cooperating more loosely),
20
00:01:28,452 --> 00:01:33,399
and a strange phenomenon we called
representation without function.
21
00:01:33,399 --> 00:01:38,579
Clusters of features that exist inside
the model but don't appear to
22
00:01:38,579 --> 00:01:43,757
do anything when you test them.
23
00:01:38,579 --> 00:01:43,757
All of that came from two corpora.
24
00:01:43,757 --> 00:01:45,354
Two types of text.
25
00:01:45,354 --> 00:01:50,034
The question we're asking today is
simple: does any of this hold up when
26
00:01:50,034 --> 00:01:56,562
we try a completely new author?
27
00:01:50,034 --> 00:01:56,562
The answer is yes.
28
00:01:50,034 --> 00:01:56,562
We fine-tuned the same
29
00:01:56,562 --> 00:02:01,360
base model, Llama 3.
30
00:01:56,562 --> 00:02:01,360
2 3B-Instruct,
31
00:02:01,360 --> 00:02:03,034
on the letters of Seneca.
32
00:02:03,034 --> 00:02:09,345
Specifically, the Gummere translation of
his Moral Letters to Lucilius.
33
00:02:09,345 --> 00:02:12,298
Same LoRA method, same settings,
34
00:02:12,298 --> 00:02:14,737
same analysis pipeline.
35
00:02:14,737 --> 00:02:19,075
A completely fresh model,
trained from scratch on Seneca.
36
00:02:19,075 --> 00:02:24,636
No connection whatsoever to the Marcus
model we built earlier.
37
00:02:19,075 --> 00:02:24,636
The corpus
38
00:02:24,636 --> 00:02:33,160
was about 209, 000 words, broken into 1,
139 evaluation chunks.
39
00:02:24,636 --> 00:02:33,160
Now,
40
00:02:33,160 --> 00:02:37,660
in Article 1, we used something called a
crosscoder to compare the base
41
00:02:37,660 --> 00:02:39,187
model and the Marcus model.
42
00:02:39,187 --> 00:02:43,661
I want to explain what that is and why we
stopped using it,
43
00:02:43,661 --> 00:02:48,700
because it matters for everything that
follows.
44
00:02:43,661 --> 00:02:48,700
A crosscoder is a special
45
00:02:48,700 --> 00:02:53,105
type of sparse autoencoder with two input
channels.
46
00:02:53,105 --> 00:02:56,458
Think of it as two doors leading into the
same room.
47
00:02:56,458 --> 00:03:00,215
The base model's activations come in
through one door,
48
00:03:00,215 --> 00:03:03,480
the LoRA model's activations come in
through the other.
49
00:03:03,480 --> 00:03:08,023
The crosscoder knows which door each
activation came through,
50
00:03:08,023 --> 00:03:09,933
and that one bit of information,
51
00:03:09,933 --> 00:03:16,560
base or LoRA, is all it needs to sort
features on its own during training.
52
00:03:16,560 --> 00:03:22,618
Features that only fire on activations
from one door get classified
53
00:03:16,560 --> 00:03:22,618
automatically.
54
00:03:22,618 --> 00:03:27,190
No manual cutoff needed.
55
00:03:22,618 --> 00:03:27,190
That worked great for a single experiment.
56
00:03:27,190 --> 00:03:28,630
But here's the problem.
57
00:03:28,630 --> 00:03:35,521
Each crosscoder learns its own dividing
line between "shared" and "LoRA-specific."
58
00:03:35,521 --> 00:03:38,617
You don't control where that line falls.
59
00:03:38,617 --> 00:03:41,725
Train one on Marcus, it draws one line.
60
00:03:41,725 --> 00:03:45,239
Train another on Seneca,
it draws a different line.
61
00:03:45,239 --> 00:03:50,181
When you want to put three corpora side
by side in a comparison table,
62
00:03:50,181 --> 00:03:53,217
you need the same measuring stick every
time.
63
00:03:53,217 --> 00:03:58,914
The crosscoder can't give you that.
64
00:03:53,217 --> 00:03:58,914
So we switched to what we call a combined
65
00:03:58,914 --> 00:03:59,554
SAE.
66
00:03:59,554 --> 00:04:02,104
And the difference, in one sentence,
67
00:04:02,104 --> 00:04:07,128
is this: the crosscoder trains on paired
vectors with labels,
68
00:04:07,128 --> 00:04:08,807
base or LoRA.
69
00:04:08,807 --> 00:04:15,322
The combined SAE puts everything in one
bucket and sorts after.
70
00:04:08,807 --> 00:04:15,322
Here's
71
00:04:15,322 --> 00:04:16,440
how it works.
72
00:04:16,440 --> 00:04:20,876
We take all 1, 139 text chunks,
73
00:04:20,876 --> 00:04:24,871
run each one through both the base model
and the Seneca LoRA,
74
00:04:24,871 --> 00:04:30,082
and shuffle all the resulting activations
together into one big training
75
00:04:30,082 --> 00:04:30,542
set.
76
00:04:30,542 --> 00:04:32,222
One door.
77
00:04:32,222 --> 00:04:34,767
Everything walks through the same
entrance.
78
00:04:34,767 --> 00:04:39,550
The SAE has no idea which activation came
from which model.
79
00:04:39,550 --> 00:04:44,478
It just learns features from the mixed
pile.
80
00:04:39,550 --> 00:04:44,478
After training is done,
81
00:04:44,478 --> 00:04:48,069
we go back and classify each feature
ourselves.
82
00:04:48,069 --> 00:04:50,466
We ask: when this feature fired,
83
00:04:50,466 --> 00:04:55,583
what fraction of the time was it firing
on LoRA activations versus base
84
00:04:55,583 --> 00:04:56,849
activations?
85
00:04:56,849 --> 00:05:00,594
That gives us a number called the
specificity ratio,
86
00:05:00,594 --> 00:05:07,064
on a scale from zero (fires only on the
base model) to one (fires only
87
00:05:07,064 --> 00:05:08,502
on the LoRA).
88
00:05:08,502 --> 00:05:13,126
Features above 0.
89
00:05:08,502 --> 00:05:13,126
8, we call Seneca-specific.
90
00:05:13,126 --> 00:05:15,720
The LoRA built those.
91
00:05:15,720 --> 00:05:21,189
Features between 0.
92
00:05:15,720 --> 00:05:21,189
2 and 0.
93
00:05:15,720 --> 00:05:21,189
8, we call shared.
94
00:05:21,189 --> 00:05:23,745
Both models use them.
95
00:05:23,745 --> 00:05:30,216
Features below 0.
96
00:05:23,745 --> 00:05:30,216
2 fire mostly on the base model and we
exclude them,
97
00:05:30,216 --> 00:05:33,328
since we're interested in what the LoRA
is doing.
98
00:05:33,328 --> 00:05:38,441
And crucially, we applied the same rule
to all three corpora.
99
00:05:38,441 --> 00:05:42,194
Marcus, Seneca, and ad copy.
100
00:05:42,194 --> 00:05:45,554
Same cutoffs, same measuring stick,
101
00:05:45,554 --> 00:05:50,476
every time.
102
00:05:45,554 --> 00:05:50,476
Out of 12, 000 features in the SAE,
103
00:05:50,476 --> 00:05:55,196
1, 194 were active on Seneca text.
104
00:05:55,196 --> 00:05:58,076
56 were Seneca-specific.
105
00:05:58,076 --> 00:06:01,112
265 were shared.
106
00:06:01,112 --> 00:06:04,951
The rest were base-dominant and got
excluded.
107
00:06:04,951 --> 00:06:07,031
That ratio matters.
108
00:06:07,031 --> 00:06:12,702
The Seneca LoRA is mostly reusing
features the base model already has.
109
00:06:12,702 --> 00:06:15,806
It's only building a small number of new
ones.
110
00:06:15,806 --> 00:06:18,086
Marcus was similar.
111
00:06:18,086 --> 00:06:23,221
Ad copy, which had to handle twelve
wildly different commercial styles,
112
00:06:23,221 --> 00:06:26,842
needed more new machinery.
113
00:06:23,221 --> 00:06:26,842
Here's the catch.
114
00:06:26,842 --> 00:06:32,089
We just said the LoRA-specific features
respond to patterns in Seneca's
115
00:06:32,089 --> 00:06:32,594
prose.
116
00:06:32,594 --> 00:06:35,791
But when we try to interpret them one at
a time,
117
00:06:35,791 --> 00:06:38,904
using a technique called the logit lens,
118
00:06:38,904 --> 00:06:41,219
most of them are incoherent.
119
00:06:41,219 --> 00:06:45,957
You project a feature through the model's
vocabulary to see what words
120
00:06:45,957 --> 00:06:56,510
it points toward,
and you get things like "undy, strup,
121
00:06:45,957 --> 00:06:56,510
Gain." Or "Paren, ethical, Sev, adem."
122
00:06:56,510 --> 00:07:04,355
Or "swer, epad, bins, Buster." These are
not meaningful word lists.
123
00:07:04,355 --> 00:07:08,193
They're noise.
124
00:07:04,355 --> 00:07:08,193
This is the same thing we found in Marcus.
125
00:07:08,193 --> 00:07:11,545
Most individual features, taken alone,
126
00:07:11,545 --> 00:07:16,754
don't map to recognizable concepts.
127
00:07:11,545 --> 00:07:16,754
There are a few exceptions.
128
00:07:16,754 --> 00:07:21,989
A couple of features project cleanly onto
the word "very." One maps to
129
00:07:21,989 --> 00:07:30,483
"running, runs, ran." One shows a mild
signal toward "surroundings, environment,
130
00:07:21,989 --> 00:07:30,483
transitional,"
131
00:07:30,483 --> 00:07:34,434
which has some connection to Seneca's
theme of equanimity.
132
00:07:34,434 --> 00:07:37,149
But these readable features are rare,
133
00:07:37,149 --> 00:07:41,287
and they share a telling property: they
have the highest peak signals
134
00:07:41,287 --> 00:07:43,136
and the least causal weight.
135
00:07:43,136 --> 00:07:47,203
The features you can interpret
individually are the ones that matter
136
00:07:47,203 --> 00:07:51,235
least.
137
00:07:47,203 --> 00:07:51,235
The interpretable unit is not the feature.
138
00:07:51,235 --> 00:07:52,595
It's the cluster.
139
00:07:52,595 --> 00:07:58,171
Think of individual features like
individual pixels in a photograph.
140
00:07:58,171 --> 00:08:03,279
You can stare at one pixel and try to
guess what the picture is,
141
00:08:03,279 --> 00:08:04,237
but you'll fail.
142
00:08:04,237 --> 00:08:11,255
The meaning only emerges when you look at
which pixels light up together.
143
00:08:04,237 --> 00:08:11,255
When
144
00:08:11,255 --> 00:08:13,838
we clustered Seneca's shared features,
145
00:08:13,838 --> 00:08:18,067
five groups,
same method we used on Marcus and ad copy,
146
00:08:18,067 --> 00:08:22,692
we got a silhouette score of 0.
147
00:08:18,067 --> 00:08:22,692
1415.
148
00:08:22,692 --> 00:08:25,246
If you haven't heard that term before,
149
00:08:25,246 --> 00:08:28,592
think of it as a measure of how
well-separated the groups are.
150
00:08:28,592 --> 00:08:32,666
A high score means the clusters are
distinct from each other.
151
00:08:32,666 --> 00:08:36,331
The features in each group really do
belong together,
152
00:08:36,331 --> 00:08:43,551
and really are different from features in
other groups.
153
00:08:36,331 --> 00:08:43,551
0.1415 is higher
154
00:08:43,551 --> 00:08:51,824
than Marcus at 0.
155
00:08:43,551 --> 00:08:51,824
0737 and more than double ad copy at 0.
156
00:08:43,551 --> 00:08:51,824
061.
157
00:08:51,824 --> 00:08:57,580
The model finds more natural groupings in
Seneca's philosophical text
158
00:08:57,580 --> 00:08:59,633
than in anything else we've tested.
159
00:08:59,633 --> 00:09:06,200
Seneca's LoRA produced the cleanest
internal organization we've seen.
160
00:08:59,633 --> 00:09:06,200
The
161
00:09:06,200 --> 00:09:11,414
tight-versus-loose pattern replicates
exactly.
162
00:09:06,200 --> 00:09:11,414
One tight cluster out of
163
00:09:11,414 --> 00:09:11,878
five.
164
00:09:11,878 --> 00:09:19,708
Cluster 4, with 64 features,
where a single axis of variation explains
165
00:09:19,708 --> 00:09:22,178
81% of everything happening in that
cluster.
166
00:09:22,178 --> 00:09:25,294
These features move in near-lockstep.
167
00:09:25,294 --> 00:09:28,009
Like a choir singing in unison.
168
00:09:28,009 --> 00:09:32,401
The remaining four clusters are loose
coalitions.
169
00:09:32,401 --> 00:09:36,064
Features that fire together but with more
independence.
170
00:09:36,064 --> 00:09:41,359
Like a jazz ensemble where everyone is
playing in the same key but improvising
171
00:09:41,359 --> 00:09:45,835
their own parts.
172
00:09:41,359 --> 00:09:45,835
This is the same ratio we found in Marcus.
173
00:09:45,835 --> 00:09:48,313
One tight, four loose.
174
00:09:48,313 --> 00:09:51,431
And the same ratio we found in ad copy.
175
00:09:51,431 --> 00:09:54,231
One tight, four loose.
176
00:09:54,231 --> 00:09:58,544
Three corpora,
three very different types of text,
177
00:09:58,544 --> 00:10:04,406
and the model organizes its adaptation
the same way every time.
178
00:09:58,544 --> 00:10:04,406
A quick
179
00:10:04,406 --> 00:10:05,868
note on methodology here.
180
00:10:05,868 --> 00:10:09,571
The Marcus numbers we're comparing
against are slightly different from
181
00:10:09,571 --> 00:10:11,931
what we reported in Articles 2 and 3.
182
00:10:11,931 --> 00:10:17,268
That's because we went back and applied
the same combined SAE approach
183
00:10:17,268 --> 00:10:20,640
and the same specificity cutoffs to
Marcus.
184
00:10:20,640 --> 00:10:23,833
We had to,
in order to make the comparison fair.
185
00:10:23,833 --> 00:10:27,511
This removed five features that were
base-dominant.
186
00:10:27,511 --> 00:10:30,227
The cluster sizes shifted slightly.
187
00:10:30,227 --> 00:10:36,437
The structural findings stayed the same.
188
00:10:30,227 --> 00:10:36,437
Seneca's Cluster 2 is a beast.
189
00:10:36,437 --> 00:10:41,787
141 features, the largest cluster in any
corpus we've tested.
190
00:10:41,787 --> 00:10:47,842
And a peak causal delta of 2.
191
00:10:41,787 --> 00:10:47,842
9482.
192
00:10:41,787 --> 00:10:47,842
What does that mean?
193
00:10:47,842 --> 00:10:50,318
We take the cluster's features,
194
00:10:50,318 --> 00:10:52,718
zero them out through the SAE,
195
00:10:52,718 --> 00:10:56,948
and measure how much the model's
next-word predictions shift.
196
00:10:56,948 --> 00:11:01,182
The bigger the number, the more the model
depends on that cluster.
197
00:11:01,182 --> 00:11:04,855
Marcus peaked at about 0.
198
00:11:01,182 --> 00:11:04,855
94.
199
00:11:04,855 --> 00:11:07,972
Ad copy peaked at 0.
200
00:11:04,855 --> 00:11:07,972
65.
201
00:11:07,972 --> 00:11:11,570
Seneca's peak is 2.
202
00:11:07,972 --> 00:11:11,570
95.
203
00:11:11,570 --> 00:11:17,618
The Seneca LoRA concentrates more of its
learned adaptation into a single
204
00:11:17,618 --> 00:11:24,291
dominant cluster than either previous
corpus.
205
00:11:17,618 --> 00:11:24,291
What does Cluster 2 actually
206
00:11:24,291 --> 00:11:25,559
respond to?
207
00:11:25,559 --> 00:11:28,118
Philosophical argumentation.
208
00:11:28,118 --> 00:11:32,664
Its top-firing passages are about the
advice of Attalus,
209
00:11:32,664 --> 00:11:35,852
the poetry of Cleanthes addressing
Jupiter,
210
00:11:35,852 --> 00:11:40,083
and a meditation on humanity's
relationship with nature.
211
00:11:40,083 --> 00:11:46,792
The top causal token is "for," the
connective tissue of reasoned argument.
212
00:11:46,792 --> 00:11:54,282
This cluster encodes how Seneca builds
and connects philosophical claims.
213
00:11:46,792 --> 00:11:54,282
When
214
00:11:54,282 --> 00:11:59,074
we measure how much better the Seneca
LoRA's SAE reconstructs Seneca text
215
00:11:59,074 --> 00:12:01,790
compared to the base model's SAE,
216
00:12:01,790 --> 00:12:04,747
on held-out text the model never trained
on,
217
00:12:04,747 --> 00:12:09,762
we get a median gap of 2.
218
00:12:04,747 --> 00:12:09,762
78 times.
219
00:12:04,747 --> 00:12:09,762
That's real.
220
00:12:09,762 --> 00:12:13,921
The LoRA genuinely learned
Seneca-specific structure.
221
00:12:13,921 --> 00:12:17,514
But it's smaller than Marcus at about
five times,
222
00:12:17,514 --> 00:12:23,752
and much smaller than ad copy at 7.
223
00:12:17,514 --> 00:12:23,752
7 times.
224
00:12:17,514 --> 00:12:23,752
The ordering makes sense.
225
00:12:23,752 --> 00:12:28,469
Seneca's philosophical prose is closer to
the kind of text the base model
226
00:12:28,469 --> 00:12:29,929
already handles well.
227
00:12:29,929 --> 00:12:34,407
Ad copy,
with its extreme stylistic variety,
228
00:12:34,407 --> 00:12:39,720
things like "bestie we need to talk about
your jewelry situation," that's
229
00:12:39,720 --> 00:12:41,918
the farthest from standard English,
230
00:12:41,918 --> 00:12:44,316
so the LoRA has to do the most work.
231
00:12:44,316 --> 00:12:46,509
Marcus sits in the middle.
232
00:12:46,509 --> 00:12:51,304
Seneca, whose prose is arguably the most
conventionally readable of the
233
00:12:51,304 --> 00:12:56,375
three,
requires the least new construction.
234
00:12:51,304 --> 00:12:56,375
This confirms something we
235
00:12:56,375 --> 00:12:57,998
suggested in Article 3.
236
00:12:57,998 --> 00:13:02,363
The cross-reconstruction gap isn't just
measuring whether the LoRA learned
237
00:13:02,363 --> 00:13:02,875
something.
238
00:13:02,875 --> 00:13:06,310
It's measuring how much work the LoRA had
to do.
239
00:13:06,310 --> 00:13:10,623
The distance between the target text and
what the model could already
240
00:13:10,623 --> 00:13:19,172
produce.
241
00:13:10,623 --> 00:13:19,172
The Seneca LoRA created 56 purpose-built
features out of 1, 194
242
00:13:19,172 --> 00:13:20,582
active features.
243
00:13:20,582 --> 00:13:23,220
That's about 4.
244
00:13:20,582 --> 00:13:23,220
7%.
245
00:13:23,220 --> 00:13:26,896
Marcus came in at 5.
246
00:13:23,220 --> 00:13:26,896
2%.
247
00:13:26,896 --> 00:13:28,333
Nearly identical.
248
00:13:28,333 --> 00:13:33,379
Both are single philosophical voices that
are relatively close to what
249
00:13:33,379 --> 00:13:37,162
the model already knows.
250
00:13:33,379 --> 00:13:37,162
Ad copy,
251
00:13:37,162 --> 00:13:41,161
with twelve wildly different styles
crammed into one LoRA,
252
00:13:41,161 --> 00:13:44,596
needed 12.
253
00:13:41,161 --> 00:13:44,596
2% purpose-built features.
254
00:13:44,596 --> 00:13:46,074
The pattern is clear.
255
00:13:46,074 --> 00:13:50,670
The more different the target text is
from what the model already knows,
256
00:13:50,670 --> 00:13:53,143
the more new machinery it has to build.
257
00:13:53,143 --> 00:13:58,453
Philosophical text reuses most of the
existing features and only builds
258
00:13:58,453 --> 00:14:00,327
a small number of new ones.
259
00:14:00,327 --> 00:14:05,369
Commercial copy across twelve registers
needs roughly twice the proportion
260
00:14:05,369 --> 00:14:09,992
of dedicated features.
261
00:14:05,369 --> 00:14:09,992
Every structural finding from Marcus and
ad copy
262
00:14:09,992 --> 00:14:11,270
holds for Seneca.
263
00:14:11,270 --> 00:14:14,543
Two encoding regimes: one tight,
264
00:14:14,543 --> 00:14:15,661
four loose.
265
00:14:15,661 --> 00:14:20,291
Cluster-level causal effects that dwarf
individual features.
266
00:14:20,291 --> 00:14:22,045
An inert cluster.
267
00:14:22,045 --> 00:14:26,507
The cross-reconstruction gap scales with
task difficulty.
268
00:14:26,507 --> 00:14:31,559
Feature specificity does too.
269
00:14:26,507 --> 00:14:31,559
Three corpora,
270
00:14:31,559 --> 00:14:34,273
three very different kinds of text,
271
00:14:34,273 --> 00:14:40,302
and the model organizes its adaptation
the same way every time.
272
00:14:34,273 --> 00:14:40,302
That's
273
00:14:40,302 --> 00:14:42,836
the replication story, and it matters,
274
00:14:42,836 --> 00:14:46,430
because a pattern that shows up once
could be an artifact.
275
00:14:46,430 --> 00:14:52,668
A pattern that shows up three times is
structure.
276
00:14:46,430 --> 00:14:52,668
But replication is only
277
00:14:52,668 --> 00:14:54,873
half of what Seneca gave us.
278
00:14:54,873 --> 00:14:58,300
Now that we know the structural findings
are real,
279
00:14:58,300 --> 00:15:03,220
the next question is: what can a second
philosophical corpus teach us
280
00:15:03,220 --> 00:15:04,827
that a single one couldn't?
281
00:15:04,827 --> 00:15:07,544
It turns out, quite a lot.
282
00:15:07,544 --> 00:15:09,780
We'll pick that up next time.
283
00:15:09,780 --> 00:15:15,694
I'm Arshavir Blackwell and this has been
Inside the Black Box.
