Etsy — Using Post-Treatment Data to Sharpen A/B Tests

WEBVTT
00:00:00.850 --> 00:00:03.950
This is Data Science in the Wild, deep dives
00:00:03.950 --> 00:00:07.250
into how teams like Netflix, Airbnb, and MetaBuild
00:00:07.250 --> 00:00:10.210
machine learning and AI in production. Hosted
00:00:10.210 --> 00:00:14.359
by Peter Liu, produced with AI Voices. Welcome
00:00:14.359 --> 00:00:17.300
to the deep dive. So we have a specific mission
00:00:17.300 --> 00:00:19.280
for you today. A really fun one, actually. Yeah.
00:00:19.559 --> 00:00:21.960
I want you to imagine, just for a moment, that
00:00:21.960 --> 00:00:24.320
you are running Etsy. Okay. So that means you've
00:00:24.320 --> 00:00:27.079
got, what, millions of users? Millions of users.
00:00:27.870 --> 00:00:30.109
billions of interactions and your product team
00:00:30.109 --> 00:00:32.350
comes to you wanting to test a tiny new feature,
00:00:32.469 --> 00:00:35.049
you would think with all that massive traffic,
00:00:35.149 --> 00:00:37.890
getting a clear yes or no on whether that feature
00:00:37.890 --> 00:00:39.829
actually works would be the easiest thing in
00:00:39.829 --> 00:00:41.350
the world, right? I mean, it totally seems like
00:00:41.350 --> 00:00:42.829
it should be. You just have this mountain of
00:00:42.829 --> 00:00:46.399
data. Exactly. But it turns out... It's actually
00:00:46.399 --> 00:00:49.560
a statistical nightmare. So today we are exploring
00:00:49.560 --> 00:00:53.600
a really fascinating new paper from ICLR 2026.
00:00:53.880 --> 00:00:57.200
It's by researchers Zhu Xiaolin from UC Berkeley
00:00:57.200 --> 00:01:01.100
and Pablo Crespo from Etsy. And it's titled Variance
00:01:01.100 --> 00:01:04.420
Reduction, Combining Pre -Experiment and In -Experiment
00:01:04.420 --> 00:01:07.159
Data. And what's really fascinating here is that
00:01:07.159 --> 00:01:10.379
this paper tackles a problem that basically every
00:01:10.379 --> 00:01:14.299
massive tech platform faces, but very few actually
00:01:14.299 --> 00:01:17.569
talk about it publicly. Right. Because A -B testing
00:01:17.569 --> 00:01:20.510
is, well, it's the gold standard for these companies.
00:01:20.629 --> 00:01:22.849
You want to know if changing the color of a buy
00:01:22.849 --> 00:01:25.469
button increases sales, you test it. Yeah, you
00:01:25.469 --> 00:01:28.750
run an A -B test. But getting clear results often
00:01:28.750 --> 00:01:32.189
requires an agonizingly large amount of traffic.
00:01:32.250 --> 00:01:34.909
Like traffic that even giants like Etsy struggle
00:01:34.909 --> 00:01:37.469
to spare for just a single test. It's a huge
00:01:37.469 --> 00:01:40.230
bottleneck. So today we are going to look at
00:01:40.230 --> 00:01:42.689
how Etsy found a way to use what has traditionally
00:01:42.689 --> 00:01:45.590
been considered like... absolute forbidden data.
00:01:45.769 --> 00:01:49.069
The inexperiment data. Right. To squeeze far
00:01:49.069 --> 00:01:51.750
more sensitivity out of their tests without breaking
00:01:51.750 --> 00:01:54.170
the rules of statistics. Yeah. And to really
00:01:54.170 --> 00:01:56.609
appreciate the elegance of what Etsy and UC Berkeley
00:01:56.609 --> 00:01:59.769
did here, we first have to understand the, well,
00:01:59.849 --> 00:02:01.730
the brick wall the industry has been banging
00:02:01.730 --> 00:02:04.109
its head against. Okay. Set the stage for us.
00:02:04.290 --> 00:02:07.489
So in online A -B testing, your ability to detect
00:02:07.489 --> 00:02:10.069
a true improvement, what statisticians would
00:02:10.069 --> 00:02:12.860
call your test sensitivity. is heavily limited
00:02:12.860 --> 00:02:15.560
by your fixed sample size. Because, I mean, you
00:02:15.560 --> 00:02:17.599
only have so many users visiting your site on
00:02:17.599 --> 00:02:20.360
a given day, you can't just magically conjure
00:02:20.360 --> 00:02:22.419
up a million new users to make your math look
00:02:22.419 --> 00:02:25.199
better. Precisely. So if you want to detect smaller,
00:02:25.360 --> 00:02:27.340
you know, more subtle improvements to your platform,
00:02:27.639 --> 00:02:30.379
the brute force mathematical answer is just that
00:02:30.379 --> 00:02:32.639
you need more traffic. Right. But if you're a
00:02:32.639 --> 00:02:34.759
platform that's already running thousands of
00:02:34.759 --> 00:02:36.960
simultaneous tests on your entire user base,
00:02:37.219 --> 00:02:40.280
finding unallocated traffic is completely impractical.
00:02:40.400 --> 00:02:43.110
You are totally tapped out. So the standard solution
00:02:43.110 --> 00:02:45.050
to this traffic wall has been something called
00:02:45.050 --> 00:02:47.530
regression adjustment, or more specifically,
00:02:47.610 --> 00:02:50.389
these techniques called Cupid or its machine
00:02:50.389 --> 00:02:53.550
learning powered cousin, QPAC. Yes. And for those
00:02:53.550 --> 00:02:55.830
who might not be deep in the statistical weeds
00:02:55.830 --> 00:02:59.069
every day, QPAT and QPAC are incredibly clever
00:02:59.069 --> 00:03:01.550
techniques. They really are. They use pre -experiment
00:03:01.550 --> 00:03:03.990
data, so things we already know about a user
00:03:03.990 --> 00:03:07.050
before the test even starts, to shrink the variance
00:03:07.050 --> 00:03:10.370
or the noise of the test results. And crucially,
00:03:10.509 --> 00:03:12.689
they do this without adding bias to the final
00:03:12.689 --> 00:03:16.210
answer. Exactly. QPAC specifically uses a machine
00:03:16.210 --> 00:03:18.990
learning prediction step to figure out what a
00:03:18.990 --> 00:03:21.050
user is likely to do, which makes the adjustment
00:03:21.050 --> 00:03:23.569
even more effective. It's basically the system
00:03:23.569 --> 00:03:26.610
saying, hey, I know exactly how this specific
00:03:26.610 --> 00:03:29.469
user behaved last month, so I can predict their
00:03:29.469 --> 00:03:31.610
baseline behavior and subtract that expected
00:03:31.610 --> 00:03:34.150
noise from our current test results. Right. It
00:03:34.150 --> 00:03:37.050
narrows the range of uncertainty. But if we connect
00:03:37.050 --> 00:03:39.969
this to the bigger picture... there is a massive
00:03:39.969 --> 00:03:43.189
claring catch. Oh, yeah. This entire approach
00:03:43.189 --> 00:03:45.550
relies totally on the assumption that past data
00:03:45.550 --> 00:03:48.069
is a good predictor of future behavior. Which,
00:03:48.150 --> 00:03:50.710
let's be honest, it isn't always. Not at all.
00:03:50.830 --> 00:03:52.889
I mean, it fails completely for brand new users
00:03:52.889 --> 00:03:56.030
who simply have no history on the platform. The
00:03:56.030 --> 00:03:57.710
model looks at them and just sees a blank slate.
00:03:57.909 --> 00:04:00.590
Yep. And furthermore, it fails for outcomes that
00:04:00.590 --> 00:04:02.889
are heavily dependent on what the user is doing
00:04:02.889 --> 00:04:06.210
in their current immediate session. Right. Relying
00:04:06.210 --> 00:04:09.080
only on pre -experiment data is... Well, it's
00:04:09.080 --> 00:04:10.900
like trying to predict if a customer will buy
00:04:10.900 --> 00:04:12.780
an umbrella based solely on their shoe purchases
00:04:12.780 --> 00:04:15.860
last month. That's a great analogy. While completely
00:04:15.860 --> 00:04:17.879
ignoring the fact that they are standing in a
00:04:17.879 --> 00:04:20.660
pouring rainstorm right now. The things happening
00:04:20.660 --> 00:04:23.259
right now, what we call in -experiment or post
00:04:23.259 --> 00:04:25.759
-treatment variables, are incredibly predictive
00:04:25.759 --> 00:04:28.720
of what they'll do next. Highly predictive. But
00:04:28.720 --> 00:04:31.879
industry practice usually outright bans data
00:04:31.879 --> 00:04:33.800
science teams from using them. The forbidden
00:04:33.800 --> 00:04:36.519
fruit of A -B testing. So why is it forbidden?
00:04:37.160 --> 00:04:40.220
The fear is post -treatment bias. If the treatment
00:04:40.220 --> 00:04:43.459
you're testing, say a new search algorithm, actually
00:04:43.459 --> 00:04:46.920
changes the variable you are adjusting for, adjusting
00:04:46.920 --> 00:04:49.180
for it ruins the mathematical validity of the
00:04:49.180 --> 00:04:52.339
test. Oh, wow. Yeah, you end up masking the very
00:04:52.339 --> 00:04:55.019
effect you are trying to measure. You're accidentally
00:04:55.019 --> 00:04:58.490
baking the outcome into your baseline. Okay,
00:04:58.569 --> 00:05:01.110
let's unpack this. If post -treatment data is
00:05:01.110 --> 00:05:04.360
so incredibly... dangerous. How did Etsy and
00:05:04.360 --> 00:05:06.860
UC Berkeley figure out a way to safely use it
00:05:06.860 --> 00:05:08.959
without ruining their tests? Well, to understand
00:05:08.959 --> 00:05:10.920
how they did it safely, we have to look at their
00:05:10.920 --> 00:05:14.439
exact statistical setup. They use a potential
00:05:14.439 --> 00:05:16.800
outcomes framework with Bernoulli assignment.
00:05:17.079 --> 00:05:19.420
Which is a fancy way of saying standard randomized
00:05:19.420 --> 00:05:22.699
A -B testing. Exactly. But the brilliance is
00:05:22.699 --> 00:05:25.399
in how they layered their approach into two distinct
00:05:25.399 --> 00:05:28.629
stages. Right. Stage one is their baseline. And
00:05:28.629 --> 00:05:30.730
it remains entirely unchanged. They don't throw
00:05:30.730 --> 00:05:33.149
away their existing engine at all. No, not at
00:05:33.149 --> 00:05:35.949
all. At Etsy, their stage one is that standard
00:05:35.949 --> 00:05:38.689
QPAC model we talked about. It's a single light
00:05:38.689 --> 00:05:41.750
GBM machine learning model trained on massive
00:05:41.750 --> 00:05:45.629
amounts of pooled data using 117 pre -treatment
00:05:45.629 --> 00:05:47.970
covariates. And that runs exactly as it always
00:05:47.970 --> 00:05:50.910
has. It provides the heavy lifting for the initial
00:05:50.910 --> 00:05:54.050
variance reduction. But then comes stage two.
00:05:54.230 --> 00:05:57.189
This is where the real innovation happens. They
00:05:57.189 --> 00:05:59.750
add a second stage linear adjustment, strictly
00:05:59.750 --> 00:06:02.829
on the residuals of that first QPAC model. Hold
00:06:02.829 --> 00:06:04.829
on. For anyone listening who might be a little
00:06:04.829 --> 00:06:07.449
rusty on their modeling terms, let's explain
00:06:07.449 --> 00:06:10.089
how adjusting residuals actually works. Sure.
00:06:11.000 --> 00:06:13.000
Think of stage one like predicting a runner's
00:06:13.000 --> 00:06:15.259
marathon time based on their past training history.
00:06:15.439 --> 00:06:18.439
The model says, based on your last year of running,
00:06:18.579 --> 00:06:20.980
you should finish in exactly four hours. The
00:06:20.980 --> 00:06:23.379
residual is the gap between that prediction and
00:06:23.379 --> 00:06:26.529
their actual race time. Right. So if they finish
00:06:26.529 --> 00:06:28.930
in four hours and 10 minutes, the residual is
00:06:28.930 --> 00:06:32.189
10 minutes. That is a perfect analogy. Stage
00:06:32.189 --> 00:06:34.889
one made its best guess. And the residual is
00:06:34.889 --> 00:06:36.970
the leftover noise it just couldn't explain.
00:06:37.269 --> 00:06:40.050
Exactly. So stage two looks at that 10 minute
00:06:40.050 --> 00:06:43.050
gap and says, oh, wait, it started pouring rain
00:06:43.050 --> 00:06:46.470
during the race. By factoring in that in -experiment
00:06:46.470 --> 00:06:49.569
rain, which the paper calls the post -treatment
00:06:49.569 --> 00:06:52.730
covariates or Z, we can explain away that remaining
00:06:52.730 --> 00:06:55.819
noise. Precisely. By running a linear adjustment
00:06:55.819 --> 00:06:59.220
on those residuals using those Z variables, they
00:06:59.220 --> 00:07:02.120
are squeezing out even more variants. But this
00:07:02.120 --> 00:07:04.019
brings us back to the core fear we discussed
00:07:04.019 --> 00:07:06.839
earlier. The forbidden fruit part. Right. How
00:07:06.839 --> 00:07:08.939
do we know these inexperiment variables, these
00:07:08.939 --> 00:07:12.459
Zs, aren't tainted by the treatment itself? If
00:07:12.459 --> 00:07:14.439
the new feature caused the rain, so to speak,
00:07:14.600 --> 00:07:17.519
the math totally breaks. And this is where the
00:07:17.519 --> 00:07:19.959
paper introduces a concept called mean equivalence.
00:07:20.100 --> 00:07:22.519
Yes. Mean equivalence is a specific mathematical
00:07:22.519 --> 00:07:25.379
condition. It means that the expected value of
00:07:25.379 --> 00:07:27.699
the variable given the user is in the treatment
00:07:27.699 --> 00:07:30.519
group equals the expected value of the variable
00:07:30.519 --> 00:07:33.199
given the user is in the control group. I'm going
00:07:33.199 --> 00:07:35.740
to push back here a little because mean equivalence
00:07:35.740 --> 00:07:37.980
sounds like a very polite way of saying it just
00:07:37.980 --> 00:07:40.660
bounces out on average. In a massive tech platform,
00:07:40.899 --> 00:07:43.000
isn't that a huge risk? How can they guarantee
00:07:43.000 --> 00:07:45.259
a user's behavior wasn't somehow tweaked by the
00:07:45.259 --> 00:07:47.680
test? It is a weaker condition than demanding
00:07:47.680 --> 00:07:49.939
full, strict statistical independence between
00:07:49.939 --> 00:07:52.300
the variable and the treatment. You are right.
00:07:52.360 --> 00:07:54.639
It just means the treatment didn't move the needle
00:07:54.639 --> 00:07:57.040
on that variable on average across the population.
00:07:57.240 --> 00:07:59.980
Right. But the most crucial thing about mean
00:07:59.980 --> 00:08:02.500
equivalence, the reason this entire paper works,
00:08:02.680 --> 00:08:06.600
is that unlike strict independence, mean equivalence
00:08:06.600 --> 00:08:09.699
is testable. OK, so you can actually run a diagnostic
00:08:09.699 --> 00:08:13.300
on it. What did Etsy actually run to test this?
00:08:13.360 --> 00:08:16.360
Because they applied this to 29 real world experiments.
00:08:16.860 --> 00:08:19.300
They were incredibly deliberate. They didn't
00:08:19.300 --> 00:08:21.279
just throw every piece of data at the wall. First,
00:08:21.360 --> 00:08:23.600
they looked at candidate variables. The paper
00:08:23.600 --> 00:08:26.420
notes these were specifically sparse count variables
00:08:26.420 --> 00:08:29.199
with many zeros. They don't name exactly what
00:08:29.199 --> 00:08:31.560
those are, but we're talking about highly specific
00:08:31.560 --> 00:08:34.620
infrequent user actions during a session, things
00:08:34.620 --> 00:08:37.860
that most users do zero times. Right. So first,
00:08:37.940 --> 00:08:39.460
they dropped the variables that were just too
00:08:39.460 --> 00:08:42.200
sparse to be statistically informative. There
00:08:42.200 --> 00:08:44.080
just wasn't enough signal there. Makes sense.
00:08:44.440 --> 00:08:47.139
Then, for the remaining candidates, they ran
00:08:47.139 --> 00:08:50.370
a Mann -Whitney U -test per experiment. Let's
00:08:50.370 --> 00:08:53.090
translate that real quick. A Mann -Whitney U
00:08:53.090 --> 00:08:54.850
-test is basically a way to see if one group
00:08:54.850 --> 00:08:57.149
tends to have higher values than another without
00:08:57.149 --> 00:08:59.529
assuming the data follows a perfect bell curve,
00:08:59.610 --> 00:09:02.509
right? Exactly. It's a very robust way to check
00:09:02.509 --> 00:09:05.169
for differences. So they ran that test for every
00:09:05.169 --> 00:09:07.350
candidate variable across all 29 experiments.
00:09:08.070 --> 00:09:10.830
Then, to get a holistic view rather than just
00:09:10.830 --> 00:09:13.610
looking at isolated tests, they combined the
00:09:13.610 --> 00:09:16.289
p -values from those tests across all 29 experiments
00:09:16.289 --> 00:09:19.190
using Fisher's method. Fisher's method is b***h.
00:09:19.159 --> 00:09:21.940
Brilliant here. If a variable is slightly tainted
00:09:21.940 --> 00:09:24.200
in one experiment, it might not trigger an alarm.
00:09:24.320 --> 00:09:27.179
But Fisher's method essentially listens to all
00:09:27.179 --> 00:09:30.639
29 experiments at once. Yes. So if it hears a
00:09:30.639 --> 00:09:33.919
faint whisper of bias in several tests, it combines
00:09:33.919 --> 00:09:36.399
them into a loud shout, flagging the variable
00:09:36.399 --> 00:09:38.879
as unsafe. That's a perfect way to visualize
00:09:38.879 --> 00:09:41.340
it. And after putting their candidate variables
00:09:41.340 --> 00:09:44.240
through that rigorous gauntlet, they were left
00:09:44.240 --> 00:09:46.840
with 23 post -treatment covariates that actually
00:09:46.840 --> 00:09:50.509
passed the test. Wow. Only 23. Yeah. Now, there
00:09:50.509 --> 00:09:52.850
is a very important technical detail we need
00:09:52.850 --> 00:09:55.009
to clarify about how they handled multiple testing.
00:09:55.250 --> 00:09:58.730
Ah, the Family Wise Error Rate? Yes. Or FWER?
00:09:59.029 --> 00:10:02.250
Yes. For those who know their statistics, when
00:10:02.250 --> 00:10:05.009
you run dozens of tests looking for a significant
00:10:05.009 --> 00:10:08.750
result, by pure random chance, some will look
00:10:08.750 --> 00:10:10.950
significant even when they aren't. The classic
00:10:10.950 --> 00:10:14.389
multiple comparisons problem. Exactly. To fix
00:10:14.389 --> 00:10:18.870
this, Statisticians usually use formal FWER corrections,
00:10:19.090 --> 00:10:22.450
like the Holm or Bonferroni methods, to make
00:10:22.450 --> 00:10:25.330
the tests stricter. But the paper explicitly
00:10:25.330 --> 00:10:27.950
states that in this empirical study at ETSI,
00:10:27.970 --> 00:10:31.409
they did not use a formal FWER correction? Correct.
00:10:31.750 --> 00:10:34.549
They didn't. The authors note that for larger
00:10:34.549 --> 00:10:36.590
libraries of candidate variables, those strict
00:10:36.590 --> 00:10:39.929
corrections can and should be added. They also
00:10:39.929 --> 00:10:43.230
mention TOST equivalence tests as an alternative
00:10:43.230 --> 00:10:46.269
for situations with impossibly huge samples,
00:10:46.409 --> 00:10:49.370
where even trivial, meaningless differences might
00:10:49.370 --> 00:10:52.210
flag as statistically significant. But we have
00:10:52.210 --> 00:10:54.009
to be really clear here for our listeners. Yes.
00:10:54.090 --> 00:10:56.769
For the specific Etsy pipeline that yielded these
00:10:56.769 --> 00:11:00.169
23 variables, they did not use TOST and they
00:11:00.169 --> 00:11:03.750
did not use Holm or Bonferroni. They relied purely
00:11:03.750 --> 00:11:06.190
on their specific workflow with Fisher's method.
00:11:06.429 --> 00:11:09.490
Exactly. OK, so if we have these statistical
00:11:09.490 --> 00:11:12.049
tests, the Mann -Whitney -Yu and Fisher's method,
00:11:12.230 --> 00:11:14.309
does that mean a data scientist listening to
00:11:14.309 --> 00:11:16.730
this can just feed all their platform's data
00:11:16.730 --> 00:11:19.350
into a Python script, let it automatically spit
00:11:19.350 --> 00:11:21.870
out a list of definitively safe covariates and
00:11:21.870 --> 00:11:25.190
go to lunch? No, absolutely not. I need to offer
00:11:25.190 --> 00:11:27.330
a very firm correction on that idea because it's
00:11:27.330 --> 00:11:30.289
a dangerous trap. This statistical screening
00:11:30.289 --> 00:11:33.649
is not a mathematical proof. OK. This process
00:11:33.649 --> 00:11:36.350
is not fully automatic. So the script doesn't
00:11:36.350 --> 00:11:39.009
just hand you a golden proven list of safe variables?
00:11:39.289 --> 00:11:42.049
Not at all. The paper is very explicit about
00:11:42.049 --> 00:11:44.909
this. A non -significant balanced test, meaning
00:11:44.909 --> 00:11:47.330
your Mann -Whitney U -test didn't raise a red
00:11:47.330 --> 00:11:50.289
flag, must be interpreted purely as a diagnostic.
00:11:50.730 --> 00:11:53.269
It does not prove a covariate is permanently
00:11:53.269 --> 00:11:56.210
safe. Because a large p -value might just mean
00:11:56.210 --> 00:11:59.149
your test has low power, it might just be failing
00:11:59.149 --> 00:12:01.669
to detect a bias that is actually sitting right
00:12:01.669 --> 00:12:03.950
there in the data. Precisely. The screening...
00:12:04.139 --> 00:12:06.679
is therefore not intended to be the sole safeguard.
00:12:07.100 --> 00:12:09.620
At Etsy, the purpose of this statistical screen
00:12:09.620 --> 00:12:12.519
was to rule out clearly treatment -affected variables.
00:12:12.820 --> 00:12:15.399
It was not to prove treatment insensitivity from
00:12:15.399 --> 00:12:18.940
p -values alone. Got it. The absolute recommended
00:12:18.940 --> 00:12:22.019
workflow here starts with human beings. Domain
00:12:22.019 --> 00:12:25.240
knowledge. Yes. You must use human domain knowledge
00:12:25.240 --> 00:12:28.139
to actively exclude obvious mediators before
00:12:28.139 --> 00:12:31.220
you even run the math. If you know a variable
00:12:31.220 --> 00:12:33.139
is logically going to be affected by the treatment,
00:12:33.450 --> 00:12:35.230
You pull it out yourself. You don't wait for
00:12:35.230 --> 00:12:37.470
the script to catch it. Because if you skip that
00:12:37.470 --> 00:12:39.710
step, like if you get lazy and a tainted variable
00:12:39.710 --> 00:12:42.289
sneaks through, the paper actually has a formula
00:12:42.289 --> 00:12:45.370
for the bias it introduces, right? It does. Let's
00:12:45.370 --> 00:12:47.250
translate it out of the Greek letters, though.
00:12:47.289 --> 00:12:49.929
Basically, the math actively punishes you. It
00:12:49.929 --> 00:12:52.509
absolutely punishes you. If you fail to exclude
00:12:52.509 --> 00:12:55.429
a mediator, and mean equivalence fails by an
00:12:55.429 --> 00:12:58.710
amount we'll call delta, the resulting bias introduced
00:12:58.710 --> 00:13:01.870
into your test is negative gamma transpose delta.
00:13:02.210 --> 00:13:05.529
Wow. In plain English, that error gets multiplied
00:13:05.529 --> 00:13:08.169
by how much your model relies on that variable,
00:13:08.330 --> 00:13:11.049
and it is directly subtracted from your final
00:13:11.049 --> 00:13:13.269
result. You end up blinding yourself to your
00:13:13.269 --> 00:13:15.549
own success. Like, if your treatment actually
00:13:15.549 --> 00:13:18.629
improved user behavior, but you accidentally
00:13:18.629 --> 00:13:21.350
adjusted for that exact behavior as a covariate,
00:13:21.490 --> 00:13:24.649
you mathematically erase your own win. You do.
00:13:25.049 --> 00:13:27.429
Which really highlights how human intuition remains
00:13:27.429 --> 00:13:30.509
entirely irreplaceable, even in highly automated
00:13:30.509 --> 00:13:34.080
ML pipelines. Exactly. The script is a diagnostic
00:13:34.080 --> 00:13:38.019
tool. It is not a replacement for an expert who
00:13:38.019 --> 00:13:40.539
actually understands the product. Okay, so keeping
00:13:40.539 --> 00:13:42.779
in mind the strict discipline required to select
00:13:42.779 --> 00:13:45.399
these 23 variables, here's where it gets really
00:13:45.399 --> 00:13:47.940
interesting. What happened when Etsy actually
00:13:47.940 --> 00:13:50.139
plugged them into their live experiments? The
00:13:50.139 --> 00:13:53.220
fun part. Yeah. Does layering these 23 survivors
00:13:53.220 --> 00:13:55.340
on top of their massive machine learning model
00:13:55.340 --> 00:13:57.700
actually move the needle on Etsy's bottom line?
00:13:57.879 --> 00:14:00.919
The realized gains on those 29 Etsy experiments
00:14:00.919 --> 00:14:04.620
were substantial. And to be specific, the primary
00:14:04.620 --> 00:14:06.799
metric they were tracking across these tests
00:14:06.799 --> 00:14:09.659
was conversion rate. Always the holy grail for
00:14:09.659 --> 00:14:12.320
e -commerce. Always. So let's look at the predictive
00:14:12.320 --> 00:14:14.879
accuracy. We are talking about the difference
00:14:14.879 --> 00:14:17.340
in the square root of R squared over the baseline
00:14:17.340 --> 00:14:21.200
QPAC model. Yes. That gain in predictive accuracy
00:14:21.200 --> 00:14:27.200
ranged from 0 .02 to above 0 .14. And to be incredibly
00:14:27.200 --> 00:14:30.460
clear here, this is from adding just 23 carefully
00:14:30.460 --> 00:14:33.960
screened in experiment variables on top of the
00:14:33.960 --> 00:14:37.100
unchanged QPAC model. They didn't replace the
00:14:37.100 --> 00:14:40.139
117 pre -experiment variables. They just layered
00:14:40.139 --> 00:14:43.399
these 23 on as that second stage. Correct. But
00:14:43.399 --> 00:14:46.039
the key comparison, the metric that really proves
00:14:46.039 --> 00:14:49.120
the value of this paper, is the variance reduction.
00:14:49.460 --> 00:14:51.639
The extra variance reduction from this added
00:14:51.639 --> 00:14:54.799
stage two is comparable or greater than QPAC's
00:14:54.799 --> 00:14:57.210
own reduction. Let me make sure I'm getting this
00:14:57.210 --> 00:15:00.330
because this is wild. Stage one, using a complex
00:15:00.330 --> 00:15:03.409
machine learning model and 117 pre -experiment
00:15:03.409 --> 00:15:07.690
variables, gets them roughly a 10 % to 25 % variance
00:15:07.690 --> 00:15:09.929
reduction. That's the industry standard win.
00:15:10.330 --> 00:15:14.009
Right. But stage two, using only 23 post -treatment
00:15:14.009 --> 00:15:17.970
variables, layers on roughly an extra 2 .5 %
00:15:17.970 --> 00:15:21.049
to 14 % reduction per experiment. On top of the
00:15:21.049 --> 00:15:23.909
original reduction. That is a massive efficiency
00:15:23.909 --> 00:15:27.500
gain. They are getting a huge boost in test sensitivity
00:15:27.500 --> 00:15:30.700
using a fraction of the covariates, just by looking
00:15:30.700 --> 00:15:33.500
at what the user is doing right now. And this
00:15:33.500 --> 00:15:35.539
is why this matters so much for the broader industry.
00:15:35.980 --> 00:15:38.940
Because these in -experiment covariates are generated
00:15:38.940 --> 00:15:41.379
right there in the current session, they are
00:15:41.379 --> 00:15:43.759
available immediately, even for brand new users.
00:15:44.120 --> 00:15:47.279
Oh, right. So this approach perfectly fits massive
00:15:47.279 --> 00:15:50.080
platforms running thousands of parallel experiments
00:15:50.080 --> 00:15:53.629
where historical data is inevitably spotty. It
00:15:53.629 --> 00:15:55.970
effectively solves the new user traffic wall.
00:15:56.230 --> 00:15:59.230
It's brilliant. But, I mean, I have to assume
00:15:59.230 --> 00:16:01.210
there is an ongoing risk here. You can't just
00:16:01.210 --> 00:16:03.850
set this and forget it, right? No, there is a
00:16:03.850 --> 00:16:06.350
critical caveat. The validity of this entire
00:16:06.350 --> 00:16:09.190
system rests on mean equivalence holding true
00:16:09.190 --> 00:16:12.710
over time. Teams must run ongoing diagnostics.
00:16:12.789 --> 00:16:15.610
Why is that? Because a future treatment in a
00:16:15.610 --> 00:16:17.730
totally different experiment next year might
00:16:17.730 --> 00:16:20.629
quietly shift one of those 23 screened covariates.
00:16:21.110 --> 00:16:23.409
If a new feature changes user behavior in a way
00:16:23.409 --> 00:16:25.850
that breaks mean equivalence, your math breaks
00:16:25.850 --> 00:16:28.169
with it. You suddenly have a hidden mediator.
00:16:28.529 --> 00:16:31.750
So what does this all mean for the field? Because
00:16:31.750 --> 00:16:33.929
let's be honest, adjusting for covariates in
00:16:33.929 --> 00:16:36.409
a regression is a pretty old statistical trick.
00:16:36.649 --> 00:16:40.110
How genuinely new is this paper? It's an important
00:16:40.110 --> 00:16:42.929
distinction to make. The true advance here isn't
00:16:42.929 --> 00:16:46.090
the regression itself. The advance is the discipline.
00:16:46.590 --> 00:16:49.950
The discipline of creating a practical, testable...
00:16:50.330 --> 00:16:53.169
platform -level screen. Exactly. It creates a
00:16:53.169 --> 00:16:55.769
systematic way to separate safe variables from
00:16:55.769 --> 00:16:59.049
mediators at scale. In the past, the field relied
00:16:59.049 --> 00:17:02.129
on older, largely untestable assumptions, things
00:17:02.129 --> 00:17:04.990
like surrogacy or principal ignorability, to
00:17:04.990 --> 00:17:07.490
try and justify using post -treatment data. You
00:17:07.490 --> 00:17:09.529
essentially had to cross your fingers and hope
00:17:09.529 --> 00:17:12.250
your assumptions were right. Pretty much. But
00:17:12.250 --> 00:17:14.490
this paper replaces those untestable assumptions
00:17:14.490 --> 00:17:17.559
with hard, testable diagnostics. It's giving
00:17:17.559 --> 00:17:20.079
data scientists an actual map through the muddy
00:17:20.079 --> 00:17:22.740
waters. It is. And to give you a holistic view,
00:17:22.900 --> 00:17:25.299
it helps to map this against other recent industry
00:17:25.299 --> 00:17:27.279
papers that are tackling A -B testing limitations.
00:17:27.779 --> 00:17:30.819
For instance, there's a 2026 paper from ByteDance
00:17:30.819 --> 00:17:34.180
titled Ensuring Trustworthy Online A -B Testing,
00:17:34.339 --> 00:17:37.279
Addressing Five Key Questions on Cupid. Right.
00:17:37.339 --> 00:17:39.400
I've seen that one, but they were tackling very
00:17:39.400 --> 00:17:41.440
different aspects of the problem, right? Correct.
00:17:41.900 --> 00:17:43.819
The ByteDance paper is primarily about choosing
00:17:43.819 --> 00:17:46.440
the correct post -Cupid estimators, ensuring
00:17:46.440 --> 00:17:49.200
regression, adjustment validity, robust variance
00:17:49.200 --> 00:17:52.019
estimation, and multi -arm or two -stage sampling
00:17:52.019 --> 00:17:54.400
designs. So they were focused on the back -end
00:17:54.400 --> 00:17:56.799
statistical design and sampling, whereas Etsy
00:17:56.799 --> 00:17:59.339
is tackling the actual in -session data you are
00:17:59.339 --> 00:18:01.900
allowed to feed into the test. Precisely. The
00:18:01.900 --> 00:18:03.960
ByteDance paper represents the rigorous baseline
00:18:03.960 --> 00:18:06.579
that this Etsy paper is building upon and extending.
00:18:07.269 --> 00:18:09.430
And there's also fascinating work from Instacart
00:18:09.430 --> 00:18:11.769
this year, exploring variance reduction below
00:18:11.769 --> 00:18:14.109
the randomization grain, which shows another
00:18:14.109 --> 00:18:16.630
totally different industry route to more sensitive
00:18:16.630 --> 00:18:19.769
tests. Right. We've also seen papers on Cupid
00:18:19.769 --> 00:18:22.230
on steroids, which looks at covariate adjustment
00:18:22.230 --> 00:18:24.970
in entirely different design settings like switchback
00:18:24.970 --> 00:18:27.910
experiments. The entire industry is aggressively
00:18:27.910 --> 00:18:30.509
attacking this sensitivity problem from multiple
00:18:30.509 --> 00:18:35.289
angles because data is money. But this Etsy and
00:18:35.289 --> 00:18:37.609
UC Berkeley paper really stands out for figuring
00:18:37.609 --> 00:18:40.490
out how to safely eat the forbidden fruit of
00:18:40.490 --> 00:18:44.109
in -experiment data. By relying on testable diagnostics
00:18:44.109 --> 00:18:46.970
rather than blind assumptions. Okay, let's quickly
00:18:46.970 --> 00:18:50.420
recap our journey today for you. Imagine running
00:18:50.420 --> 00:18:53.380
Etsy again. You hit a wall. Your pre -experiment
00:18:53.380 --> 00:18:55.259
data just isn't enough to get the sensitivity
00:18:55.259 --> 00:18:57.680
you need without demanding impossible amounts
00:18:57.680 --> 00:19:00.299
of traffic. Right. So you look at the forbidden
00:19:00.299 --> 00:19:03.099
post -treatment data. You carefully screen it
00:19:03.099 --> 00:19:05.920
using mean equivalence, specifically running
00:19:05.920 --> 00:19:08.519
your sparse count variables through Mann -Whitney
00:19:08.519 --> 00:19:10.539
U -tests and combining them with Fisher's method.
00:19:10.799 --> 00:19:13.839
You take the 23 survivors and layer them as a
00:19:13.839 --> 00:19:16.240
linear adjustment on top of your existing CUPEC
00:19:16.240 --> 00:19:19.859
model. And the result? Massive compounded variance
00:19:19.859 --> 00:19:22.680
reductions that finally let you test features
00:19:22.680 --> 00:19:25.920
on new users effectively. All while proving that
00:19:25.920 --> 00:19:29.250
statistical tests are just diagnostics. And deep
00:19:29.250 --> 00:19:31.829
human domain knowledge is still absolutely required
00:19:31.829 --> 00:19:35.009
to filter out the noise. It is a brilliant synthesis
00:19:35.009 --> 00:19:37.269
of machine learning power and human statistical
00:19:37.269 --> 00:19:39.470
discipline. Which brings us to a final thought
00:19:39.470 --> 00:19:41.990
I want to leave you with. We all crave clean,
00:19:42.269 --> 00:19:46.109
binary, easy answers in our data. But as machine
00:19:46.109 --> 00:19:48.529
learning models and A -B testing get better and
00:19:48.529 --> 00:19:50.589
better at predicting our behavior during a session,
00:19:50.750 --> 00:19:54.390
the line between What a user naturally does and
00:19:54.390 --> 00:19:57.829
what a treatment forced them to do is only going
00:19:57.829 --> 00:20:00.109
to get blurrier. The statistical waters will
00:20:00.109 --> 00:20:03.789
get muddier. Exactly. So ponder this. In a future
00:20:03.789 --> 00:20:06.289
where platforms run millions of automated tests,
00:20:06.470 --> 00:20:09.349
constantly optimizing every pixel of your experience,
00:20:09.690 --> 00:20:12.049
how will companies ensure they haven't accidentally
00:20:12.049 --> 00:20:14.750
hard -coded hidden mediators into their safe
00:20:14.750 --> 00:20:17.349
diagnostic pipelines? When the machines are running
00:20:17.349 --> 00:20:19.650
the tests, who is double -checking the math?
00:20:20.170 --> 00:20:22.130
A very important question for the industry to
00:20:22.130 --> 00:20:24.569
answer. Thanks for joining us on this deep dive.
00:20:24.750 --> 00:20:26.109
Keep questioning the data.