Episode 101: The COLDCARD Hack

WEBVTT - Automatic transcript; may contain recognition errors. Speaker labels from separate source tracks. Times match the published episode.
00:00:07.800 --> 00:00:10.760
Live from the internet, this is Bitcoin.
00:00:12.342 --> 00:00:12.902
Explained.
00:00:13.780 --> 00:00:14.140
Hey Sjors.
00:00:16.730 --> 00:00:21.790
Yeah, we're doing another remote episode because, well, we promised to make one
00:00:21.790 --> 00:00:27.520
in our last emergency broadcast. So we're going to discuss the cold
00:00:27.520 --> 00:00:33.220
card hack as many other podcasts have done before us and many articles and,
00:00:33.220 --> 00:00:35.220
you know, everyone has shared their opinion.
00:00:36.080 --> 00:00:41.560
We thought we should still make this episode and I can pick your brain on this a
00:00:41.560 --> 00:00:43.600
little bit. It's sort of your expertise.
00:00:44.960 --> 00:00:49.960
Needless to say, COLDCARD is not our sponsor, at least not for this episode. We
00:00:49.960 --> 00:00:52.620
haven't really discussed it. I don't think we're going back.
00:00:52.392 --> 00:00:56.492
No, they stopped sponsoring podcasts in general. That's my understanding.
00:00:54.890 --> 00:00:56.370
Yeah, okay, right.
00:00:58.200 --> 00:01:00.140
Yeah, luckily we have a new sponsor,
00:01:01.102 --> 00:01:03.592
That's right. That's right.
00:01:01.500 --> 00:01:07.160
Mt. Gox. Mt. Gox, check it out. It's a brand new, brilliant Bitcoin exchange.
00:01:09.130 --> 00:01:12.890
Perfectly fine to store your Bitcoin. I hear they have Litecoin coming in two
00:01:12.890 --> 00:01:13.590
weeks from now.
00:01:14.582 --> 00:01:16.242
Oh, nice. I should try that.
00:01:17.740 --> 00:01:22.160
Yeah, jokes aside, that's actually something I also want to discuss. We'll do
00:01:22.160 --> 00:01:25.660
that a bit later in the podcast because obviously, so I want to discuss the cold
00:01:25.660 --> 00:01:31.560
card hack itself and then also... Yeah, kind of share my thoughts,
00:01:31.680 --> 00:01:36.440
your thoughts on them being our sponsor, because that kind of became part of the
00:01:36.440 --> 00:01:40.320
whole story that, you know, COLDCARD, Coinkite was sponsoring like every podcast
00:01:40.320 --> 00:01:40.900
out there.
00:01:41.100 --> 00:01:46.920
And to what extent, you know, are we to blame for all of this? Where do our
00:01:46.920 --> 00:01:49.020
own responsibilities begin and end?
00:01:51.450 --> 00:01:57.210
It sucks what happened. It's painful. But let's first discuss
00:01:57.210 --> 00:01:59.430
the COLDCARD hack itself.
00:02:00.882 --> 00:02:01.442
Let's do that.
00:02:03.800 --> 00:02:07.740
By now, probably everyone knows at least roughly what happened.
00:02:10.040 --> 00:02:15.680
But still, Sjors, what went wrong here? Obviously,
00:02:16.140 --> 00:02:21.140
everyone knows about the news. The cold card turned out to be vulnerable in
00:02:21.140 --> 00:02:26.460
basically the worst way possible. People that, as they say, did everything right.
00:02:26.460 --> 00:02:31.140
They generated their seed on a hardware wallet, never used it online even.
00:02:31.640 --> 00:02:34.460
But still, Bitcoins were being drowned from it.
00:02:35.632 --> 00:02:37.012
Yeah, so...
00:02:35.720 --> 00:02:40.380
Just in short, not to dwell on it too much because everyone knows this, but what
00:02:40.380 --> 00:02:40.700
happened?
00:02:41.282 --> 00:02:45.802
No, I guess there are sort of two bad failures for a hardware wallet. One is if
00:02:45.802 --> 00:02:49.762
somebody can steal the coins from your hardware wallet if they have physical
00:02:49.762 --> 00:02:54.922
access to it or through some trick, like with, you know, we've talked about
00:02:54.922 --> 00:02:59.082
earlier with malicious PSBTs or tricks with fees, like malware on your computer,
00:02:59.182 --> 00:02:59.502
basically.
00:02:59.982 --> 00:03:04.302
And the other that is actually much worse, and that is what happened, is if the
00:03:04.302 --> 00:03:08.812
seed was never secure in the first place. And that's basically what happened,
00:03:08.952 --> 00:03:10.812
right? The seed had too little entropy.
00:03:12.442 --> 00:03:18.122
So basically the number of possible seeds in the world was just a couple of
00:03:13.040 --> 00:03:14.280
Yeah, too little randomness.
00:03:18.122 --> 00:03:23.402
trillion, I think, which is just not enough. It needs to be, you know, 256 bit or
00:03:23.402 --> 00:03:28.002
128 bits. So like the number of possible seeds should be more than the number of
00:03:28.002 --> 00:03:30.762
molecules in the universe. And it wasn't.
00:03:31.540 --> 00:03:36.880
Yeah, the core security function of Bitcoin, I think you can put it that way,
00:03:37.020 --> 00:03:42.780
like the central reason that your Bitcoins are secure or should be secure is
00:03:42.780 --> 00:03:47.340
that the private key or the private keys or whatever the private keys are derived
00:03:47.340 --> 00:03:53.210
from is... A number that's so big that it's impossible to guess even with
00:03:53.210 --> 00:03:55.130
all the computers in the world, blah, blah, blah.
00:03:55.430 --> 00:03:57.550
There's just too many, too many.
00:03:55.812 --> 00:04:01.592
Yeah, I mean, it's one of the foundations, right? The other foundations are that
00:04:01.592 --> 00:04:06.132
the cryptography itself is secure, because if tomorrow it turns out that the SECP
00:04:06.132 --> 00:04:11.932
curve is insecure, then everybody loses their Bitcoin. And the
00:04:09.380 --> 00:04:09.980
Right.
00:04:11.932 --> 00:04:16.352
third one, I guess, is proof of work. If that doesn't work, then Bitcoin is also
00:04:16.352 --> 00:04:20.512
insecure in the sense that you just can't reliably make transactions anymore.
00:04:20.752 --> 00:04:22.852
But you can't steal somebody's coins then, but yeah.
00:04:23.540 --> 00:04:28.240
Yeah, you can double spend, etc. But yeah, at least one of the core security
00:04:28.240 --> 00:04:33.500
assumptions in Bitcoin is that when you generate private keys, then it's going to
00:04:33.500 --> 00:04:38.280
be random and out of such a big amount of possible numbers that no one can
00:04:38.280 --> 00:04:41.860
possibly guess it. And that's what went wrong here, right?
00:04:42.200 --> 00:04:45.100
So again, it's
00:04:42.732 --> 00:04:47.362
Yeah, and then the question, of course, how it went wrong, how that went wrong.
00:04:47.110 --> 00:04:53.030
also a core file, arguably, or maybe not even arguably, just... The number
00:04:53.030 --> 00:04:57.370
one most important thing that a hardware wallet is supposed to do is generate one
00:04:57.370 --> 00:05:02.650
of those random numbers. And it turns out that the way these random numbers were
00:05:02.650 --> 00:05:06.790
generated meant that they weren't exactly random or not random enough.
00:05:07.110 --> 00:05:08.610
And then the question is indeed,
00:05:07.722 --> 00:05:13.322
Well, I would say it's obviously it needs to be done correctly. I would say the
00:05:13.322 --> 00:05:16.582
number one task of a hardware wallet in general is to keep the secrets safe.
00:05:18.352 --> 00:05:21.932
But if it is not capable of making it securely, then it shouldn't make it
00:05:21.932 --> 00:05:24.852
securely, right? Then it should say, look, I'm not going to make a secret for
00:05:24.852 --> 00:05:29.992
you. You make it yourself and give it to me. But if the hardware wallet makes the
00:05:27.840 --> 00:05:28.160
right,
00:05:29.992 --> 00:05:33.072
secret, then it has to do it correctly. Otherwise, yeah, it's terrible.
00:05:33.360 --> 00:05:38.590
yeah. Yeah, so that's what went wrong in a nutshell. And then, yeah, so let's get
00:05:38.590 --> 00:05:43.950
into a little bit of the technical nitty gritty of what the actual problem was.
00:05:44.050 --> 00:05:45.150
Why did this not happen?
00:05:46.162 --> 00:05:51.622
Yeah, so the cold card has a chip in it. I think it's a secure element chip. And
00:05:51.622 --> 00:05:55.972
one of the things that secure element chip can do is make random numbers. And
00:05:55.972 --> 00:06:00.512
making random numbers is, you know, it's not that easy because you need lots of
00:06:00.512 --> 00:06:02.512
entropy from the universe, essentially.
00:06:03.812 --> 00:06:07.072
So there's, I don't know how these chips do that internally. They might just
00:06:07.072 --> 00:06:11.652
measure some noise in the environment, electricity, whatever. Whatever they do,
00:06:11.812 --> 00:06:15.452
they create a secure random number. And as far as we know, they do that
00:06:15.452 --> 00:06:17.392
correctly. So that's always a bit scary.
00:06:18.230 --> 00:06:24.150
Yeah, and so I made a, just real quick, I made a whole episode on hardware
00:06:24.150 --> 00:06:28.230
wallets. That was one of the few episodes I made without you. Back a couple of
00:06:28.230 --> 00:06:32.950
years ago, I made it with Lawrence Nahum of the Jade Wallet and Ben Kaufman of
00:06:32.950 --> 00:06:36.870
Spectre. And COLDCARD is one of the wallets we discussed.
00:06:36.870 --> 00:06:42.850
And what I remember from that is that... Yeah, so the secure element,
00:06:43.170 --> 00:06:48.150
we don't actually know how that part works internally, right? In the case of
00:06:47.922 --> 00:06:52.122
Yes. So what can go wrong, if
00:06:48.150 --> 00:06:53.010
COLDCARD. So the randomness is generated in some way that we,
00:06:54.260 --> 00:06:58.280
yeah, we don't actually know how it's generated in the case of the COLDCARD, I
00:06:58.280 --> 00:06:58.740
believe, right?
00:07:00.662 --> 00:07:05.522
you only use a secure element to generate the entropy, then at least the outside
00:07:05.522 --> 00:07:10.742
world has no way of knowing how that works. Why? Because these secure elements
00:07:10.742 --> 00:07:13.502
are typically patented and secret.
00:07:14.310 --> 00:07:19.110
Yeah, but that's not, but that, to be clear, that wasn't, to be clear, that
00:07:14.942 --> 00:07:17.362
But you can compensate for that by generating
00:07:19.110 --> 00:07:24.730
wasn't the issue here, right? That's not the, no, exactly. Okay, so, yeah,
00:07:19.442 --> 00:07:21.122
no, exactly.
00:07:22.822 --> 00:07:25.802
No, that's the kind of thing that still could go wrong with other hardware
00:07:25.802 --> 00:07:30.242
wallets or even this one. We don't know, right? But there are ways to compensate
00:07:30.242 --> 00:07:30.682
for that.
00:07:32.030 --> 00:07:32.810
yeah.
00:07:32.362 --> 00:07:36.802
But let's assume that the secure element just did its work, right? If it doesn't,
00:07:36.862 --> 00:07:40.602
then you would expect even older cold card versions also to start being drained,
00:07:40.642 --> 00:07:45.162
and we haven't seen that. No, something went wrong in the software that then sits
00:07:45.162 --> 00:07:50.972
on the cold card between you and the secure element. And its job was,
00:07:51.112 --> 00:07:56.832
and it did that correctly initially up to 2021, its job is to
00:07:56.832 --> 00:08:00.052
ask the chip for entropy, basically, and then use that.
00:08:00.452 --> 00:08:05.232
That's the simplest way to frame the job. Now, the problem here is that if you're
00:08:05.232 --> 00:08:10.972
running software on, well,
00:08:11.052 --> 00:08:16.392
let's go back a little bit. As a developer of hardware wallet firmware,
00:08:16.872 --> 00:08:20.692
you need to be able to test things. And the way you typically do that is in a
00:08:20.692 --> 00:08:23.052
simulator on your computer, not on the device itself.
00:08:23.292 --> 00:08:27.992
You probably do both, right? But it's a bit easier to work on a simulator because
00:08:27.992 --> 00:08:31.052
you need to quickly change something, try it, and you don't have to flash the
00:08:31.052 --> 00:08:31.772
firmware all the time.
00:08:33.442 --> 00:08:39.192
And therefore, the code somehow needs to work on both these
00:08:39.192 --> 00:08:42.952
environments. It needs to know whether it's running on the device or whether it's
00:08:42.952 --> 00:08:48.302
running... On the simulator. But that's a little bit dangerous because you don't
00:08:48.302 --> 00:08:49.322
want...
00:08:48.730 --> 00:08:52.570
Simulator, just to be clear, is basically a regular computer. It's what we're
00:08:52.432 --> 00:08:57.052
Yes, but that means it does not have that secure element. Might have another
00:08:52.570 --> 00:08:53.110
talking about.
00:08:54.610 --> 00:08:54.690
Right.
00:08:56.890 --> 00:08:59.420
Yes, right. Right.
00:08:57.052 --> 00:09:01.152
secure element, but not the same one. And so then the question is, how do you
00:09:01.152 --> 00:09:05.052
make that code work on both these environments? One very naive way is to say, I'm
00:09:05.052 --> 00:09:10.132
just going to check in real time where I am. But that's kind of dangerous because
00:09:10.132 --> 00:09:10.912
if...
00:09:10.912 --> 00:09:15.312
If you're trying to inspect how the device works, maybe something goes wrong with
00:09:15.312 --> 00:09:19.692
that check in the software in real time. And you don't want it to fall back to
00:09:19.692 --> 00:09:24.272
the simulator way of doing things. Also, sometimes you just can't even compile it
00:09:24.272 --> 00:09:29.522
because you can't run actual hardware wallet
00:09:29.582 --> 00:09:35.392
machine code on a normal computer sometimes because that chip doesn't exist.
00:09:35.552 --> 00:09:39.022
So the simulator is like, I don't, you know. I can't simulate that chip. I can
00:09:39.022 --> 00:09:42.462
simulate everything else, but I can't simulate the chip. There's some reasons why
00:09:42.462 --> 00:09:47.082
you can't run the exact same code on both the device and the simulator. And
00:09:47.082 --> 00:09:53.042
that's where languages like C have things called macros. And a macro is a way
00:09:53.042 --> 00:09:57.902
to say, or one thing you can do with a macro is to say, if I'm on the device,
00:09:58.102 --> 00:09:59.082
compile this code.
00:09:59.222 --> 00:10:03.242
And if I'm in the simulator, compile this other code. So the actual code that's
00:10:03.242 --> 00:10:07.062
compiled is different for the simulator and the real device.
00:10:08.842 --> 00:10:14.812
And the bug happened basically in that distinction. So the compiler would
00:10:14.812 --> 00:10:20.352
basically use some of the simulator code on the real device. Not all of it, but
00:10:20.352 --> 00:10:25.952
some of it. And in particular, the most critical bit of it, namely how to make
00:10:25.952 --> 00:10:26.572
the random data.
00:10:29.002 --> 00:10:34.022
So the real device uses or should have used the secure element to generate the
00:10:34.022 --> 00:10:39.362
random data. The simulator instead uses a Python library to do it. And that
00:10:39.362 --> 00:10:44.982
Python library was deliberately not actually cryptographically random. And that's
00:10:44.982 --> 00:10:47.782
fine because that Python library wasn't designed for any of this.
00:10:49.462 --> 00:10:52.782
But this is also where it gets really dangerous if you want to detect it.
00:10:49.890 --> 00:10:50.770
Okay. So,
00:10:54.082 --> 00:10:59.182
Because let's say the simulator had a very, very, very weak random number
00:10:59.182 --> 00:11:03.942
generator that only produces the number one. Like whenever you call it, it just
00:11:03.942 --> 00:11:07.562
says one. Well, that means anytime you run the simulator, you get the same seed.
00:11:08.022 --> 00:11:12.782
And so it'd be very obvious. And then if you had this bug, then the devices would
00:11:12.782 --> 00:11:13.882
all be making the same seed.
00:11:14.202 --> 00:11:18.122
That would have been caught like almost immediately, right? The first. QA
00:11:18.122 --> 00:11:20.602
probably would have seen it. And otherwise the customers would be like, why is
00:11:20.602 --> 00:11:23.402
there money on my account? Why is my money disappearing from my account?
00:11:24.762 --> 00:11:25.082
Right?
00:11:24.990 --> 00:11:30.150
yeah, which some people did claim even happened now, but we'll leave that aside.
00:11:32.082 --> 00:11:35.302
Collisions, right? So I think the number of secrets that these devices were
00:11:33.160 --> 00:11:33.740
Yeah.
00:11:35.302 --> 00:11:40.502
generating with the bug is still in the trillions. And then you can do some math
00:11:40.502 --> 00:11:45.282
on how many you need. I always forget. I think it's the square root. So you need
00:11:45.282 --> 00:11:49.602
a million devices to have any chance of overlap. But ask a statistician.
00:11:51.520 --> 00:11:57.210
Yeah, it's the birthday paradox, probably, right? And anyways, like I said, let's
00:11:51.872 --> 00:11:52.992
So probably not.
00:11:54.812 --> 00:11:55.892
Yeah, exactly.
00:11:57.210 --> 00:12:01.170
leave that aside. Yeah, you were saying if the randomness is like really bad,
00:12:01.310 --> 00:12:05.050
then it's obvious enough for anyone to detect. And if it's really good, then
00:12:05.050 --> 00:12:08.530
there's no problem. But it's this awkward middle space where it's still kind of
00:12:08.530 --> 00:12:10.770
good-ish and then no one...
00:12:08.942 --> 00:12:13.582
Yeah, it's a very, very dangerous middle space, right? It's good enough that you
00:12:11.700 --> 00:12:12.280
Yeah.
00:12:13.582 --> 00:12:18.582
can fool a human tester. It's probably even good enough that you can fool an
00:12:18.582 --> 00:12:24.032
automated tester because let's say you... As a company, you might have 100
00:12:24.032 --> 00:12:29.672
test devices or 1,000 test devices, and you could connect them all via USB to
00:12:29.672 --> 00:12:34.032
some computer and have that computer constantly flash it and make a new seed,
00:12:34.052 --> 00:12:37.712
make a new seed, make a new seed, and then see if there's any overlap.
00:12:39.392 --> 00:12:45.222
So if the number of random numbers was maybe 10,000, You know, maybe the machine
00:12:45.222 --> 00:12:48.662
would have caught it. But the number of unique outcomes, I think, is in the
00:12:48.662 --> 00:12:53.282
billions or more. And so it was good enough to fool the humans, good enough to
00:12:53.282 --> 00:12:59.182
fool any kind of automation, even good enough that users wouldn't see these like,
00:12:59.202 --> 00:13:01.102
hey, why is there money in my fresh wallet?
00:13:02.422 --> 00:13:04.862
But not good enough to defeat an attacker.
00:13:06.810 --> 00:13:12.810
Right. So in short, the problem as you describe it is that the test code was
00:13:12.810 --> 00:13:18.370
ultimately used as the real code on the device. That's the bug, basically. Is
00:13:18.370 --> 00:13:21.710
that a good shortcut to explaining it?
00:13:20.872 --> 00:13:24.112
Yeah, not all the test code, right? But the very critical part of the test code.
00:13:24.862 --> 00:13:27.982
Yes. And so, you know, then you can look at how do you prevent problems like
00:13:24.910 --> 00:13:25.230
Right.
00:13:27.982 --> 00:13:28.302
this?
00:13:29.562 --> 00:13:31.122
Well, obviously,
00:13:29.760 --> 00:13:32.480
Yeah. Or how could this happen in the first place?
00:13:34.502 --> 00:13:36.202
yeah. So how it happened?
00:13:34.840 --> 00:13:38.280
You're a developer, right? Like, is this not an obvious thing that you would see?
00:13:38.400 --> 00:13:40.900
Apparently not, right? Because no one saw it for a long time.
00:13:39.862 --> 00:13:40.382
No.
00:13:42.302 --> 00:13:47.722
Well, I guess it depends, right? If your expertise is embedded systems, and
00:13:47.722 --> 00:13:52.182
you saw the change that was causing this, say, the switching between two
00:13:52.182 --> 00:13:56.022
libraries, maybe you would have noticed it. You're like, hey, this is a
00:13:56.022 --> 00:14:01.932
sloppy change. But if you look at the final result, maybe you
00:14:01.932 --> 00:14:03.992
wouldn't because it's very subtle.
00:14:04.112 --> 00:14:09.942
I think it's the difference between ifdef and if zero. So one says, is this
00:14:09.942 --> 00:14:14.182
variable defined? And the other says, is this variable not zero? And these are
00:14:14.182 --> 00:14:18.962
quite typical mistakes to make in code. We run into them in Bitcoin Core multiple
00:14:18.962 --> 00:14:23.602
times where some components shouldn't be included or should be included.
00:14:23.782 --> 00:14:26.242
It's just not happening in any critical place.
00:14:29.880 --> 00:14:31.220
Right, okay, so,
00:14:30.912 --> 00:14:36.892
But there's a second bug that made this worse. Or, you know, this bug
00:14:36.892 --> 00:14:39.712
could have existed and nothing would have gone wrong if there were some other
00:14:39.712 --> 00:14:42.852
things that had happened. So that might be useful to get into.
00:14:41.910 --> 00:14:47.050
okay, and just to be clear, because there's different versions of the gold cards,
00:14:47.230 --> 00:14:51.370
right, and some of them were affected more than others, so what you just
00:14:51.370 --> 00:14:55.730
described, there was test code, and even though the secure element presumably
00:14:55.730 --> 00:15:00.850
worked, the test code didn't call on the secure element for the randomness, and
00:15:00.850 --> 00:15:01.190
it was,
00:15:00.992 --> 00:15:06.112
The real code, yeah, exactly. The real code didn't use the test path to get the
00:15:06.112 --> 00:15:08.192
randomness instead of the secure element.
00:15:07.330 --> 00:15:13.250
yeah, and is this the MK3 problem already, or is there more to it there as
00:15:13.250 --> 00:15:13.450
well?
00:15:14.342 --> 00:15:18.922
No, that's what happened on MK3. But even on MK3 could have been prevented with
00:15:18.922 --> 00:15:20.082
additional randomness.
00:15:21.372 --> 00:15:26.732
So there's something quite nice about entropy, which is that if you take bad
00:15:26.732 --> 00:15:32.052
entropy, so low entropy, not enough randomness in it, and you add to it
00:15:32.052 --> 00:15:37.072
very good entropy, or the other way around, I guess, easier to understand. If you
00:15:37.072 --> 00:15:43.012
take very good entropy, so completely quantum number generated noise, and
00:15:43.012 --> 00:15:46.912
you add to it bad entropy, like just the word hello, it doesn't matter.
00:15:47.092 --> 00:15:50.192
Like you do not weaken entropy. Entropy only gets stronger. Entropy just
00:15:50.192 --> 00:15:55.202
increases if you add it. That's a very nice property. And that means what you can
00:15:55.202 --> 00:15:59.962
do is you can take, once as a device that's trying to generate a key, you can
00:15:59.962 --> 00:16:03.442
take the entropy from the skewer element, and then you can start adding other
00:16:03.442 --> 00:16:08.632
stuff to it. And the coldcard was doing that a little bit, but not enough.
00:16:04.740 --> 00:16:05.180
Right.
00:16:07.880 --> 00:16:09.260
The MK3 also.
00:16:10.272 --> 00:16:12.192
I think so, yes.
00:16:14.142 --> 00:16:19.622
I'm not sure. I think, yes, the amp-key3 added a little bit about the timing of
00:16:19.622 --> 00:16:20.802
where it was in the boot sequence.
00:16:22.612 --> 00:16:26.512
Because if you just take the Python random number generator that they used
00:16:26.512 --> 00:16:31.432
without any other data, then they would have all had the same seed because it was
00:16:31.432 --> 00:16:35.372
a deterministic random number generator. This always produces the same thing
00:16:33.620 --> 00:16:34.800
Oh, right. Okay.
00:16:35.372 --> 00:16:39.172
given the same input. So they already, in a way, added a little bit of entropy,
00:16:39.272 --> 00:16:43.822
but it was just not enough. But typically what you do is you generate your seed,
00:16:44.022 --> 00:16:48.602
or sorry, you generate some entropy with one source, and then you XOR, it's like
00:16:48.602 --> 00:16:52.722
an addition, essentially, another source of entropy into it, and you just keep
00:16:52.722 --> 00:16:54.302
doing that with other sources.
00:16:54.482 --> 00:16:59.632
And these other sources can be, well, on a normal computer, you can take a
00:16:59.632 --> 00:17:05.432
screenshot off your desktop, you can measure, you can read a bunch of files, just
00:17:05.432 --> 00:17:06.812
random files and hash those.
00:17:08.772 --> 00:17:13.372
You can look at mouse movements. I don't know if people used PGP back in the 90s,
00:17:13.472 --> 00:17:18.932
but then you'd have to wiggle your mouse for like a minute for it to have enough
00:17:18.932 --> 00:17:24.142
entropy to create a secure PGP key. I don't know if that was actually true or
00:17:22.150 --> 00:17:27.920
Yeah, also with the original paper wallet websites
00:17:24.142 --> 00:17:25.042
just therapeutic, but
00:17:27.920 --> 00:17:31.380
that you had in like 2013, 2014, that was also a thing.
00:17:31.852 --> 00:17:35.572
yeah, but some of these paper wallets were highly insecure despite that.
00:17:36.140 --> 00:17:39.980
I know I'm not recommending them, but I'm just pointing out that was like a thing
00:17:37.022 --> 00:17:37.422
Yeah,
00:17:39.980 --> 00:17:42.530
back then. And should
00:17:40.672 --> 00:17:43.312
so mouse movement makes sense. Now, the problem with hardware wallets is they
00:17:43.312 --> 00:17:48.552
have less of that, right? Because they are isolated devices, they have very
00:17:48.552 --> 00:17:52.852
little software, so the file system is very predictable. It's not like you can
00:17:52.852 --> 00:17:57.752
just look at somebody's desktop background or whenever they last opened their
00:17:57.752 --> 00:17:59.972
email. There's none of that on the hardware wallet.
00:18:00.072 --> 00:18:05.972
It's much more sterile, much more similar. So it probably
00:18:05.972 --> 00:18:09.172
wouldn't be enough to just look at random things on the hardware wallet. The
00:18:09.172 --> 00:18:12.232
hardware wallet also doesn't know the actual time in the universe. It just knows
00:18:12.232 --> 00:18:16.202
how long it's been awake. But that's typically very short because you're just
00:18:16.202 --> 00:18:21.692
creating a new one. So things you could probably do is have a
00:18:21.692 --> 00:18:23.872
gravity sensor or accelerometer.
00:18:24.192 --> 00:18:27.592
You know, your phone knows which side is up because it knows which way the
00:18:27.592 --> 00:18:31.612
gravity is pulling on the sensor. And so if you just tell the user to rotate
00:18:31.612 --> 00:18:35.332
their device a couple of times and you have such a sensor, that could be an
00:18:35.332 --> 00:18:36.512
additional source of entropy.
00:18:38.442 --> 00:18:41.222
We just tell the user, like, shake the device for, like, two seconds.
00:18:42.582 --> 00:18:46.882
And you just add that to what the secure element is giving. Some hardware
00:18:46.882 --> 00:18:50.062
wallets, I think, nowadays have cameras. Those are great, too, for entropy.
00:18:50.832 --> 00:18:54.372
Because they just take, essentially, they just take photons from the universe.
00:18:54.872 --> 00:18:59.032
And these are all just completely unique. Assuming the camera is working, right?
00:18:59.072 --> 00:19:02.772
If the camera is black, like there's nothing coming in, well, then they're all
00:19:02.772 --> 00:19:03.092
the same.
00:19:04.152 --> 00:19:07.372
So you would never want to rely on any of these single sources. The sensor is
00:19:07.372 --> 00:19:12.132
broken, it's not giving random data, then you still have a problem. So if they
00:19:12.132 --> 00:19:17.322
had done that, that would have compensated for this, basically. But on the MK4,
00:19:17.402 --> 00:19:21.582
there was an even better situation, in theory, because they had multiple secure
00:19:21.582 --> 00:19:25.462
elements. And they were combining entropy from all three of them.
00:19:26.012 --> 00:19:27.112
They just did it wrong.
00:19:29.510 --> 00:19:35.020
we... About the MK3 real quick, because that's the worst one,
00:19:35.300 --> 00:19:39.180
the one that's worst effective. So I think what we just described about the test
00:19:39.180 --> 00:19:43.680
code and all that, that's true for all of them, right? For all reasons, once, but
00:19:43.512 --> 00:19:47.672
Yeah, that's the fundamental thing that went wrong. What I just described after
00:19:43.680 --> 00:19:44.520
then some added,
00:19:46.580 --> 00:19:47.660
yeah, but then some,
00:19:47.672 --> 00:19:49.772
this is like, what could have made it less bad?
00:19:49.880 --> 00:19:55.180
yeah, but then some did make it less bad, right? So how do you know the
00:19:55.180 --> 00:19:59.280
differences between like what else went wrong or didn't go wrong or what
00:19:56.982 --> 00:19:57.382
Yeah.
00:19:59.280 --> 00:20:02.860
mitigation did some of the wallets still have?
00:20:01.342 --> 00:20:05.862
Yeah, so what happened in the MK4 is that they did add more entropy, namely from
00:20:05.862 --> 00:20:09.942
these other secure elements. And if they had done that correctly, namely adding
00:20:09.942 --> 00:20:13.782
all the entropy from the other secure elements, then there would not have been a
00:20:13.782 --> 00:20:17.142
problem at all, because as long as one of them works, you're good.
00:20:18.170 --> 00:20:21.730
Okay, so they did not do that correctly either. So what did they do then?
00:20:18.722 --> 00:20:19.562
But they didn't.
00:20:20.832 --> 00:20:26.172
No. They took some entropy from these other devices, from these other secure
00:20:26.172 --> 00:20:29.792
elements, but then they trimmed it. So they only took, like, I think it was four
00:20:29.792 --> 00:20:33.312
bytes of it. 4 times 8 is 32 bits.
00:20:34.812 --> 00:20:37.872
So they had only 32 bits of real extra entropy.
00:20:39.720 --> 00:20:42.020
So they took four bits out of how many?
00:20:43.522 --> 00:20:49.162
Well, they could have taken infinity, but let's say normally you'd want a seed
00:20:49.162 --> 00:20:55.142
that's 256 bit, so 32 bytes. So they could have taken
00:20:55.142 --> 00:21:00.702
32 bytes from one of the secure enclaves, taken another 32 bytes from the other
00:21:00.702 --> 00:21:02.582
one, and XOR all of them.
00:21:02.642 --> 00:21:07.082
So just take all the 32 bytes from each of the devices. But instead, they took
00:21:07.082 --> 00:21:11.642
only four. And I don't know why, because that is the kind of thing that I think
00:21:11.642 --> 00:21:17.012
anyone looking at that code would have spotted. But anyone knowing a little bit
00:21:17.012 --> 00:21:17.492
about entropy.
00:21:18.170 --> 00:21:23.940
Okay, and this effect, correct me if I'm wrong. I believe so that they took four
00:21:23.940 --> 00:21:29.420
bytes out of 32 for a reason that you don't understand. But then this was
00:21:29.420 --> 00:21:35.260
itself also hashed again, right? With the Yasmarang. That's just
00:21:35.260 --> 00:21:38.020
a MK4 and your specific feeling.
00:21:36.482 --> 00:21:40.922
Yeah, so the Yasmarang is this Python library. So the Yasmarang is this Python
00:21:38.140 --> 00:21:38.500
Is that correct?
00:21:40.922 --> 00:21:44.442
pseudorandom number generator. We talked about that in the milk set episode,
00:21:44.602 --> 00:21:49.532
about the general phenomena of a pseudorandom number generator. It takes some
00:21:49.532 --> 00:21:54.322
input. And it gives a random looking number, but it's actually, if you give the
00:21:54.322 --> 00:21:55.902
same input, you always get the same output.
00:21:56.002 --> 00:22:01.632
So it's not really random. And so if the input to that generator
00:21:57.940 --> 00:21:58.420
Yeah.
00:22:01.632 --> 00:22:07.612
is four bytes, essentially, four bytes of uniqueness, then you're only going
00:22:07.612 --> 00:22:11.602
to get four bytes of worth of entropy in the end.
00:22:13.242 --> 00:22:17.382
Because you essentially just have these four bytes and the pseudorandom number
00:22:17.382 --> 00:22:21.402
generator doesn't add anything. It's just four bytes in is four bytes out in
00:22:21.402 --> 00:22:21.942
terms of entropy.
00:22:24.260 --> 00:22:29.300
So this really does sound like it's disaster upon disaster upon disaster then.
00:22:30.682 --> 00:22:36.622
No, it's a disaster that could have been fixed in the MK4, could
00:22:36.622 --> 00:22:40.622
have been harmless in the MK4, but wasn't harmless because they didn't use the
00:22:40.622 --> 00:22:45.042
entropy they had, the extra entropy they had. They just didn't use it. So like
00:22:44.220 --> 00:22:46.700
Yeah, well, that's a second disaster then, right?
00:22:45.042 --> 00:22:47.102
you add two chips that could have given you...
00:22:48.512 --> 00:22:51.412
Yeah, I guess it depends on your definition of disaster. Yeah.
00:22:51.290 --> 00:22:54.010
How would you call it? Blunder?
00:22:55.872 --> 00:23:01.672
It's one disaster plus one bad implementation,
00:23:01.672 --> 00:23:02.532
I guess. I don't know.
00:23:03.972 --> 00:23:07.552
I don't know what the reason is why they only kept four bytes, but my guess would
00:23:07.552 --> 00:23:12.312
be that they did not have this intuition that good entropy plus bad entropy is
00:23:12.312 --> 00:23:18.092
good entropy. Because if you think, generally, if you think a nice cup of
00:23:18.092 --> 00:23:22.632
tea plus poison is just bad tea, right?
00:23:23.612 --> 00:23:26.992
But entropy doesn't work that way. You can just add as much badness or poison. So
00:23:26.992 --> 00:23:30.992
they might have just mistrusted these other secure devices. They're like, well,
00:23:31.412 --> 00:23:34.012
they're different manufacturers, but we don't fully trust them, so we don't use
00:23:34.012 --> 00:23:37.812
all of it, what's coming out of it. But that's the wrong intuition.
00:23:39.450 --> 00:23:43.450
Right. Let me pull this apart real quick just to make sure that I understand and
00:23:43.450 --> 00:23:48.630
then hopefully our listeners also. Because I do want to point out, it sounds to
00:23:48.630 --> 00:23:54.510
me there are three different mistakes that happened on different...
00:23:53.282 --> 00:23:55.822
Probably more, but there's two critical ones, yeah.
00:23:56.370 --> 00:24:01.890
Yeah, so the first one is that there was a bug in the firmware, in the code that
00:24:01.890 --> 00:24:06.710
made the COLDCARD use test codes, which therefore didn't pull the randomness from
00:24:06.710 --> 00:24:12.550
the secure element. That's one. We discussed that one. Two is only
00:24:12.550 --> 00:24:18.070
four bytes out of 32 were used when the secure elements were combined.
00:24:18.550 --> 00:24:22.270
Instead of just using all the randomness, they only use part of the randomness.
00:24:22.470 --> 00:24:23.970
And then the third mistake.
00:24:25.750 --> 00:24:30.770
Did I get this straight? Is that the Yasmarang thing is a pseudo-random, is not
00:24:27.662 --> 00:24:29.642
Yeah, so another way to put that is to say,
00:24:30.770 --> 00:24:34.950
even a random number generator at all, not a good one. So there were three
00:24:34.162 --> 00:24:37.942
well, that is part of the test code. The pseudo-random number generator is the
00:24:34.950 --> 00:24:36.030
mistakes in that sense.
00:24:37.942 --> 00:24:43.932
test code. So you basically take on the... Yes.
00:24:40.420 --> 00:24:42.210
Is that Yasmarang?
00:24:44.172 --> 00:24:49.632
So what happens is on the MK3, you take nothing, you don't take the
00:24:44.340 --> 00:24:45.320
Oh, okay.
00:24:49.632 --> 00:24:52.192
random number generator, and you just give it to Yasmarang.
00:24:53.152 --> 00:24:58.552
And on the MK4, I think you did that, plus you then added
00:24:53.670 --> 00:24:54.190
Okay.
00:24:58.552 --> 00:25:01.372
real secure elements, but you only added four bytes of them.
00:25:02.850 --> 00:25:05.030
Okay, I see. So then there's...
00:25:03.612 --> 00:25:07.972
And I think those did not go to the Yasmarang. So it's like you take nothing, you
00:25:07.972 --> 00:25:12.482
put it through Yasmarang, and then that's one source, and then you add another
00:25:12.482 --> 00:25:14.022
source, but you don't add the full source.
00:25:17.100 --> 00:25:17.620
Right.
00:25:17.112 --> 00:25:22.612
And that first one, I think, could have been missed in a security audit of that
00:25:22.612 --> 00:25:25.992
part of the code. The second, I don't understand it.
00:25:27.862 --> 00:25:31.082
I mean, of course, everything can be missed. And it depends on what you tell an
00:25:28.180 --> 00:25:28.700
Okay.
00:25:31.082 --> 00:25:34.822
auditor and what that auditor's experience is. You know, if you tell the auditor,
00:25:35.002 --> 00:25:38.482
like, make sure nobody can steal the secret, then they might not even look at
00:25:38.482 --> 00:25:39.202
this part of the code.
00:25:40.350 --> 00:25:46.160
Yeah, so let's get there next. But first, does this cover the bugs? Like the
00:25:46.160 --> 00:25:50.940
actual mistakes, you know, the heart of the disaster. Does this cover that part
00:25:50.940 --> 00:25:55.100
before we move on to? Yeah. OK, so that
00:25:51.612 --> 00:25:55.572
Yeah, I think so. I mean, you can make it more subtle, but those are the two
00:25:55.572 --> 00:25:55.832
things.
00:25:56.770 --> 00:26:01.870
this is basically what went wrong. OK, so then, yeah, let's get into a little bit
00:26:01.870 --> 00:26:04.770
of, you know, how could this have happened?
00:26:06.590 --> 00:26:11.150
One of the one of the things I saw out there is.
00:26:12.520 --> 00:26:17.160
That, you know, one of the mistakes that were made is sort of a meta mistake is
00:26:17.160 --> 00:26:22.360
that the firmware, the source code itself was no longer open source since a
00:26:22.360 --> 00:26:27.220
couple of years. You know, the backstory is another hardware wallet manufacturer
00:26:27.220 --> 00:26:32.830
foundation. Wanted to take the open source code that COLDCARD was using.
00:26:34.330 --> 00:26:39.690
The Coinkite people, Novak, didn't like it. And they decided to
00:26:39.690 --> 00:26:45.210
turn the source code into source available rather than open source. Which, if I
00:26:45.210 --> 00:26:49.170
understand correctly, means the code is out there, but it cannot be commercially
00:26:49.170 --> 00:26:54.390
used by any other manufacturer or product. And this would have taken away, the
00:26:54.390 --> 00:26:57.450
claim is this would have taken away the incentive for other
00:26:58.710 --> 00:27:02.630
Developers, security, auditors, whatever, to actually properly look at the code.
00:27:02.770 --> 00:27:07.070
And that's why it could stay in there for so long. What do you think of this
00:27:07.070 --> 00:27:09.430
claim? Do you agree with this?
00:27:09.022 --> 00:27:14.542
First, we should disentangle two things. One is the changing of the library is
00:27:14.542 --> 00:27:18.102
what introduced this bug, just the act of changing it, regardless of whether the
00:27:18.102 --> 00:27:21.722
new library was good or bad. It was the act of swapping the libraries where the
00:27:21.722 --> 00:27:23.922
mistake was introduced, the first mistake.
00:27:26.082 --> 00:27:28.582
But your question is more important, I think.
00:27:30.022 --> 00:27:33.982
I don't think that explains it. It's one of those things that could have made a
00:27:33.982 --> 00:27:38.062
marginal difference. Maybe some would have caught it. But you have to understand
00:27:38.062 --> 00:27:44.052
that Source available. If it was completely open source, then there would have
00:27:44.052 --> 00:27:48.512
been more companies using the source. Therefore, maybe more people incentivized
00:27:48.512 --> 00:27:49.112
to look at it.
00:27:49.372 --> 00:27:55.292
But if these companies would have just lazily copied the source and not looked
00:27:55.292 --> 00:28:00.852
at it, they might have made the same mistake. That's one aspect of it. The other
00:28:00.852 --> 00:28:04.312
is that there was quite an incentive out there to look at the source anyway.
00:28:05.042 --> 00:28:09.302
Because there's a lot of prestige for security researchers and competitors to
00:28:09.302 --> 00:28:11.622
find bad bugs in their competitor.
00:28:12.022 --> 00:28:15.222
And we know that companies, you know, there's been many conferences where people
00:28:15.222 --> 00:28:18.962
would show, oh, look, I can play snake on the ledger. Or like, look, you know, I
00:28:18.962 --> 00:28:21.042
can take a hammer and take the seed out of a treasure.
00:28:22.300 --> 00:28:22.740
Right.
00:28:22.742 --> 00:28:27.262
So we know people were doing that. People were spending quite serious money on
00:28:27.262 --> 00:28:30.692
trying to do that. So that's why I don't think in this case it would have made a
00:28:30.692 --> 00:28:34.552
difference. Just because you can't commercially reuse the software source doesn't
00:28:34.552 --> 00:28:39.902
mean you're not going to look at it. Another thing that's important is that the
00:28:39.902 --> 00:28:45.442
new library they use is kind of a wrapper library around existing ones that are
00:28:45.442 --> 00:28:46.262
free and open source.
00:28:46.782 --> 00:28:51.662
So it uses internally libsecp, you know, the library we did in episode two, I
00:28:51.662 --> 00:28:57.142
think about. And that's a very mature, very well audited library. The problem is
00:28:57.142 --> 00:29:02.062
that even libsecp tells you to bring your own entropy. So the critical part is
00:29:02.062 --> 00:29:03.382
something that's not in the library.
00:29:06.602 --> 00:29:10.842
So even if the library was completely fine, it wouldn't have mattered because it
00:29:10.842 --> 00:29:13.962
is basically the final part where you say, okay,
00:29:15.442 --> 00:29:19.882
give me some entropy because how you get entropy depends on the device. So if you
00:29:19.882 --> 00:29:23.502
are a hardware wallet, you have a very different way of getting entropy than when
00:29:23.502 --> 00:29:27.152
you are a software wallet. So that's just left to the wallet developer.
00:29:29.650 --> 00:29:31.710
Yeah, I mean, this is what I'm getting at is that,
00:29:29.922 --> 00:29:32.922
And that part is going to be unique anyway, and you can't really copy it without
00:29:32.922 --> 00:29:34.402
completely cloning the device.
00:29:36.810 --> 00:29:42.790
yeah. What about, you know, I'm just going down some claims and just kind
00:29:42.790 --> 00:29:48.720
of picking your brain on, you know, because I... I do get the impression a
00:29:48.720 --> 00:29:52.700
little bit that there's a lot of, you know, it's a bit of Captain Hind sighting
00:29:52.700 --> 00:29:58.480
where, you know, after the disaster happens, you know, people are looking,
00:29:57.012 --> 00:30:02.332
It's not entirely kept in hindsight because, like I can speak of my own
00:30:02.332 --> 00:30:06.212
experience, I have worked on two software walls, right? On the blockchain.com
00:30:06.212 --> 00:30:10.792
wallet back in the day when I was still blockchain.info and on the Bitcoin Core
00:30:10.792 --> 00:30:14.472
wallet. And in both cases, I've looked at some of that random number generation
00:30:14.472 --> 00:30:19.512
code. And so I know at least, you know, some best practices there.
00:30:21.462 --> 00:30:25.842
So in other words, I think if I had looked at this code, I would have caught the
00:30:21.490 --> 00:30:23.530
okay, well, that's,
00:30:25.842 --> 00:30:29.762
second problem. Not the first one. Not the wrong if def statement. I don't think
00:30:29.762 --> 00:30:33.462
I would have seen it. But the like, hey, why are you not adding much more entropy
00:30:33.462 --> 00:30:38.362
if you have it? That I would have complained about. In an audit, personally. It's
00:30:38.362 --> 00:30:40.262
not the kind of thing that you go on Twitter and rant about.
00:30:40.842 --> 00:30:41.682
But it's one of those.
00:30:43.270 --> 00:30:48.440
yeah, yeah, yeah, but my question is a bit more. Yeah, but my question is a
00:30:43.272 --> 00:30:46.432
Right? So I think other people with this experience would have seen that too.
00:30:48.440 --> 00:30:54.380
little bit more because now you're addressing like the software, the bug itself,
00:30:54.780 --> 00:30:59.520
which that's where, you know, I'm not an expert in that at all. And you are. So I
00:30:55.332 --> 00:30:55.612
Mm-hmm.
00:30:59.520 --> 00:31:04.500
believe you when you say that. But I'm a little bit more asking about sort of the
00:31:04.500 --> 00:31:09.260
meta behavior or best practices in the sense of, you know, how was the software
00:31:09.260 --> 00:31:09.820
developed?
00:31:10.180 --> 00:31:15.840
So indeed, for example, this difference between open sourcing and code available.
00:31:10.622 --> 00:31:11.082
Mm-hmm.
00:31:16.530 --> 00:31:21.150
Or another example that people have brought up is that the COLDCARD wouldn't have
00:31:21.150 --> 00:31:26.790
had like a proper bug bounty program. Like these kinds of things. How do you see
00:31:26.790 --> 00:31:30.910
these concerns? So you address the code available source.
00:31:28.992 --> 00:31:32.232
Well, the bug bounty program, like I said, I don't think that matters because
00:31:32.232 --> 00:31:36.072
there's enough prestige to find these bugs without a bug bounty program. I mean,
00:31:36.152 --> 00:31:39.712
it's not that, you know, it would be nice if they did, but I don't think it's
00:31:39.712 --> 00:31:41.872
going to make that much of a difference. There is an incentive to look.
00:31:42.250 --> 00:31:42.830
Yeah.
00:31:43.082 --> 00:31:46.722
The other part is the open source versus closed source. Now, it's been a few
00:31:46.722 --> 00:31:50.522
years, but I have actually made a pull request to the cold card to add some
00:31:50.522 --> 00:31:55.042
descriptor support like many years ago, you know, with Bitcoin Core. And so they,
00:31:55.202 --> 00:31:58.982
at least then, I don't know if they still do it, but... They then did have a
00:31:58.982 --> 00:32:02.702
somewhat collaborative culture where other people would contribute to their code.
00:32:03.082 --> 00:32:06.722
And people, I don't know if they're still doing that, but that's a way to get
00:32:06.722 --> 00:32:07.722
eyes on your code.
00:32:09.002 --> 00:32:12.342
One of the problems in open source is that you can publish and you can make it
00:32:12.342 --> 00:32:18.182
free, like in Libra, as much as you want, but if nobody contributes to
00:32:18.182 --> 00:32:24.102
it, they're also not going to look at it. So there's plenty of, there might
00:32:21.480 --> 00:32:22.480
Yeah, sure.
00:32:24.102 --> 00:32:27.902
be, you know, super critical code out there that is completely free and open
00:32:27.902 --> 00:32:31.822
source, has a giant bug in it, and just people don't see it because nobody's
00:32:31.822 --> 00:32:32.322
looking at it.
00:32:32.462 --> 00:32:32.882
I
00:32:32.790 --> 00:32:37.760
Yeah. Yeah. Well, to that point, another, you know.
00:32:38.510 --> 00:32:42.730
The complaint or concern that people brought up is like, were there or should
00:32:42.730 --> 00:32:47.950
there have been like formal security audits on this kind of code? And if there
00:32:47.950 --> 00:32:51.410
weren't, then why didn't that happen? Do you have any thoughts on that?
00:32:53.012 --> 00:32:57.132
mean, I guess you can come up with a, you know, a regulatory framework where you
00:32:57.132 --> 00:33:03.132
can say like, hey, if you are producing hardware wallets or some class of
00:33:03.132 --> 00:33:08.412
security products, you need to do this and this and this audits. That's a
00:33:08.412 --> 00:33:13.482
government thing. Or like, you know, it's a marketing thing where you say, we
00:33:13.482 --> 00:33:16.762
have, even without the government intervention, you can say, look, we have this
00:33:16.762 --> 00:33:17.422
security audit.
00:33:17.902 --> 00:33:21.582
That's a part of your marketing then. And the customers can decide whether they
00:33:21.582 --> 00:33:26.482
want to buy a product that has this certification or not. Now, of course, we know
00:33:26.482 --> 00:33:31.342
that all sorts of certification schemes can become paper tigers.
00:33:32.372 --> 00:33:36.192
Like the DigiNotar example in the Netherlands, where like a domain.
00:33:36.822 --> 00:33:41.042
I think it was a top-level domain provider or something that passed all the
00:33:41.042 --> 00:33:45.162
audits, but they were completely insecure. So I don't know if that would have
00:33:45.162 --> 00:33:45.402
helped.
00:33:46.802 --> 00:33:52.282
But maybe it is good to have some sort of quality standard. But then still, what
00:33:52.282 --> 00:33:57.212
do you tell the auditor? Which auditor do you hire? What should their expertise
00:33:57.212 --> 00:34:01.932
be? Sure, now you can hire auditors to look for the exact same bug, and you'll
00:34:01.932 --> 00:34:03.512
find it if it's in other devices.
00:34:05.342 --> 00:34:11.182
You don't know what the next disaster is going to be. And especially if you have
00:34:11.182 --> 00:34:15.712
a hardware wallet with lots of whistles and bells, bells and whistles, you know,
00:34:15.772 --> 00:34:19.752
at least the cold card and some other devices only support Bitcoin. But now
00:34:19.752 --> 00:34:24.312
imagine you need to audit a hardware wallet that has 15,000 shitcoins on it.
00:34:26.542 --> 00:34:30.342
And some of the bugs have nothing to do with the cryptography. They're just
00:34:30.342 --> 00:34:34.812
like... Oh, if you don't check this field in an Ethereum transaction, somebody
00:34:34.812 --> 00:34:39.092
can trick you into losing all your wallet. And that's a huge class of bugs that
00:34:39.092 --> 00:34:43.032
you're just not going to find in an audit. That's just a North Korean audit,
00:34:43.172 --> 00:34:44.952
basically. You find out when it goes wrong.
00:34:47.262 --> 00:34:52.582
So I'm not saying that they were fine, they did enough. I'm just saying I
00:34:47.520 --> 00:34:48.040
Yeah, so,
00:34:52.582 --> 00:34:57.342
wouldn't count on it that it would have... It's all incremental. More security
00:34:57.342 --> 00:35:03.082
audits, good. More open source. Instead of view only, it's all better.
00:35:03.302 --> 00:35:06.502
So you're just decreasing the chances by doing all these things better.
00:35:06.882 --> 00:35:12.402
And the other is just write code much more carefully with much more review. You
00:35:12.402 --> 00:35:18.042
don't swap a giant library into commits with 2,000 lines of code. That's not very
00:35:18.042 --> 00:35:20.202
good software development style.
00:35:22.102 --> 00:35:26.542
But again, if you look at software that, you know, in retrospect, when you find a
00:35:22.480 --> 00:35:23.700
right. Well, that's...
00:35:26.542 --> 00:35:29.162
bug, it's often in a sloppy part of the code base.
00:35:30.872 --> 00:35:32.892
But I don't know what that means.
00:35:34.140 --> 00:35:40.020
Yeah, well, so I wanted to ask you, but you sort of answered it in a,
00:35:40.020 --> 00:35:45.120
well, kind of in a depressing way, you kind of non-answered it. It is like,
00:35:45.380 --> 00:35:51.200
how do we move forward? How do we prevent something like this? So all of
00:35:51.200 --> 00:35:55.260
these things would have been a bit better, you say, like it would have been a bit
00:35:55.260 --> 00:35:56.500
better if it was open source.
00:35:56.720 --> 00:36:02.000
It would have been a bit better if there were maybe security audits or, but.
00:36:02.710 --> 00:36:07.550
None of it would have necessarily prevented this. And probably there is no way.
00:36:06.622 --> 00:36:10.342
No, there is, I mean, what is interesting here is that,
00:36:08.390 --> 00:36:13.410
What are the lessons, Sjors? What do we learn from this, if anything?
00:36:11.602 --> 00:36:17.562
well, this class of bug where you think you have randomness but
00:36:17.562 --> 00:36:21.442
you don't have enough has happened a few times before, right? We did an episode
00:36:21.442 --> 00:36:26.962
on Milksad where this wasn't an actual wallet, it was just a tutorial. The
00:36:26.962 --> 00:36:29.802
tutorial used to be safe and then something changed in the library and the
00:36:29.802 --> 00:36:31.442
tutorial is suddenly completely unsafe.
00:36:31.962 --> 00:36:36.642
But it wasn't obvious enough to immediately see it. And historically, there was,
00:36:36.742 --> 00:36:40.542
I think, the blockchain.info wallet just before I was there, or at least not in
00:36:40.542 --> 00:36:45.602
the area I was working on, on the Android wallet, where they were, again, they
00:36:45.602 --> 00:36:49.502
were using randomness from the device, and then they would fall back to some
00:36:49.502 --> 00:36:54.402
random.org website that would, you know, just in case, just give you something,
00:36:54.582 --> 00:36:54.702
right?
00:36:55.162 --> 00:36:58.882
And I think what turned out is that that website was gone, so it was just saying
00:36:58.882 --> 00:37:04.342
404. So there's no entropy there. And then the device entropy either didn't work
00:37:04.342 --> 00:37:08.742
or they didn't take enough bits. And again, it wasn't obvious enough that users
00:37:08.742 --> 00:37:13.842
would say, hey, why do I have balance in my new wallet? QA wouldn't have seen it,
00:37:13.902 --> 00:37:18.942
but it was bad enough that anybody understanding what's going on could steal all
00:37:18.942 --> 00:37:19.282
the coins.
00:37:19.422 --> 00:37:23.202
And in that case, it was all, I think, sweeped by a good actor and eventually
00:37:23.202 --> 00:37:29.012
redistributed. This class of bug, there is some discussion of how you would
00:37:29.012 --> 00:37:34.742
prevent it. One line of argument would say make it an all or nothing proposition.
00:37:35.662 --> 00:37:41.482
Make sure the test code has absolutely no entropy in it so that it's
00:37:41.482 --> 00:37:42.762
very clearly test code.
00:37:43.002 --> 00:37:48.042
And don't add mediocre entropy. So don't add keystrokes or timing from the device
00:37:48.042 --> 00:37:52.182
because it's enough to obfuscate the bug, but it's not enough to actually save
00:37:52.182 --> 00:37:56.602
it. Whereas another line of thinking, which I kind of prefer, is saying, no, you
00:37:56.602 --> 00:38:00.302
should just add more entropy. Like you should make the device such that even if
00:38:00.302 --> 00:38:03.582
the secure element doesn't work, it doesn't matter.
00:38:03.742 --> 00:38:09.482
Like you have enough entropy from sensors and voltage and whatever it is. And
00:38:09.482 --> 00:38:13.422
just force the user to shake or type on the keyboard. Like those are kind of the
00:38:13.422 --> 00:38:16.742
two different approaches there. But that just deals with this particular issue.
00:38:18.890 --> 00:38:24.850
Yeah, exactly. So, I mean, that's also great. And any issue that can be solved
00:38:24.850 --> 00:38:29.670
in the future is, of course, good. But at least in this sort of...
00:38:27.392 --> 00:38:31.232
Yeah, and libraries are a good way to prevent bugs in general, right? So good
00:38:31.232 --> 00:38:36.952
open source, well-reviewed libraries like SecP make it less likely. But entropy
00:38:36.952 --> 00:38:42.132
is device-specific, is wallet-specific. So it's very hard to have a reusable
00:38:42.132 --> 00:38:46.092
thing because every physical chip is different. You're not going to have shared
00:38:46.092 --> 00:38:46.612
code there.
00:38:46.652 --> 00:38:52.132
It's just one project. So then the solution is, I guess you just combine wallets.
00:38:52.232 --> 00:38:54.352
So you use multisig and with music too.
00:38:55.862 --> 00:39:01.182
If one device has a flaw, the other one maybe not have the flaw. Problem there is
00:39:00.520 --> 00:39:05.480
Right, yeah, I mean, that's more a user-side solution, of course.
00:39:01.182 --> 00:39:01.622
complexity.
00:39:06.042 --> 00:39:11.722
But that might be the only solution, because if everybody relies on a single
00:39:06.080 --> 00:39:07.360
I was...
00:39:11.722 --> 00:39:16.272
device, even if different people rely on different devices, then there's going to
00:39:16.272 --> 00:39:18.092
be bugs. It's just inevitable.
00:39:20.890 --> 00:39:25.710
Right. Okay. So that's, you know, insofar that you want to give any, you know,
00:39:25.950 --> 00:39:31.950
officially Sjors mandated security advice, then you do think the
00:39:31.950 --> 00:39:36.750
sort of multi-fender hardware wallet, multisig setup, that's a good way to go.
00:39:37.942 --> 00:39:43.182
Well, with this really, really big trade-off, that is complicated. And complexity
00:39:43.182 --> 00:39:47.182
is how people lose backups. People have two or three...
00:39:49.362 --> 00:39:53.182
And they assume that if they lose one of the three devices, they're fine. But
00:39:53.182 --> 00:39:56.702
they're not because you still need the xpub from that third device that you lost.
00:39:56.882 --> 00:40:02.492
And if you didn't... So it is, in theory, if we can get the software to the point
00:40:02.492 --> 00:40:06.472
where it's idiot-proof and everybody knows what to backup. But other than that,
00:40:06.572 --> 00:40:10.272
if you have to decide, like, am I going to, you know, what's my long-term risk?
00:40:10.752 --> 00:40:15.152
Is it... My device that I randomly bought, like I just pick one of the reputable
00:40:15.152 --> 00:40:20.022
manufacturers, I buy their device, put all my savings on it, and then hope that
00:40:20.022 --> 00:40:25.102
device is not the one that's broken? Or do I do this complicated multisig setup
00:40:25.102 --> 00:40:30.602
where now my risks are I lose my backup, I did something just wrong in the setup,
00:40:30.742 --> 00:40:34.642
I used a bunch of additional software to do this, that software had a backdoor
00:40:34.642 --> 00:40:38.602
and it was tricking me into sending money to the wrong place because it...
00:40:38.772 --> 00:40:42.312
Took over my browser and replaced the exchange address and I checked it on the
00:40:42.312 --> 00:40:47.192
devices, on multiple devices, the destination address, but the money went to the
00:40:47.192 --> 00:40:51.452
hacker because there was malware in one of these complicated pieces of software.
00:40:51.692 --> 00:40:55.332
Or do I use a third party that helps me with the multisig? There's companies out
00:40:55.332 --> 00:41:00.832
there. Yeah, now I've lost all my privacy because of the way they generally do
00:41:00.832 --> 00:41:01.072
that.
00:41:02.352 --> 00:41:07.952
Or I'm subject to KYC now. There just is no good way to deal with this, I think.
00:41:08.422 --> 00:41:12.722
We should continue improving things until there is. But currently, I think the
00:41:12.722 --> 00:41:16.502
current state of the art is you kind of, yeah, it's not good.
00:41:18.032 --> 00:41:21.512
And the solution is not like send it to a custodian because that's also not good.
00:41:18.320 --> 00:41:20.220
Yeah, that's, yeah,
00:41:21.592 --> 00:41:22.252
We already know that.
00:41:24.420 --> 00:41:29.540
that's also my impression. And that's, you know, it sucks. And in a way, I've
00:41:29.540 --> 00:41:34.060
been saying this for a long time that I actually think, yeah.
00:41:31.142 --> 00:41:33.182
I guess I can add one more point maybe.
00:41:36.282 --> 00:41:41.022
We probably shouldn't be writing off a cold card immediately either. Permanently,
00:41:41.162 --> 00:41:44.622
anyway. I mean, obviously, you probably want to, you know, ignore them for a year
00:41:44.622 --> 00:41:48.842
or so to see how they, you know, how they behave, if they really make their
00:41:48.842 --> 00:41:52.882
quality higher. But there aren't that many hardware wallets out there. And they
00:41:52.882 --> 00:41:53.722
all have trade-offs.
00:41:55.092 --> 00:41:59.312
You know, some of them have shitcoins, which means how many developer hours are
00:41:59.312 --> 00:42:02.392
actually going into the part that we care about versus how much goes into other
00:42:02.392 --> 00:42:04.052
stuff that might itself be a weakness.
00:42:06.352 --> 00:42:09.492
Some don't have secure elements, so they're maybe less secure against physical
00:42:09.492 --> 00:42:14.052
attack. We're not in a space yet where we have like 100 products, and if one of
00:42:14.052 --> 00:42:17.852
them fails and misbehaves, we just completely ignore them and we don't worry
00:42:17.852 --> 00:42:20.112
about it because there's 99 others out there. We're not there yet.
00:42:21.840 --> 00:42:27.680
Okay, so you think Coinkite, COLDCARD, you know, can recover from this
00:42:27.680 --> 00:42:32.820
and in a way you're sort of saying they deserve a second chance even? Yeah, of
00:42:30.772 --> 00:42:33.892
Technically, they can. I don't know if financially they can. They might get sued
00:42:32.820 --> 00:42:35.640
course. Right. Yeah.
00:42:33.892 --> 00:42:34.652
to death, basically.
00:42:37.320 --> 00:42:38.640
Yeah, no, but...
00:42:37.402 --> 00:42:40.582
But the spirit of, you know, and even if they don't, then hopefully they're nice
00:42:40.582 --> 00:42:44.702
enough to say, you know what, we as a company are doomed. We're going to
00:42:44.702 --> 00:42:48.442
completely open source the hardware and the software of what we have and we'll
00:42:48.442 --> 00:42:52.702
let other people take another stab at it. Though their bankruptcy lawyer might
00:42:52.702 --> 00:42:56.602
not like that because it's like giving away value.
00:42:56.762 --> 00:42:59.182
So, yeah, maybe that whole design is trapped.
00:43:02.580 --> 00:43:08.040
Yeah, no, generally speaking, I, you know, this has been a, I think everyone
00:43:08.040 --> 00:43:12.580
agrees is basically a disaster for Bitcoin. It's a disaster for people that lost
00:43:12.580 --> 00:43:18.520
money. But also it really drills down, drills home the idea that there just
00:43:18.520 --> 00:43:20.780
is no good solution of how to actually.
00:43:21.330 --> 00:43:26.030
Store your coins. Everything has pretty significant tradeoffs. I personally don't
00:43:26.030 --> 00:43:30.450
know what to recommend anymore. I mean, I will make a best effort recommendation
00:43:30.450 --> 00:43:35.030
to, you know, whoever is asking, depending on the technical capability or how
00:43:35.030 --> 00:43:39.790
much Bitcoin they own. But the idea that there is anything resembling a one
00:43:39.790 --> 00:43:45.350
-size-fits-all solution is just not there, which is,
00:43:44.672 --> 00:43:50.152
No, and then you start adding inheritance to the problem where some non-technical
00:43:45.490 --> 00:43:46.710
yeah.
00:43:50.152 --> 00:43:55.992
successor has to be able to use it. But also the $5 range attack shouldn't be
00:43:55.992 --> 00:43:59.312
able to take advantage of whatever that mechanism is by just triggering your
00:43:59.312 --> 00:44:03.752
inheritance basically. So yeah, it's a completely unsolved problem.
00:44:03.872 --> 00:44:09.432
We need to keep working on it. And this was probably maybe a warning to not think
00:44:09.432 --> 00:44:10.432
that we've already solved it.
00:44:11.820 --> 00:44:17.620
Right. You briefly mentioned COLDCARD, Coinkite potentially
00:44:17.620 --> 00:44:23.620
getting sued or whatever, but as a more general point, what is your
00:44:23.620 --> 00:44:28.440
assessment of people potentially getting any of their money back if they lost it
00:44:28.440 --> 00:44:32.140
in this hack? Do you think there's a path to that at all?
00:44:33.052 --> 00:44:37.512
Well, assuming that they catch whoever did it, right? If they don't catch it or
00:44:37.512 --> 00:44:42.292
if it turns out it was North Korea, then nobody's getting anything back. But
00:44:42.292 --> 00:44:48.042
let's say that the, you know, people think that some script kiddie used an AI and
00:44:48.042 --> 00:44:53.382
found the bug and then kind of in a rush took the money because there were some
00:44:53.382 --> 00:44:59.302
patterns that they were using APIs of Block Explorer services instead of
00:44:59.302 --> 00:45:01.822
just, you know, doing a very high OPSEC.
00:45:05.502 --> 00:45:10.762
And who knows? Maybe they used an AI with actual logging. Probably not.
00:45:11.122 --> 00:45:13.962
I don't know if they used AI. Maybe they didn't. Maybe they just looked at the
00:45:13.962 --> 00:45:15.502
code and realized what was going on.
00:45:18.052 --> 00:45:22.452
Anyway, let's assume that the money is retrieved. Then it presumably goes into
00:45:22.452 --> 00:45:26.952
some sort of foundation, either back to the company or like a dedicated vehicle
00:45:26.952 --> 00:45:31.392
that's focused on dealing with this because as a company, it's a massive
00:45:31.392 --> 00:45:31.872
distraction.
00:45:33.242 --> 00:45:38.712
Now such a vehicle could, I think part of the secret was, part of the mnemonic
00:45:38.712 --> 00:45:43.882
was derived from the device identifier, or at least the chip series identifier,
00:45:44.282 --> 00:45:48.622
something not unique, maybe not unique to devices, but at least narrows it down a
00:45:48.622 --> 00:45:52.822
bit. And so people can, if they hold on to the physical device, can use that
00:45:52.822 --> 00:45:56.062
device as part of the evidence that they are the owner.
00:45:56.672 --> 00:46:00.652
But the thing is, I don't think it's good enough that you can automate it. So you
00:46:00.652 --> 00:46:04.522
can't make a DAO. You know, like in Ethereum land, they might have set up a smart
00:46:03.440 --> 00:46:03.740
Sure.
00:46:04.522 --> 00:46:08.022
contract, put all the money in a smart contract, and then you can redeem yourself
00:46:08.022 --> 00:46:12.122
automatically from the smart contract. I don't think that's possible because the
00:46:12.122 --> 00:46:14.762
device just doesn't have a way to prove it, right?
00:46:14.762 --> 00:46:19.562
You can just use a simulator and pretend that you're the owner. That means humans
00:46:19.562 --> 00:46:24.862
have to be involved. And given that they're a company based in Canada, those
00:46:24.862 --> 00:46:28.962
humans are going to be very heavily regulated. That means anybody who wants to
00:46:28.962 --> 00:46:32.902
file a claim probably has to prove their funds, not just from a point of view of
00:46:32.902 --> 00:46:37.002
proving that you are the owner, but proving that you're not a criminal, right?
00:46:37.302 --> 00:46:42.822
So it's full KYC, AML. So that makes it even more tragic in a way because people
00:46:42.822 --> 00:46:45.362
who are doing self-custody, it's not even that they're trying to evade the law or
00:46:45.362 --> 00:46:49.422
anything, but they want to be independent of the system and they're kind of
00:46:49.422 --> 00:46:51.822
forced into the system to get their money back.
00:46:54.570 --> 00:46:58.730
Yeah, I mean, I'm sure everyone would take that trade off at this point.
00:46:56.262 --> 00:47:00.432
I don't see how you can just anonymously... Sorry?
00:47:00.310 --> 00:47:03.650
I mean, I'm sure everyone would take that trade-off at this point if you at least
00:47:03.650 --> 00:47:05.870
get your money back that way or at least part of it.
00:47:06.532 --> 00:47:08.912
Yeah, yeah, of course. But it's painful.
00:47:07.170 --> 00:47:08.730
But yeah,
00:47:10.250 --> 00:47:16.150
so you're saying indeed there's an identifier in the hardware itself
00:47:16.150 --> 00:47:22.030
that could help. So if you're listening and your device was drained, don't throw
00:47:22.030 --> 00:47:25.870
it out yet because it might come in helpful.
00:47:22.872 --> 00:47:27.672
Yeah, definitely don't throw it out. But then in addition, my guess is because
00:47:27.672 --> 00:47:32.662
you need to prove that you own those coins for legal reasons. And that also
00:47:32.662 --> 00:47:37.182
serves as proof that, you know, that makes it more likely that they're sending it
00:47:37.182 --> 00:47:40.142
to the correct person. So such an, I would imagine, some sort of trust or
00:47:40.142 --> 00:47:44.222
foundation makes an inventory of people who are making claims, all the evidence
00:47:44.222 --> 00:47:48.322
they've submitted, and then like a waiting period, say like, hey, if you think,
00:47:48.402 --> 00:47:50.162
you know, somebody's taking your money, let us know.
00:47:50.902 --> 00:47:54.482
And otherwise, we're just going to give it to the people who made a claim and
00:47:54.482 --> 00:47:59.122
reasonably prove that it was them, whatever's left. And then these lawyers are
00:47:59.122 --> 00:48:03.722
charging $1,000 an hour at least. Just draining half the pot and it's going to
00:48:03.722 --> 00:48:08.002
take longer than Mt. Gox. That's my guess. But if the Bitcoin price, you know,
00:48:08.422 --> 00:48:14.062
10x in the meantime, then that might still be a decent outcome compared to losing
00:48:14.062 --> 00:48:14.422
everything.
00:48:15.510 --> 00:48:21.210
Yeah, and then also one of the ways... Yeah, yeah. And then also one of the ways
00:48:16.342 --> 00:48:18.422
You know, which is what happened with Mt. Gox, right?
00:48:21.210 --> 00:48:25.550
you can potentially prove that these coins were yours is ironically to show the
00:48:25.550 --> 00:48:29.010
KYC records that you bought them and then exchange and withdrew or something like
00:48:29.010 --> 00:48:31.610
that. That could be one way.
00:48:29.622 --> 00:48:30.522
Yep, exactly.
00:48:34.580 --> 00:48:40.300
Okay, I think that's rough. Is there anything else you want to say about like
00:48:40.300 --> 00:48:45.080
either the bug or the aftermath or the process or anything we've been discussing
00:48:45.080 --> 00:48:45.540
so far?
00:48:46.382 --> 00:48:49.442
No, I think we covered most of that or all of it.
00:48:50.770 --> 00:48:56.270
Yeah, so then let's get into sort of our part because
00:48:56.270 --> 00:49:02.150
CoinKite was a sponsor of ours, of course. In fact, it seems
00:49:02.150 --> 00:49:07.810
that they sponsored the entire Bitcoin ecosystem or the entire podcastosphere,
00:49:07.810 --> 00:49:09.090
it looked like.
00:49:10.030 --> 00:49:15.830
And that's become part of the story also, like all
00:49:10.342 --> 00:49:11.622
Quite a few, yeah.
00:49:15.830 --> 00:49:16.770
these podcasters.
00:49:18.170 --> 00:49:23.750
They look stupid for promoting this hardware wallet now. And yeah, we're
00:49:23.750 --> 00:49:29.750
part of that. So yeah, I have some thoughts on that. I think
00:49:29.750 --> 00:49:34.030
you have some thoughts on that. As a sort of general question,
00:49:34.410 --> 00:49:38.830
yeah, what do you think we are part of the problem here?
00:49:40.262 --> 00:49:43.422
I think that's a good question. And I'm not sure.
00:49:45.762 --> 00:49:49.952
Because... Like we know that sponsorships in general can cause problems, right?
00:49:50.152 --> 00:49:54.412
And especially on YouTube, like any sponsor you will see, you will be able to
00:49:54.412 --> 00:49:58.532
find out somebody debunking it as a complete scam. And that has something to do
00:49:58.532 --> 00:50:03.892
with the companies with the most budget, the most marketing budget causing being
00:50:03.892 --> 00:50:04.432
problematic.
00:50:05.372 --> 00:50:10.592
However, I think with a lot of these high budget sponsors out there,
00:50:10.962 --> 00:50:16.242
if you actually do some like... Asking around between competent people, they'll
00:50:16.242 --> 00:50:21.082
explain to you why it's not good. That was not the case here at all. There are
00:50:21.082 --> 00:50:25.342
many technically reputable people who are using COLDCARD right until the end.
00:50:25.702 --> 00:50:30.122
So it wasn't the case of, oh, if you just asked two technical people, they would
00:50:30.122 --> 00:50:35.962
have told you it's a scam. That's just not the case. So the question is, how much
00:50:35.962 --> 00:50:40.802
due diligence should you do as a podcaster? I think we did at least some by not
00:50:40.802 --> 00:50:44.532
just... I'm sure there would have been other companies, other hardware
00:50:44.532 --> 00:50:48.752
manufacturers out there that nobody's even heard of that are less good,
00:50:48.872 --> 00:50:49.372
basically.
00:50:52.030 --> 00:50:57.470
Yeah, I can share my thoughts on this because you're touching
00:50:57.470 --> 00:51:03.430
on... I think there's kind of two aspects to it. Or first of all, generally
00:51:03.430 --> 00:51:08.500
speaking, you know, as media producers,
00:51:08.860 --> 00:51:14.320
or I would even say, you know, in my case, I think we're making a journalistic
00:51:14.320 --> 00:51:18.300
podcast ultimately, or at least I consider myself a journalist.
00:51:18.600 --> 00:51:24.310
And for better or worse, mostly for worse, For the most part
00:51:24.310 --> 00:51:28.770
these days, that is funded by sponsorships, advertisements, that kind of thing.
00:51:29.150 --> 00:51:33.930
In an ideal world, that wouldn't even be the case, right? In an ideal world, you
00:51:33.930 --> 00:51:38.830
could make media based on subscription models.
00:51:39.030 --> 00:51:43.230
And there are some examples of that that are successful or even donations or
00:51:43.230 --> 00:51:49.190
something like that. But other than that, for a... The overwhelming majority of
00:51:49.190 --> 00:51:54.950
media out there, it's funded through advertisements, sponsorships, and that's
00:51:55.370 --> 00:51:57.550
the case for our podcast as well.
00:51:57.710 --> 00:52:03.490
Although, I considered it more of a hobby what we were doing. We were not getting
00:52:03.490 --> 00:52:05.330
a lot of money out of it, as you know.
00:52:06.272 --> 00:52:10.952
No, and that's kind of tricky as well, right? Because the budget is so small that
00:52:10.952 --> 00:52:16.532
we couldn't hire a separate auditor to investigate their product because it would
00:52:16.532 --> 00:52:20.392
just have been more expensive than anything we ever got from the podcast. But
00:52:20.392 --> 00:52:23.392
that's more of a situation because we are a small podcaster.
00:52:24.020 --> 00:52:29.000
Yeah, but just generally speaking, the problem exists in journalism for a long
00:52:29.000 --> 00:52:34.940
time, which is essentially a journalist. Journalism is
00:52:35.840 --> 00:52:41.760
built on reputation and telling the truth and informing your audience
00:52:41.760 --> 00:52:44.160
in an objective way, essentially.
00:52:44.700 --> 00:52:50.620
While an advertiser or marketing that's in conflict with that, there's always a
00:52:50.620 --> 00:52:56.060
tension there. They want to... Use the reputation of the journalism, of the
00:52:56.060 --> 00:53:01.060
journalist, of the publication to sell their products. But it's not the same
00:53:01.060 --> 00:53:06.280
thing. It's journalism and marketing are kind of, you know, they need each other,
00:53:06.300 --> 00:53:09.240
but they're also in tension with each other.
00:53:09.740 --> 00:53:10.440
So then,
00:53:09.882 --> 00:53:15.522
Yeah, and it's a combination of audience size and reputation, right? It's two
00:53:15.522 --> 00:53:21.052
equally sized audiences, but a more reputable journal is, as a sponsor,
00:53:21.172 --> 00:53:25.572
presumably more attractive. So the trust is part of the equation. Yeah,
00:53:22.800 --> 00:53:24.000
yeah, of course.
00:53:25.460 --> 00:53:29.340
Yeah, of course. I'm just pointing there's a natural tension there always. That
00:53:29.340 --> 00:53:34.940
always existed. But there's also solutions for that, essentially. And
00:53:34.940 --> 00:53:39.960
I think there's two big responsibilities that we have and had.
00:53:40.500 --> 00:53:46.020
So the first of all, and this is the one you pointed out, is that we should
00:53:46.020 --> 00:53:51.980
do at least basic due diligence to make sure that we're not promoting a
00:53:51.980 --> 00:53:52.320
scam.
00:53:53.290 --> 00:53:58.150
If we're promoting something like OneCoin that's super obviously a scam and
00:53:58.150 --> 00:54:02.270
people are going to lose money if they invest in it, that's a bad look on us. We
00:54:02.270 --> 00:54:07.850
need to take some measures to avoid that. But I
00:54:07.850 --> 00:54:13.170
don't think that applies here. I don't think Coinkite or ColdCard was a scam.
00:54:13.670 --> 00:54:17.690
As you mentioned, there were a lot of people that looked at this product,
00:54:17.990 --> 00:54:23.900
including pretty serious security researchers. That did not see
00:54:23.900 --> 00:54:28.480
this bug. They did not find this bug. So it was a reputable
00:54:28.480 --> 00:54:34.160
company. And in fact, that's why we took them as a sponsor.
00:54:35.240 --> 00:54:40.320
One of the claims that's gone out there is, you know, Coinkite kind of bought off
00:54:40.320 --> 00:54:45.790
the entire podcasting system to sell their defunct products. But that's, speaking
00:54:45.790 --> 00:54:49.670
for ourselves, that's the opposite of what happened. We were happy that a
00:54:49.670 --> 00:54:53.560
reputable... Seemingly solid company wanted to sponsor it.
00:54:53.800 --> 00:54:58.970
That is why we said yes to it. It's not like some kind of hush. Yeah, it's not
00:54:56.872 --> 00:55:00.722
I agree with that. And I actually liked the situation.
00:54:58.970 --> 00:55:00.530
like we took hush money.
00:55:02.530 --> 00:55:03.170
Go on.
00:55:03.012 --> 00:55:06.532
I like the situation much more than our previous arrangement where we had no
00:55:06.532 --> 00:55:10.112
control of the advertisements. I think technically we had some sort of veto, but
00:55:10.112 --> 00:55:14.852
when we were part of the Bitcoin magazine, they would splice in the ads and they
00:55:14.852 --> 00:55:17.612
would be for stuff that we wouldn't know anything about.
00:55:18.972 --> 00:55:21.492
Whereas now at least we had a sponsor that we didn't know.
00:55:23.030 --> 00:55:28.430
Yeah, in our judgment, it was, and I would even say still is, like I don't think
00:55:28.430 --> 00:55:31.390
they were scammers. I don't think that is what happened. I think it was a
00:55:31.390 --> 00:55:37.050
disaster and a really bad bug, but I don't think they, you know, they stole the
00:55:37.050 --> 00:55:40.970
money. At least I would want to see real evidence before I believe something like
00:55:40.970 --> 00:55:41.170
that.
00:55:41.310 --> 00:55:45.090
I don't think that's what happened. And at the very least, I can say that, you
00:55:45.090 --> 00:55:50.070
know, no one, there was no one that suggested to us that we should drop them or,
00:55:50.250 --> 00:55:55.740
you know, that... We had them because we thought they were reputable. That's the
00:55:55.740 --> 00:55:59.710
reason we had them as sponsor. So I think that's our first part.
00:55:59.810 --> 00:56:05.620
We need to make sure that we're not promoting a scam.
00:56:05.900 --> 00:56:11.220
And reasonably thinking, I do think we did that. And then I think the other
00:56:11.220 --> 00:56:15.120
aspect is that we do need to make very clear.
00:56:16.910 --> 00:56:22.190
What is sponsored content essentially, right? Like what part of the podcast is us
00:56:22.190 --> 00:56:27.590
making content? Is me being a journalist and you being a Bitcoin core developer?
00:56:27.890 --> 00:56:32.510
And in what part of the podcast we're actually doing like an advertisement or a
00:56:32.510 --> 00:56:36.990
sponsored thing? And I do think also in that we were pretty clear in that.
00:56:37.190 --> 00:56:42.730
Like it was kind of over the top clear. I listened back to it and it's cringe to
00:56:42.730 --> 00:56:47.310
listen back to it, right? Because we are saying, or especially me, like I'm
00:56:47.310 --> 00:56:51.930
saying these very positive things about the cold card and indeed, even like over
00:56:51.930 --> 00:56:56.210
the top, like I'm calling it like a piece of art and these things, but at least
00:56:55.452 --> 00:56:59.492
I think it was so over the top that it kind of proved that they were sponsoring
00:56:56.210 --> 00:56:56.770
I,
00:56:59.492 --> 00:57:03.052
us mostly just because they wanted to help out, not because they expected to make
00:57:03.052 --> 00:57:08.982
money from it. In a more serious, larger podcasting setup, I'm not even
00:57:08.982 --> 00:57:14.882
sure if any company would appreciate when there's so much tongue-in-cheek
00:57:12.840 --> 00:57:14.180
right, yeah.
00:57:14.882 --> 00:57:15.942
in the advertisement.
00:57:16.322 --> 00:57:16.822
I
00:57:17.410 --> 00:57:21.530
Yeah, I think that's, you know, that's the second part of our response. Like the
00:57:21.530 --> 00:57:25.490
first part is, you know, it shouldn't be an obvious scam. And the second part, it
00:57:25.490 --> 00:57:29.250
does also need to be very obvious and clear that we're doing like an
00:57:29.250 --> 00:57:34.530
advertisement rather than content. And I do think we did that on both counts. So,
00:57:34.990 --> 00:57:37.950
yeah, it's a disaster what happened and it really sucks.
00:57:38.170 --> 00:57:43.050
And, you know, if we could turn back the hands of time, like we wouldn't take
00:57:43.050 --> 00:57:43.730
them as a sponsor.
00:57:45.380 --> 00:57:50.460
At the same time, I find it very hard to blame myself for it or
00:57:50.460 --> 00:57:52.020
to...
00:57:50.582 --> 00:57:54.602
mean, if we had a time machine, and this raises all sorts of physics questions,
00:57:54.722 --> 00:57:55.702
but ignoring those.
00:57:55.100 --> 00:57:57.000
We
00:57:56.932 --> 00:58:00.632
If we had known about this bug earlier, we would have responsibly disclosed it
00:57:59.430 --> 00:58:01.110
would have prevented it. Yeah, exactly.
00:58:00.632 --> 00:58:04.172
and we would have told them like, hold off with sponsorship until you've fixed
00:58:04.172 --> 00:58:08.292
this totally, total vulnerability. Now, maybe a more realistic scenario would
00:58:08.292 --> 00:58:13.782
have been if we had known about... The precise engineering style or something.
00:58:14.602 --> 00:58:18.302
But again, it's not like I went to conferences and people were telling me, oh,
00:58:18.302 --> 00:58:19.902
you should look at that cold card software.
00:58:20.082 --> 00:58:24.642
It's absolute crap. Look, so sloppy. There must be a bug in there. That's totally
00:58:24.642 --> 00:58:28.462
not what happened. And that does happen with other projects. So, yeah.
00:58:28.980 --> 00:58:34.600
Yeah, so I mean, the one thing I've been mulling over whether or
00:58:33.892 --> 00:58:38.532
But obviously, if we had known there was a red flag, yeah, then we would have had
00:58:34.600 --> 00:58:35.400
not...
00:58:38.532 --> 00:58:42.052
to make that more ethical discussion. Like, do we want to have a sponsor knowing
00:58:42.052 --> 00:58:44.472
that we don't like how they do things?
00:58:46.210 --> 00:58:51.500
Yeah, the one thing I have been mulling over is that, you know,
00:58:46.532 --> 00:58:48.172
But I don't think that was the case.
00:58:51.720 --> 00:58:57.720
in the... On the point of making it very clear which part
00:58:57.720 --> 00:59:02.400
of the podcast is us making content, is me being a journalist and you being a
00:59:02.400 --> 00:59:05.360
developer versus what part is sponsorship.
00:59:06.740 --> 00:59:12.500
It has become a norm in podcast land that... The same people, the hosts
00:59:12.500 --> 00:59:16.860
also do the ad reads. And again, in our case, it was tongue in cheek. And I think
00:59:16.860 --> 00:59:20.940
it was pretty obvious, but, you know, it is maybe better to not have that and to
00:59:20.940 --> 00:59:24.720
have what we had before, where it's just a recorded piece of audio. There's like
00:59:24.720 --> 00:59:29.180
some sound effects and you get a completely different person doing the ad like
00:59:29.180 --> 00:59:29.440
that.
00:59:29.520 --> 00:59:31.860
That is maybe better than what we did.
00:59:33.080 --> 00:59:34.400
So that that's sort of the.
00:59:33.202 --> 00:59:39.072
I think that would be fine if the sponsor did not, like if
00:59:39.072 --> 00:59:42.112
let's say we have a new sponsor and they don't want us to be tongue-in-cheek
00:59:42.112 --> 00:59:45.212
about it but much more serious about it, then I think you would get into this
00:59:45.212 --> 00:59:49.632
territory where we might say, well, maybe we should just have another voice do it
00:59:49.632 --> 00:59:53.532
to make it clear that we're not endorsing something or, you know, to keep more
00:59:53.532 --> 00:59:54.452
distance, basically.
00:59:56.292 --> 01:00:00.872
Because it's not just about, I think that might be a good distinction. There's
00:59:56.560 --> 00:59:57.560
Yeah, exactly.
01:00:00.872 --> 01:00:05.112
the distinction between making it clear that it is an ad, but also whether you're
01:00:05.112 --> 01:00:09.212
endorsing it to what degree you are. I think in our case, you were doing it so
01:00:09.212 --> 01:00:14.572
tongue-in-cheek that, I don't know, but at the same time, of course, the fact
01:00:14.572 --> 01:00:17.332
that you list an ad at all is some level of endorsement.
01:00:17.872 --> 01:00:21.312
But a lot of sponsors will go much beyond that, right? They will say, oh, I
01:00:21.312 --> 01:00:25.822
personally use this, blah, blah, blah, all my wealth is on it. And then you get
01:00:23.010 --> 01:00:23.330
Right.
01:00:25.822 --> 01:00:30.822
into murky territory where I tend to agree with, then you really need a separate
01:00:30.822 --> 01:00:35.402
sound or a separate person to make that distinction. Like, yes, we think they're
01:00:35.402 --> 01:00:40.382
reliable enough to be a sponsor, but it's not more than that.
01:00:41.692 --> 01:00:44.692
But it depends on what it is. If it's like a conference, like, yeah, sure, go to
01:00:41.760 --> 01:00:42.740
Yeah, I mean, and.
01:00:44.692 --> 01:00:48.852
the conference. Like, that's not a recommendation that is dangerous to make, even
01:00:48.852 --> 01:00:51.472
if the conference is terrible. Like, that's not the end of the world.
01:00:52.760 --> 01:00:54.400
Yeah, yeah, and not
01:00:52.972 --> 01:00:55.432
But if you tell people, like, oh, put your life savings here,
01:00:56.230 --> 01:01:02.210
to mention that there's, of course, also, I won't name any names, but
01:01:02.210 --> 01:01:06.230
there are, of course, also podcasts that, like, where the content is just
01:01:06.230 --> 01:01:10.740
recommending. COLDCARD, where it's not specifically an advertisement, but it's
01:01:10.740 --> 01:01:14.580
the actual host actually recommending, like, this is what you need to do.
01:01:15.220 --> 01:01:20.940
We didn't do that. Again, I think in our case, it was clear. I hope for everyone
01:01:20.940 --> 01:01:26.260
that there's a distinction between the sponsored part and us being a journalist
01:01:26.260 --> 01:01:27.120
and developer part.
01:01:28.400 --> 01:01:32.760
So, yeah, I mean, at the end of the day, I just hope no one got hurt. You know, I
01:01:32.760 --> 01:01:36.320
hope no one that listened to our podcasts.
01:01:34.892 --> 01:01:39.372
None of our listeners, you mean, or not as a result of us recommending it. But,
01:01:39.872 --> 01:01:40.312
you know.
01:01:39.890 --> 01:01:42.590
Yeah, that, well, both, I guess, yeah.
01:01:44.610 --> 01:01:49.410
But I mean, it's very possible, of course. Well, I do think in our case, because
01:01:49.410 --> 01:01:53.730
by the time we started our, not that this is important, but by the time we
01:01:53.730 --> 01:01:58.930
started having them as a sponsor, the MK4 was already out. So it kind of reduces
01:01:58.930 --> 01:02:04.880
the risk that maybe one of our listeners got the MK3 and their money is stolen
01:02:04.242 --> 01:02:07.282
Yeah, I don't think that's a strong argument, though, because you could still
01:02:04.880 --> 01:02:05.180
away.
01:02:06.320 --> 01:02:10.780
No, no, I mean, I'm not making an argument. I'm just, you know, some thoughts
01:02:07.282 --> 01:02:09.062
say, well, no.
01:02:10.780 --> 01:02:13.860
that have been floating through my head because at the end of the day, it does,
01:02:14.040 --> 01:02:17.620
you know, it is a disaster and it does suck that it happened to our sponsor.
01:02:19.302 --> 01:02:23.882
Yeah, and, you know, even if the MK4 came out, like, somebody might have had an
01:02:23.882 --> 01:02:27.802
MK3, and because we talked about the MK4, they were like, okay, I guess Coldcard
01:02:27.802 --> 01:02:33.712
is solid, and I'm not going to throw it away and buy some competitor. So, but, I
01:02:33.712 --> 01:02:37.572
mean, you weren't making the case anyway, but I'm just... Thoroughly dismissing
01:02:36.670 --> 01:02:41.290
Yeah, but so as a, I guess as a general question, like,
01:02:37.572 --> 01:02:38.392
that argument.
01:02:38.832 --> 01:02:39.392
But
01:02:41.820 --> 01:02:45.700
let's say, you know, one of the other reputable,
01:02:44.902 --> 01:02:49.482
also, let's say we've had Ledger as a sponsor and their social media hack
01:02:47.060 --> 01:02:49.000
well, I was just going to ask that.
01:02:49.482 --> 01:02:53.142
happened and people got kidnapped. That wouldn't look good either. So again,
01:02:54.282 --> 01:02:56.552
they're not unique in this.
01:02:55.150 --> 01:03:01.150
Yeah, or so to ask the question in a very general way to you, like let's say, you
01:03:01.150 --> 01:03:06.690
know, someone from Ledger or Trezor or Jade or Foundation or whatever, Bitbox. So
01:03:06.690 --> 01:03:10.490
someone is listening and they think, oh, they don't have a sponsor anymore. Let's
01:03:10.490 --> 01:03:15.990
reach out. Would we do anything differently compared to what we did?
01:03:16.150 --> 01:03:21.110
Like, is there anything we should have done differently in your view? Or is this
01:03:21.110 --> 01:03:24.630
just kind of a disaster?
01:03:23.552 --> 01:03:28.612
Well, this comes to the bigger problem is our podcast is too small. So taking on
01:03:28.612 --> 01:03:34.552
a sponsor is not giving me any meaningful income. But it is exposing us to
01:03:34.552 --> 01:03:40.352
reputational risk. So I would say we just shouldn't take a new sponsor for that
01:03:40.352 --> 01:03:45.252
reason unless we scale the podcast or it's a really large amount.
01:03:47.402 --> 01:03:50.682
So it's kind of a difficult discussion for us. I mean, if we were to do it, yeah,
01:03:47.990 --> 01:03:48.950
Yeah, I
01:03:50.802 --> 01:03:55.842
you can do the things like having them record the audio and splicing it in, et
01:03:55.842 --> 01:04:00.082
cetera, et cetera. But I think fundamentally... It's like pennies in front of
01:04:00.082 --> 01:04:04.702
steamrollers. It was like 0.1% of my income, but it still
01:04:02.230 --> 01:04:05.570
suppose
01:04:04.702 --> 01:04:06.142
happens.
01:04:06.542 --> 01:04:09.682
I would have rather just paid the cost ourselves, right?
01:04:11.230 --> 01:04:16.910
I agree with that. In our case specifically, it's the amount of money. We're a
01:04:16.910 --> 01:04:20.570
tiny podcast, no one listens to us, and the amount of money we were getting
01:04:20.570 --> 01:04:26.330
reflects that. We would have been better off to just not have a
01:04:26.330 --> 01:04:29.630
sponsor and consider it a pure hobby, which we're kind of doing anyways.
01:04:30.332 --> 01:04:34.532
Well, look, in the beginning, we were doing at least one episode a week, and that
01:04:34.532 --> 01:04:38.792
would have been at least some reasonable income. But once we started doing four
01:04:38.792 --> 01:04:43.652
episodes a year, at the same time, it's nice to be paid by somebody because it's
01:04:43.652 --> 01:04:47.812
a form of appreciation, right? It does give you a motive. It does motivate you a
01:04:47.812 --> 01:04:52.252
little bit to see that people appreciate it enough to sponsor it.
01:04:52.472 --> 01:04:52.812
Yeah,
01:04:52.500 --> 01:04:58.140
It made it feel a little bit real. It made it feel like we were making a podcast
01:04:58.140 --> 01:05:02.280
rather than just sort of friends hanging out and talking about Bitcoin for an
01:05:02.280 --> 01:05:03.440
hour. So, yeah.
01:05:04.720 --> 01:05:09.800
Well, we'll see. In case there is a potential sponsor listening, feel free to
01:05:09.800 --> 01:05:13.860
reach out and we'll at least think about it, I guess. But, yeah.
01:05:13.592 --> 01:05:15.292
starting at 1 million a year. Yeah.
01:05:17.690 --> 01:05:22.610
Okay, well, Sjors, anything else on this topic, on the topic of them having
01:05:21.402 --> 01:05:22.262
No, I think
01:05:22.610 --> 01:05:28.310
us as a sponsor or the bug itself
01:05:28.310 --> 01:05:31.950
or lessons learned or how to move forward from here or anything else?
01:05:32.502 --> 01:05:37.402
we've covered a lot. So thank you all for listening to Bitcoin.
01:05:38.550 --> 01:05:39.150
Explained.